Optimisation about betfair HOT 17 CLOSED

Just pushed turbo, got 1000 marketbooks down to 1.1s ish, this includes the use of ciso8601 and slots on a few of the classes.

Need to think of a neat way to return just the json if required, maybe add a 'lightweight' variable to APIClient could work, if True return just the json from the endpoints. Is that what you guys are after?

@agberk, yeh I am keen to add the raw streaming JSON messages as well.

from betfair.

With the branch 'streaming_refactor' seeing a 60% increase in speed, purely based off the changes to how the price/size objects are updated, stored and serialised.

Processing 'horse-racing-pro-sample' through the historical endpoint (lightweight=True):

Before: 2829.5s
After: 994.1s

from betfair.

So setattr is very very very slow, currently takes around 11s to create 1000 market books which is unacceptable. With a refactor of BaseResource it's possible to get this below 0.04s, 275x faster!

from betfair.

@liampauling on that note it would be great to have the option to just return JSON rather than these resources. I have my own order book implementations and currently have a bastardised betfairlightweight with your resources stripped out due to their added latency.

I also use an asyncio event loop and it would be great to split the business logic in the functions from the IO, giving users the freedom to execute the request with whatever http library they desire. I'll try and get a PR example together for you.

from betfair.

Also interested in having access to the raw JSON

from betfair.

Its always been there:

https://github.com/liampauling/betfairlightweight/blob/master/betfairlightweight/resources/baseresource.py#L68

from betfair.

Will be pushing a branch in an hour or so for review, could maybe move the serialisation of the json into a function which could be disabled. However this branch is considerably quicker which should also dramatically reduce CPU when streaming.

from betfair.

But all of the functions still build the base resources regardless correct?

from betfair.

Yeh at the moment, which is why its so slow.

from betfair.

Cool - I'll try and find some time to whip up a PR for you. Looking forward to checking our your new branch

from betfair.

Whoops I think I may have confused things - I was talking about the raw JSON messages from the streaming API, which this thread isn't about :)

from betfair.

• ujson now used instead of json
• Added a lightweight flag to APIClient and to all endpoint functions, when True this returns the serialised json returned from betfair
• Added a lightweight flag to Listener so that streaming returns raw data
• Clean up to baseendpoint

MarketBook request with requests mocked:

• v1.0.1 : 0.01s
• turbo: lightweight=False : 0.001s (x10)
• turbo: lightweight=True: 0.00005s (x20)

So a x200 speed improvement when setting lightweight=True compared to the current release with no breaking changes.

from betfair.

Sounds great, nice one!

from betfair.

• Refactor of BaseEndpoint.request as it was calling .json() twice
• Addition of some headers that betfair recommend
• utils/metadata clean up

Now available on pypi, v1.1.1

from betfair.

Left this open as I still think there are some options available, profiling the code it seems as though a lot of time is spent serialising the data (ordering/list creation) when nothing has changed. Want to look into the following:

• Caching certain function calls

• Storing the data in a format already ordered, this would slow down any updates but speed up the creation of a MarketBook

• Move the prosessing / storage of streaming to C (need to learn C first, long term goal...)

I have converted my framework to use the historical endpoint (streaming data using flumine). When it comes to backtesting I am seeing that it takes around 40-60s per horse racing market, want to get this down as much as possible.

from betfair.

@limx0 -- it's been a while, but have you had any luck with an asyncio compatible api? If not I might think about how to do this in a clean way.

from betfair.

@synapticarbors I ended up just re-implementing functions that I wanted to use in an async context. I've been off for a few months but I might try and whip up a PR example in the next few days.

@liampauling is this something you'd be open to? (Now that I'm not the only one :) )

from betfair.