_read_loop does not recover after Socket timeout about betfair HOT 9 CLOSED

Not sure I agree this is the concern of the library - I think it's up to the user to decide the behaviour in this situation; it might be that some users want to perform error recovery in other parts of their system if a socket error occurs.

Another question is deciding/configuring the behaviour - how many times would it retry? Infinitely? What if the Betfair streaming API is down? Would it continue to do this for [potentially] hours until the API came back up?

initial_clk and clk are already available on the BaseStream which MarketStream and OrderStream derive from and they can be passed into subscribe_to_markets or subscribe_to_orders when reconnecting which can be done by the user when catching a SocketError.

from betfair.

Agreed, at the moment I use retrying in my framework but this is a bit lazy as it recreates everything. It would be nice if lightweight took advantage of initial_clk / clk to resubscribe when a socket error occurs.

When initially developed closing the socket and raising an error was the easy option, welcome any thoughts on how this should be refactored.

from betfair.

At the moment I have just patched my local version to pass when a timeout occurs and it works but I will have to work on the error mode a bit more (using clk as you mentioned).

from betfair.

Yeh I think you are right Aaron, in principle it would be nice for it to reconnect but if there is an issue with the API its going to make a mess.

from betfair.

Personally I would retry for hours but that reflects my approach to APIs and automation. I can see both sides of the argument.
I'm finding it a bit tricky to deal with it all in a nice way on my side of the code (as it's all wrapped up in a thread), but I'll work on it a bit more and see if I can make this issue redundant.

from betfair.

@agberk, how can you catch the socket error in the main thread if you are running the socket/listener in a Thread (ie called start with async=True parameter)? I am trying to implement what you suggested in your post but most advice in tutorials online have more control over the thread itself to do things like pass exceptions into a queue. Thanks for your help.

from betfair.

You can't catch any errors when using async, you need to create your own thread/function with error catching / retrying logic.

from betfair.

That's right - you can't do it with async but that's just doing the thread management for you, so as Liam says you can create your own one.

For example I do something like this:

def betfair_stream_run(config, price_queue):
    betfair_api = betfairlightweight.APIClient(
        config['betfair_username'],
        config['betfair_password'],
        config['betfair_application_key'],
        config['betfair_certificate_dir'])

    while True:
        try:
            betfair_api.login()
            betfair_stream_socket = betfair_api.streaming.create_stream(
                unique_id=1,
                description=config['description'],
                listener=StreamListener(output_queue=price_queue)
            )
            betfair_stream_socket.subscribe_to_markets(
                unique_id=12345,
                market_filter=betfairlightweight.filters.streaming_market_filter(**config.get('market_filter', {})),
                market_data_filter=betfairlightweight.filters.streaming_market_data_filter(**config.get('market_data_filter', {}))
            )
            betfair_stream_socket.start()
        except betfairlightweight.exceptions.SocketError:
            logger.warning('SocketError in betfair stream thread.')
            time.sleep(5.0)

You can create a thread for this and run it in the main thread:

t = threading.Thread(target=betfair_stream_run, args=(config, price_queue))
t.daemon = True
t.start()

from betfair.

Thanks @liampauling and @agberk , that makes perfect sense. I was a bit too stuck in my mind on using bfl's start for this because it's otherwise very effective, but I think I can follow this model and set up a thread.

from betfair.