betcode-org / betfair Goto Github PK

I find certificate handling a bit confusing at the moment, since I would expect it to work like for requests. That is the certs argument for APIClient should be either a path to a .pem file, or a tuple with the paths to cert and key files.

Also, here you want to use ```os.path.join``, because if self.certs doesn't end with `/`, then the generated path is wrong, but I was not sure about opening a PR just for that.

Any objections?

Its a pain to use compared to the scores endpoint because its used for the website, whereas scores just requires appKey to be authorised.

Given my experience trying to get betfairlightweight built as a conda package on conda-forge, I think it would be good to have tests run on Windows via appveyor:

https://www.appveyor.com

Like other CI platforms, it's free for open source projects. This would help the project catch platform dependent errors during development.

• trading.streaming.*
• trading.in_play_service.*
• marketFilter data structure
• August 8th API release integration

Streaming to be updated to use BaseResource along with refactoring. In play service to be implemented although the api constantly changes due to it not being private.
Investigate the use of a marketFilter data structure to make it easier to make requests, may not be implemented if it is found to be slow.
New placeOrder options now available, customerRefStrategy/timeInForce etc. Need to add to resources / test integration.

betfairstream.py's _connect sets self._running = True before the socket connection actually succeeds.

If the socket fails to connect for some reason (see example stacktrace that occurs when the internet is not connected), betfairlightweight will think it is running. This causes issues in error handling because you can't trust the _running attribute.

(PS sorry for filing another bug without resolving my others first!)

Example of socket connect failure when the internet is down:

  File "/home/anon/.virtualenvs/anon-app/local/lib/python2.7/site-packages/betfairlightweight/streaming/betfairstream.py", line 194, in _send
    self._connect()
  File "/home/anon/.virtualenvs/anon-app/local/lib/python2.7/site-packages/betfairlightweight/streaming/betfairstream.py", line 131, in _connect
    self._socket = self._create_socket()
  File "/home/anon/.virtualenvs/anon-app/local/lib/python2.7/site-packages/betfairlightweight/streaming/betfairstream.py", line 139, in _create_socket
    s.connect((self.__host, self.__port))
  File "/usr/lib/python2.7/ssl.py", line 433, in connect
    self._real_connect(addr, False)
  File "/usr/lib/python2.7/ssl.py", line 420, in _real_connect
    socket.connect(self, addr)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
gaierror: [Errno -3] Temporary failure in name resolution

I wanted to play around with this package but due to some constraints with other tooling, couldn't upgrade to Python 3. I spent a little bit of time making a few modifications such that the test suite passes under both Python 2.7 and 3.5. I'm happy to open a pull request, but wanted to know if you were open to possibly merging it or if you'd prefer to keep it Python 3 only for code purity (which I totally understand).

betfairlightweight was initially developed to be a wrapper that only abstracted the requests part of calling endpoints and required json as a parameter. However in order to make things simpler I would like to propose the replacing params with the correct parameters per endpoint.

def list_event_types(self, params=None, session=None):

def list_event_types(self, filter=MarketFilter(), locale=None, session=None, params=None):
    :param filter: MarketFilter
    :param locale: The language used for the response
    :param session: Requests session object
    :param params: Json request, default if not None

The added complication will be that more objects will have to be created/imported, for example the MarketFilter before a request can be made.

#34 provided by @rozzac90 is basically what I am referring to but with backwards compatibility.

Add listRunnerBook to .betting

http://docs.developer.betfair.com/docs/display/1smk3cen4v3lu3yomq5qye0ni/listRunnerBook

The examples are broken, probably since commit e16e0d9

I've tried to fix it but was not successful. I'll try again if nobody else can get to it sooner.

market_ids does not get returned in the same way as in the betfair API

event_ids = [28200940, 28200941, 28200939, 28200937, 28202622, 28188158, 28202623, 28188159, 28202621, 28188157, 28202619, 28202616, 28202617, 28202614, 28202615, 28196677, 28202626, 28188160, 28202625, 28188176, 28188177]

markets = betfairclient.trading.betting.list_market_catalogue(
            max_results=100,
            filter=filters.market_filter(
                 event_ids=event_ids,
                 market_type_codes = ["MATCH_ODDS"]
            ))

yields mismatch between market_id and eventId

I looked briefly at the code but could not see any reordering, I could've missed something to. :)

In the betfair documentation, a RunnerCatalogue's metadata is not mandatory. For some sports/events, particularly non-horse racing, metadata is not returned. Betfair RunnerCatalog

However in bettingresources.py#L190, metadata is configured as mandatory.

Since release 1.1.2, list_market_catalogue method fails when it tries to make RunnerCatalogue resources as part of the MarketCatalogue resource.

In this traceback, debugging showed that there was no metadata in the runners returned by listMarketCatalogue: {u'handicap': 0.0, u'runnerName': u'San Antonio Spurs', u'selectionId': 237471, u'sortPriority': 1}

Traceback:

Traceback (most recent call last):
  File "anon-app.py", line 63, in <module>
    jobs.update_market_subscription_list()
  File "/home/anon/github/anon-app/jobs/job.py", line 185, in update_market_subscription_list
    market_id_list = self.betfair_api.betting.list_market_catalogue(market_filter, market_projection=market_projection, max_results=100, lightweight=False)
  File "/home/anon/.virtualenvs/anon-app/local/lib/python2.7/site-packages/betfairlightweight/endpoints/betting.py", line 158, in list_market_catalogue
    return self.process_response(response, resources.MarketCatalogue, elapsed_time, lightweight)
  File "/home/anon/.virtualenvs/anon-app/local/lib/python2.7/site-packages/betfairlightweight/endpoints/baseendpoint.py", line 100, in process_response
    return [resource(elapsed_time=elapsed_time, **x) for x in result]
  File "/home/anon/.virtualenvs/anon-app/local/lib/python2.7/site-packages/betfairlightweight/resources/bettingresources.py", line 222, in __init__
    self.runners = [RunnerCatalogue(**i) for i in kwargs.get('runners', [])]
TypeError: __init__() takes exactly 6 arguments (5 given)

Similar to issue #69.

Contrary to the betfair documentation, the returned json of a placeOrders call won't have a persistenceType attribute if the timeinForce of the submitted bet is FILL_OR_KILL.

When building the resources from a successful place_orders call, building the LimitOrder resource for a PlaceOrderInstructionReport will fail:
https://github.com/liampauling/betfairlightweight/blob/master/betfairlightweight/resources/bettingresources.py#L593

{u'timeInForce': u'FILL_OR_KILL', u'price': 1.4, u'size': 5.0}

 File "/home/anon/.virtualenvs/anon-app/local/lib/python2.7/site-packages/betfairlightweight/endpoints/betting.py", line 299, in place_orders
   return self.process_response(response, resources.PlaceOrders, elapsed_time, lightweight)
 File "/home/anon/.virtualenvs/anon-app/local/lib/python2.7/site-packages/betfairlightweight/endpoints/baseendpoint.py", line 102, in process_response
   return resource(elapsed_time=elapsed_time, **result)
 File "/home/anon/.virtualenvs/anon-app/local/lib/python2.7/site-packages/betfairlightweight/resources/bettingresources.py", line 664, in __init__
   PlaceOrderInstructionReports(**i) for i in kwargs.get('instructionReports')
 File "/home/anon/.virtualenvs/anon-app/local/lib/python2.7/site-packages/betfairlightweight/resources/bettingresources.py", line 644, in __init__
   self.instruction = PlaceOrderInstruction(**instruction) if instruction else None
 File "/home/anon/.virtualenvs/anon-app/local/lib/python2.7/site-packages/betfairlightweight/resources/bettingresources.py", line 621, in __init__
   self.order = LimitOrder(**limitOrder)
TypeError: __init__() takes at least 4 arguments (4 given)

I've run into an issue using the order stream which results in exceptions like the following (order_queue is a Queue which is the output_queue for a StreamListener):

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/aaron/workspace/xxx/xxx/order_feed/order_feed.py", line 35, in run
    order_book = self.order_queue.get()
  File "/usr/local/lib/python2.7/dist-packages/betfairlightweight/resources/baseresource.py", line 63, in json
    return json.dumps(self._data)
  File "/usr/lib/python2.7/json/__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: datetime.datetime(2017, 2, 16, 11, 18, 59) is not JSON serializable

On further testing, this appeared to happen when the CurrentOrders returned from the OrderBookCache contained an UnmatchedOrder which had a matched_date field that was not None (it was a datetime.datetime object).

As you can see from the placedDate field a few lines down, that datetime.datetime object is converted to its string representation when being serialised:

https://github.com/liampauling/betfairlightweight/blob/master/betfairlightweight/resources/streamingresources.py#L425

The terminology as described in the documentation is quite confusing:

http://docs.developer.betfair.com/docs/display/1smk3cen4v3lu3yomq5qye0ni/Exchange+Stream+API#ExchangeStreamAPI-OCM/OrderChangeMessag

It seems like an UnmatchedOrder can contain a matched_date in the cases of partial matches.

I've created a pull request which contains the change I made locally to get it to work - you may have a nicer way of dealing with it.

#39

A race condition can occur between _add_stream and MarketStream's _process if you subscribe on an open/running socket.

Subscribing to markets creates an entire new MarketStream object which has an empty cache. If then an mcm arrives on the stream before the img, there will be a KeyError in the cache because _process can't update or initialise the cache unless it has received an img update.

If the listener already has a market stream set when the user is re-subscribing to markets, the existing MarketStream object/cache could be re-used (validated by its unique_id perhaps). Alternatively you can stop, start the socket and subscribe again but that seems expensive.

@synapticarbors regarding OpenSSL, this is a requirement of Requests, what errors are you getting on the conda build?

Betfair are adding a streaming status:

http://forum.bdp.betfair.com/showthread.php?t=3428

Probably worth adding the ability to choose the streaming endpoint as well.

In preparing a conda recipe on conda-forge, I observed several tests failing on windows when appveyor tried to run the tests:

======================================================================
FAIL: test_get_app_key (tests.test_baseclient.BaseClientTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\users\josh\documents\github\betfairlightweight\tests\test_baseclient.py", line 94, in tes
t_get_app_key
    self.client.get_app_key()
AssertionError: AppKeyError not raised

======================================================================
FAIL: test_strip_datetime (tests.test_baseresource.BaseResourceInit)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\users\josh\documents\github\betfairlightweight\tests\test_baseresource.py", line 87, in t
est_strip_datetime
    assert type(stripped) == datetime.datetime
AssertionError

======================================================================
FAIL: test_strip_datetime_resource (tests.test_baseresource.BaseResourceInit)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\users\josh\documents\github\betfairlightweight\tests\test_baseresource.py", line 116, in
test_strip_datetime_resource
    assert model_response.datetime_int == datetime.datetime.utcfromtimestamp(1465631675000 / 1e3)
AssertionError

----------------------------------------------------------------------
Ran 203 tests in 0.844s

FAILED (failures=3)

test_get_app_key:

This test fails because os.environ.get(self.username), where self.username=="username" returns a non-None response since USERNAME is a standard windows environment variable (note that Windows env variables are case insensitive).

test_strip_datetime:

Fails on py27 since integer is a long rather than an int. There is an easy fix for this.

test_strip_datetime_resource

There is a platform dependent exception that is raised so only catching ValueError is insufficient.

Hi, @liampauling.

I think it's not a good idea to use snake_case in func/arg names.
Of course, it is accepted in the python, but the API-NG is done in camelCase mode. Continuous conversions are hardly reasonable. Also, this adds confusion to the code, for example, when creating an object, we use one style
cancel_bet = filters.cancel_instruction(bet_id=bet.id)

and accessing it's fields in a different style
foo(cancel_bet["betId"])

It's easy to make a typo that will be difficult to catch. Working with API-NG documentation you get used to camel style, etc. Therefore, I suggest using camelCase in the library.

What do you think?

I've run into the following error when trying to run an order stream:

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/aaron/xxx/betfair_order_feed.py", line 94, in betfair_stream_run
    betfair_stream_socket.start()
  File "/home/aaron/tmp/venv/betfairlightweight/local/lib/python2.7/site-packages/betfairlightweight/streaming/betfairstream.py", line 49, in start
    self._read_loop()
  File "/home/aaron/tmp/venv/betfairlightweight/local/lib/python2.7/site-packages/betfairlightweight/streaming/betfairstream.py", line 177, in _read_loop
    self._data(received_data)
  File "/home/aaron/tmp/venv/betfairlightweight/local/lib/python2.7/site-packages/betfairlightweight/streaming/betfairstream.py", line 204, in _data
    if self.listener.on_data(received_data) is False:
  File "/home/aaron/tmp/venv/betfairlightweight/local/lib/python2.7/site-packages/betfairlightweight/streaming/listener.py", line 83, in on_data
    self._on_change_message(data, unique_id)
  File "/home/aaron/tmp/venv/betfairlightweight/local/lib/python2.7/site-packages/betfairlightweight/streaming/listener.py", line 108, in _on_change_message
    self.stream.on_subscribe(data)
  File "/home/aaron/tmp/venv/betfairlightweight/local/lib/python2.7/site-packages/betfairlightweight/streaming/stream.py", line 40, in on_subscribe
    self._process(book_data, publish_time)
  File "/home/aaron/tmp/venv/betfairlightweight/local/lib/python2.7/site-packages/betfairlightweight/streaming/stream.py", line 132, in _process
    self._caches[market_id] = OrderBookCache(publish_time=publish_time, **order_book)
  File "/home/aaron/tmp/venv/betfairlightweight/local/lib/python2.7/site-packages/betfairlightweight/resources/streamingresources.py", line 488, in __init__
    self.runners = [OrderBookRunner(**i) for i in kwargs.get('orc', [])]
TypeError: __init__() got an unexpected keyword argument 'hc'

This is because of data like the following received from Betfair:

{
{
    "id": "1.131975445",
    "orc": [
        {
            "hc": -0.25,
            "fullImage": true,
            "id": 4917452,
            "mb": [
                [
                    1.27,
                    4.37
                ]
            ]
        }
    ]
}

There's no documentation about this field in the API docs: http://docs.developer.betfair.com/docs/display/1smk3cen4v3lu3yomq5qye0ni/Exchange+Stream+API#ExchangeStreamAPI-OCM/OrderChangeMessage

I've opened a ticket with BDP.

PR incoming.

After reviewing other wrappers looking to change apiclient to be similar to twilio:

• trading.login()
• trading.keep_alive()
• trading.logout()
• trading.navigation.*
• trading.betting.*
• trading.account.*
• trading.scores.*
• trading.streaming.*
• trading.in_play_service.*

Enables the wrapper to more closely resemble the API as well as removing the spaghetti code that is apimethod. This will of course break everything that is built using this wrapper, welcome any comments.

When I call stop() on the listener socket I get the following AttributeError.

<snip>
  File "/home/anon/.virtualenvs/anon/local/lib/python2.7/site-packages/betfairlightweight/streaming/betfairstream.py", line 56, in stop
    if self._socket is not None and not self._socket._closed:
AttributeError: 'SSLSocket' object has no attribute '_closed'

I can't find this _closed attribute in any of the Python documentation or else I'd try and file a PR for you!

Looking at the cancelOrders API call:

http://docs.developer.betfair.com/docs/display/1smk3cen4v3lu3yomq5qye0ni/cancelOrders
https://github.com/liampauling/betfairlightweight/blob/master/betfairlightweight/endpoints/betting.py#L303-L320

According to the Betfair documentation there are no required parameters - however the Betting.cancel_orders method requires market_id and instructions as positional args.

I think the most intuitive way to do things is that positional args correspond to required parameters and kwargs correspond to parameters which are not required.

At the moment in this specific instance (cancel_orders), the user is required to pass parameters which they do not want to use and it's not immediately obvious what they should set it these parameters to if they don't want them to be used.

The utils.clean_locals(...) function will not include keys whose value is None so any parameters which are not required can just be defaulted to None.

To resolve the issue Betting.cancel_orders needs to have market_id and instructions changed to kwargs defaulted to None.

I haven't looked at any of the other methods but there's the possibility there are other instances of this, so I think a full pass of the methods on the endpoints should performed, comparing the args / kwargs to the correspondng Betfair API required / not required parameters.

I'm happy to both look through the API calls and put the resulting changes into a PR.

The BetfairStream class has a member _running which is used to indicate if the stream is... running or not.

If you run a BetfairStream in async mode and you get a socket exception in _read_loop, there's no way to determine if the thread has died as the class doesn't store a reference to the thread it starts and _running isn't updated in the case of the SocketError exceptions; checking _running from an outside thread would indicate that it's still going.

I think the best way to solve this is just with a finally block when _running is set to False. I'm not sure whether any socket cleanup needs to be done in case of an error? My brief googling didn't give me a definitive answer.

Raised PR #46

Just profiled my framework when backtesting, when ordered by cumtime you can see that the creation of the base resource is slow. When streaming this is amplified because so many market books are created.

Implementing #57 should help.

Welcome any thoughts on how to speed this up.

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
655587/1320    3.554    0.000    8.619    0.007 //resources/baseresource.py:19(__init__)
655587/1320    2.660    0.000    8.591    0.007 //resources/baseresource.py:52(set_attributes)
21608/1276    0.290    0.000    8.503    0.007 //resources/baseresource.py:34(set_sub_resources)
30599/1282    0.862    0.000    8.488    0.007 //resources/baseresource.py:45(<listcomp>)

This is not actually an issue, But there is no way to contact people on github. I am having issues with my app key not being authorized, Do you happen to know if this has to be a live key, And if it has to be specially activated by betfair?

Thanks,
Eyal.

Would you be interested in adding functionality like this (or does it exist and I am missing it again..)? https://github.com/limx0/btcmarkets#even-more-control

Basically, I have an optional arg that allows the api to simply build a dict of the request to be sent, but allows the user more control over which http library they use, rather than being tied to requests.

This is useful for async workflows, and particularly important when doing Inplay betting (don't want code to be handing while your order inserts)

Thoughts?

Add atomic function to get current marketbook or orderbook.

listener.snap(market_id='1.1234567')

This could remove the dependency for an output queue (make optional) and allow the user to use the listener as a cache that can be requested when required.

Create stream 'on_update' function:

def on_update(self, market_books): if self.output_queue: self.output.queue.put(market_books)

Today I had some issues with the keepalive not returning json.
Of course this caused an exception where it could not decode the json.

Should unexpected, non-json responses be caught more tidily?

I wasn't able to debug whether the keepAlive succeeded despite not returning json nor whether this was just a temporary issue with betfair.

Currently if clk/initial_clk are provided BetfairStream will attempt a reconnect however listener.register_stream wipes all stream data. This will then cause issues because updates will be provided from betfair but the images have been removed.

If it is a reconnect the listener needs to just update the unique_id of the stream and not delete anything. This could maybe be the default and a new function created which 'refreshes' the listener if it wants to be reused.

At the moment logging statements are called directly from the logging module (e.g. logging.info(...)) - this implicitly creates a root logger and adds a default handler if it doesn't exist.

This can cause unwanted output for application developers.

The following links propose some best practice for logging in a library:
http://docs.python-guide.org/en/latest/writing/logging/
https://docs.python.org/3/howto/logging.html#configuring-logging-for-a-library

My understanding from the links above would be to make the following changes in betfairlightweight:

1. Add the following to betfairlightweight.__init__.py:

# Set default logging handler to avoid "No handler found" warnings.
import logging
try:  # Python 2.7+
    from logging import NullHandler
except ImportError:
    class NullHandler(logging.Handler):
        def emit(self, record):
            pass

logging.getLogger(__name__).addHandler(NullHandler())

This would mean by default nothing would get logged by library unless configured by the user (this is optional since if the next point is implemented, it's easy for the user to configure the library to disable logging).

2. In all modules which use logging.* add the following at the module level:

logger = logging.getLogger(__name__)

And then switch all logging.* statements to logger.* statements.

I'm happy to submit a pull requests with the changes, but I wanted to check whether you had any issues with it or other ideas first.

betfairlightweight-1.1.8

list_market_book() work fine, until market has no orders.
But with placed order it raise exception __init__() got an unexpected keyword argument 'matchesByStrategy'

This is because RunnerBook does not take a matchesByStrategy as keyword argument.

Hi,

Thanks for your library.

Im facing some issues with streaming only, the rest working just fine ! (great work though)

I tried this

betfair_socket = trading.streaming.create_stream(unique_id=1) betfair_socket.start(async=False) betfair_socket.authenticate() betfair_socket.subscribe_to_markets(market_filter={'eventTypeIds':['1'], 'inPlayOnly': True, 'marketTypes':["MATCH_ODDS"]}, market_data_filter={'fields': ['EX_BEST_OFFERS']})

and I'm ending up with this result:

`1 {"op":"connection","connectionId":"006-051016081016-209520"}

None {"op":"status","statusCode":"FAILURE","errorCode":"TIMEOUT","errorMessage":"Connection is not subscribed and is idle: 15000 ms","connectionClosed":true,"connectionId":"006-051016081016-209520"}
`

Any help pls where im messing up ?

Once more thanks again for your effort.

Regards

When recording data from a market I currently use .json() however this doesn't take advantage of streaming. The update or initial image is provided in .streaming_update but you then have to record data from the beginning of the session (not ideal if you only want in play data)

Therefore propose a new param called .streaming_image which creates a synthetic betfair image based on the current state of the market cache. The challenge will be limiting CPU as it will probably have to be calculated at the cache level for every update rather than a function in the MarketBook class.

Found an issue where the following became present in MarketBook:

{'size': 0, 'price': 0}

Tried to recreate but I can't looks like it was due to receiving an update to delete a price when it was not present in the cache.

If there is an error in the stream and connectionClosed is True the BetfairStream will quietly close, considering changing this to raise an error instead.

https://github.com/liampauling/betfairlightweight/blob/master/betfairlightweight/streaming/listener.py#L138

Add a raise error after this line so that the socket cleans itself up first.
https://github.com/liampauling/betfairlightweight/blob/master/betfairlightweight/streaming/betfairstream.py#L205

It seems that on handicap markets the selectionId will be repeated:

[{"marketId":"1.131434037","isMarketDataDelayed":false,"status":"OPEN","betDelay":0,"bspReconciled":false,"complete":true,"inplay":false,"numberOfWinners":0,"numberOfRunners":66,"numberOfActiveRunners":66,"lastMatchTime":"2017-05-13T09:06:13.495Z","totalMatched":350.69,"totalAvailable":15649.09,"crossMatching":true,"runnersVoidable":false,"version":1645619435,"runners":[{"selectionId":579339,"handicap":-4.0,"status":"ACTIVE"},{"selectionId":197960,"handicap":4.0,"status":"ACTIVE"},{"selectionId":579339,"handicap":-3.75,"status":"ACTIVE"},{"selectionId":197960,"handicap":3.75,"status":"ACTIVE"},{"selectionId":579339,"handicap":-3.5,"status":"ACTIVE"},{"selectionId":197960,"handicap":3.5,"status":"ACTIVE"},{"selectionId":579339,"handicap":-3.25,"status":"ACTIVE"},{"selectionId":197960,"handicap":3.25,"status":"ACTIVE"}, ...

The streaming cache uses the selectionId as a primary key which isn't going to work on handicap markets.

Add thread lock to TransactionCount to prevent race conditions when using multiple threads.

I have created a slack group to discuss any issues:

betfairlightweight.slack.com

I was getting this error:
self.output_queue.put(output_market_book)
AttributeError: 'NoneType' object has no attribute 'put'

To fix this in apiclient.py I added:

import queue
output_queue = queue.Queue()
and passed this to StreamListener in create_stream()

If a socket timeout occurs in the _read_loop function then the socket is stopped and this affects services that are expecting data from the listener.

It's my opinion that the _read_loop should recover from the timeout and keep receiving data without manual intervention from the user.

https://github.com/liampauling/betfairlightweight/blob/master/betfairlightweight/streaming/betfairstream.py#L159

            except (socket.timeout, socket.error) as e:
                self.stop()
                raise SocketError('[Connect: %s]: Socket %s' % (self.unique_id, e))

Would it be possible to just pass a lightweight flag to the APIClient rather than to every function? I think it's more likely people would use it for all/none of their API calls rather than for specific API calls.

Thoughts?

Use ciso8601 for parsing datetimes as its super quick.

https://pypi.python.org/pypi/ciso8601/1.0.1

• order limits
• request limits
• currency limits (stake/liability)

I've come across the following in my streaming orders:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/xxx/betfair_order_feed.py", line 79, in betfair_stream_run
    betfair_stream_socket.start()
  File "/xxx/local/lib/python2.7/site-packages/betfairlightweight/streaming/betfairstream.py", line 49, in start
    self._read_loop()
  File "/xxx/local/lib/python2.7/site-packages/betfairlightweight/streaming/betfairstream.py", line 177, in _read_loop
    self._data(received_data)
  File "/xxx/local/lib/python2.7/site-packages/betfairlightweight/streaming/betfairstream.py", line 204, in _data
    if self.listener.on_data(received_data) is False:
  File "/xxx/venv/local/lib/python2.7/site-packages/betfairlightweight/streaming/listener.py", line 84, in on_data
    self._on_change_message(data, unique_id)
  File "/xxx/local/lib/python2.7/site-packages/betfairlightweight/streaming/listener.py", line 115, in _on_change_message
    self.stream.on_update(data)
  File "/xxx/local/lib/python2.7/site-packages/betfairlightweight/streaming/stream.py", line 58, in on_update
    self._process(book_data, publish_time)
  File "/xxx/local/lib/python2.7/site-packages/betfairlightweight/streaming/stream.py", line 132, in _process
    order_book_cache.update_cache(order_book, publish_time)
  File "/xxx/local/lib/python2.7/site-packages/betfairlightweight/resources/streamingresources.py", line 506, in update_cache
    runner.update_unmatched(order_changes.get('uo', []))
  File "/xxx/local/lib/python2.7/site-packages/betfairlightweight/resources/streamingresources.py", line 476, in update_unmatched
    self.unmatched_orders[n] = UnmatchedOrder(**unmatched_order)
TypeError: __init__() takes at least 18 arguments (17 given)

This is from the following JSON:

"uo":[
	{
		"status":"EC",
		"md":1497288589000,
		"pd":1497288587000,
		"pt":"L",
		"sr":0,
		"sv":34.2,
		"id":"95102351117",
		"p":3,
		"s":34.2,
		"rfs":"xxx",
		"sm":0,
		"sl":0,
		"sc":0,
		"ot":"L",
		"rfo":"xxx",
		"side":"L"
	}
]

The unmatched order is missing the "rac" and "rc" fields for regulator auth code and regulator code respectively. These are defined as positional args and are thus required when constructing the object. I suggest they simply be turned into kwargs so that it's handled if they're missing.

PR incoming.

When the stream connects a connection-id is returned, this is currently stored in the class (not mentioned in the init), will be handy to log when there is an error to make any bug communications with betfair easier.

The following MarketCatalogue JSON was received from Betfair and caused the following error:

{
	"marketStartTime": "2017-06-06T09:00:00.000Z",
	"competition": {
		"id": "11330922",
		"name": "Prostejov Challenger 2017"
	},
	"marketId": "1.132045327",
	"totalMatched": 17.06,
	"marketName": "Match Odds",
	"event": {
		"timezone": "UTC",
		"openDate": "2017-06-06T09:00:00.000Z",
		"id": "28260245",
		"countryCode": "CZ",
		"name": "Marrero/Paes v Knowle/Zelenay"
	},
	"runners": [
		{
			"handicap": 0,
			"runnerName": "Knowle/Zelenay",
			"selectionId": 5717065,
			"sortPriority": 1
		},
		{
			"handicap": 0,
			"runnerName": "Marrero/Paes.",
			"selectionId": 8786145,
			"sortPriority": 2
		},
		{
			"handicap": 0,
			"sortPriority": 3,
			"selectionId": 8781561
		}
	]
}

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/aaron/workspace/sportsrisq.traderng/sportsrisq/traderng/feed/betfair_market_feed.py", line 42, in betfair_market_feed
    locale=config.get('markets', {}).get('locale')
  File "/home/aaron/tmp/venv/betfairlightweight/local/lib/python2.7/site-packages/betfairlightweight/endpoints/betting.py", line 158, in list_market_catalogue
    return self.process_response(response, resources.MarketCatalogue, elapsed_time, lightweight)
  File "/home/aaron/tmp/venv/betfairlightweight/local/lib/python2.7/site-packages/betfairlightweight/endpoints/baseendpoint.py", line 100, in process_response
    return [resource(elapsed_time=elapsed_time, **x) for x in result]
  File "/home/aaron/tmp/venv/betfairlightweight/local/lib/python2.7/site-packages/betfairlightweight/resources/bettingresources.py", line 231, in __init__
    self.runners = [RunnerCatalogue(**i) for i in kwargs.get('runners', [])]
TypeError: __init__() takes at least 5 arguments (4 given)

The third runner is missing the runnerName field - which is listed a required field in the Betfair API docs: http://docs.developer.betfair.com/docs/display/1smk3cen4v3lu3yomq5qye0ni/Betting+Type+Definitions#BettingTypeDefinitions-RunnerCatalog

Because of this (and I think rightly so) the constructor for a RunnerCatalogue doesn't have a default value for runnerName: https://github.com/liampauling/betfairlightweight/blob/master/betfairlightweight/resources/bettingresources.py#L190-L195

I'm not really sure what the solution is here - maybe it's just to do nothing and accept that Betfair sends malformed data sometimes - however that will cause a listMarketCatalogue call to error in the above way. If all you want is the JSON then you can avoid it by using the lightweight parameter in the API call.

I have raised a ticket with BDP but I imagine the response will be it was just a mistake.

The bigger question is whether resources should handle fields which are missing which are listed as required in the API specification? It's also a question which I don't have the answer to :)

Betfair has a rest API endpoint, although it doesn't follow any of the REST principles it does remove some of the complications lightweight solves. Not sure why I used the json-rpc endpoint in the first place.

Outcome will make the code easier to read and slightly quicker.