I just experienced an error with the combination of the libraries websocket and websocket-client: AttributeError: 'module' object has no attribute 'WebSocketApp'

It appears that websocket overrides the what is actually needed from websocket-client. I could resolve the issue with: pip uninstall websocket; pip uninstall websocket-client; pip install websocket-client.

Therefore I suggest to remove websocket from requirements.

Hi @gavincyi , how can i configure each exchange API to use custom authentication credentials?

Support exchange Yunbi

https://yunbi.com/documents/api/guide

if this subscription gets interrupted or my server restarts/etc, that's running this...
I'll get an error when trying to restart this again.
It says the "id" already exists and errors out.
Is there any way to restart this program and have it continue with the next incremental id in the mysql table?

The application is not working in csv mode

Reported by Shreyanshupare in #27

Refactor subscription.ini

• Remove all configuration items of order book and trade fields
• Write guidelines

Due to the socketIO-client

Provide an input parameter to configure the timestamp format of all messages. For example,

bitcoinexchangefh -dtformat epoch ...

the expected date time format should be 1500601665090 (in miliseconds), while

bitcoinexchangefh -dtformat "%Y-%m-%dT%H:%M:%S.%f" ...

the expected date time format should be 2017-07-21T00:01:02.345678.

Hi @gavincyi , is there a problem getting data from Bitstamp?

The print is stating that the connection is open but there is no instrument

2017-03-02 04:32:21,151 - INFO
[ExchGwApiBitstamp]
Socket is opened.

2017-03-02 04:32:21,151 - INFO
[ExchGwBitstamp]
Instrument "" is subscribed in channel Bitstamp

Thanks

Hi @gavincyi , how are you doing?

Is there a way to handle the APIs that responds with "too many requests", "unsubscribed" and "close".

Could you put the parameter "sleep", at the subscriptions.ini file?
It would be nice to control the waiting time for the next request

It would be great to have multiple sources for the same instrument as a redundancy mechanism
(Websocket, HTTP, FIX)

This are the status messages that i got in my log:

[ExchGwOkCoinWs] - Socket error: Connection is already closed.
[ExchGwOkCoinWs] - Socket is closed.
[ExchGwOkCoinWs] - Socket is going to reconnect...

[ExchGwOkCoin] - Instrument spotusd_btc is unsubscribed in channel OkCoin
[ExchGwOkCoin] - Instrument futureusd_btc_quarter is unsubscribed in channel OkCoin

[ExchGwQuoine] - Error in trades: HTTP Error 429: Too Many Requests
[ExchGwQuoine] - Error in order book: HTTP Error 429: Too Many Requests

For performance. Due to GIL, multi-threading is a bottleneck to the performance. So replace the thread with process to support running in multiple cores.

Is it available, or is it planned support for multiple pairs from each exchange?

I'm trying to log all ticker information from bittrex into a KDB+ database using the following subscriptions.ini file. This is almost a fulll list of tickers for Bittrex:

[Bittrex-BTC1ST-Restful]
exchange = Bittrex
instmt_name = BTC1ST
instmt_code = BTC-1ST
enabled = 1
[Bittrex-BTCAGRS-Restful]
exchange = Bittrex
instmt_name = BTCAGRS
instmt_code = BTC-AGRS
enabled = 1
[Bittrex-BTCAMP-Restful]
exchange = Bittrex
instmt_name = BTCAMP
instmt_code = BTC-AMP
enabled = 1
[Bittrex-BTCARDR-Restful]
exchange = Bittrex
instmt_name = BTCARDR
instmt_code = BTC-ARDR
enabled = 1
[Bittrex-BTCBAY-Restful]
exchange = Bittrex
instmt_name = BTCBAY
instmt_code = BTC-BAY
enabled = 1
[Bittrex-BTCBCY-Restful]
exchange = Bittrex
instmt_name = BTCBCY
instmt_code = BTC-BCY
enabled = 1
[Bittrex-BTCBYC-Restful]
exchange = Bittrex
instmt_name = BTCBYC
instmt_code = BTC-BYC
enabled = 1
[Bittrex-BTCCLOAK-Restful]
exchange = Bittrex
instmt_name = BTCCLOAK
instmt_code = BTC-CLOAK
enabled = 1
[Bittrex-BTCCLUB-Restful]
exchange = Bittrex
instmt_name = BTCCLUB
instmt_code = BTC-CLUB
enabled = 1
[Bittrex-BTCCPC-Restful]
exchange = Bittrex
instmt_name = BTCCPC
instmt_code = BTC-CPC
enabled = 1
[Bittrex-BTCCRB-Restful]
exchange = Bittrex
instmt_name = BTCCRB
instmt_code = BTC-CRB
enabled = 1
[Bittrex-BTCDASH-Restful]
exchange = Bittrex
instmt_name = BTCDASH
instmt_code = BTC-DASH
enabled = 1
[Bittrex-BTCDGD-Restful]
exchange = Bittrex
instmt_name = BTCDGD
instmt_code = BTC-DGD
enabled = 1
[Bittrex-BTCDYN-Restful]
exchange = Bittrex
instmt_name = BTCDYN
instmt_code = BTC-DYN
enabled = 1
[Bittrex-BTCEDG-Restful]
exchange = Bittrex
instmt_name = BTCEDG
instmt_code = BTC-EDG
enabled = 1
[Bittrex-BTCETH-Restful]
exchange = Bittrex
instmt_name = BTCETH
instmt_code = BTC-ETH
enabled = 1
[Bittrex-BTCFCT-Restful]
exchange = Bittrex
instmt_name = BTCFCT
instmt_code = BTC-FCT
enabled = 1
[Bittrex-BTCFTC-Restful]
exchange = Bittrex
instmt_name = BTCFTC
instmt_code = BTC-FTC
enabled = 1
[Bittrex-BTCGEO-Restful]
exchange = Bittrex
instmt_name = BTCGEO
instmt_code = BTC-GEO
enabled = 1
[Bittrex-BTCGNO-Restful]
exchange = Bittrex
instmt_name = BTCGNO
instmt_code = BTC-GNO
enabled = 1
[Bittrex-BTCGNT-Restful]
exchange = Bittrex
instmt_name = BTCGNT
instmt_code = BTC-GNT
enabled = 1
[Bittrex-BTCGRC-Restful]
exchange = Bittrex
instmt_name = BTCGRC
instmt_code = BTC-GRC
enabled = 1
[Bittrex-BTCHKG-Restful]
exchange = Bittrex
instmt_name = BTCHKG
instmt_code = BTC-HKG
enabled = 1
[Bittrex-BTCLBC-Restful]
exchange = Bittrex
instmt_name = BTCLBC
instmt_code = BTC-LBC
enabled = 1
[Bittrex-BTCLTC-Restful]
exchange = Bittrex
instmt_name = BTCLTC
instmt_code = BTC-LTC
enabled = 1
[Bittrex-BTCMEME-Restful]
exchange = Bittrex
instmt_name = BTCMEME
instmt_code = BTC-MEME
enabled = 1
[Bittrex-BTCMYST-Restful]
exchange = Bittrex
instmt_name = BTCMYST
instmt_code = BTC-MYST
enabled = 1
[Bittrex-BTCNAUT-Restful]
exchange = Bittrex
instmt_name = BTCNAUT
instmt_code = BTC-NAUT
enabled = 1
[Bittrex-BTCNAV-Restful]
exchange = Bittrex
instmt_name = BTCNAV
instmt_code = BTC-NAV
enabled = 1
[Bittrex-BTCNLG-Restful]
exchange = Bittrex
instmt_name = BTCNLG
instmt_code = BTC-NLG
enabled = 1
[Bittrex-BTCNXC-Restful]
exchange = Bittrex
instmt_name = BTCNXC
instmt_code = BTC-NXC
enabled = 1
[Bittrex-BTCNXS-Restful]
exchange = Bittrex
instmt_name = BTCNXS
instmt_code = BTC-NXS
enabled = 1
[Bittrex-BTCNXT-Restful]
exchange = Bittrex
instmt_name = BTCNXT
instmt_code = BTC-NXT
enabled = 1
[Bittrex-BTCPKB-Restful]
exchange = Bittrex
instmt_name = BTCPKB
instmt_code = BTC-PKB
enabled = 1
[Bittrex-BTCPPC-Restful]
exchange = Bittrex
instmt_name = BTCPPC
instmt_code = BTC-PPC
enabled = 1
[Bittrex-BTCPTC-Restful]
exchange = Bittrex
instmt_name = BTCPTC
instmt_code = BTC-PTC
enabled = 1
[Bittrex-BTCQRL-Restful]
exchange = Bittrex
instmt_name = BTCQRL
instmt_code = BTC-QRL
enabled = 1
[Bittrex-BTCRADS-Restful]
exchange = Bittrex
instmt_name = BTCRADS
instmt_code = BTC-RADS
enabled = 1
[Bittrex-BTCSC-Restful]
exchange = Bittrex
instmt_name = BTCSC
instmt_code = BTC-SC
enabled = 1
[Bittrex-BTCSIB-Restful]
exchange = Bittrex
instmt_name = BTCSIB
instmt_code = BTC-SIB
enabled = 1
[Bittrex-BTCSJCX-Restful]
exchange = Bittrex
instmt_name = BTCSJCX
instmt_code = BTC-SJCX
enabled = 1
[Bittrex-BTCSNRG-Restful]
exchange = Bittrex
instmt_name = BTCSNRG
instmt_code = BTC-SNRG
enabled = 1
[Bittrex-BTCSWIFT-Restful]
exchange = Bittrex
instmt_name = BTCSWIFT
instmt_code = BTC-SWIFT
enabled = 1
[Bittrex-BTCTX-Restful]
exchange = Bittrex
instmt_name = BTCTX
instmt_code = BTC-TX
enabled = 1
[Bittrex-BTCVIA-Restful]
exchange = Bittrex
instmt_name = BTCVIA
instmt_code = BTC-VIA
enabled = 1
[Bittrex-BTCVOX-Restful]
exchange = Bittrex
instmt_name = BTCVOX
instmt_code = BTC-VOX
enabled = 1
[Bittrex-BTCWINGS-Restful]
exchange = Bittrex
instmt_name = BTCWINGS
instmt_code = BTC-WINGS
enabled = 1
[Bittrex-BTCXEM-Restful]
exchange = Bittrex
instmt_name = BTCXEM
instmt_code = BTC-XEM
enabled = 1
[Bittrex-BTCXLM-Restful]
exchange = Bittrex
instmt_name = BTCXLM
instmt_code = BTC-XLM
enabled = 1
[Bittrex-BTCXMR-Restful]
exchange = Bittrex
instmt_name = BTCXMR
instmt_code = BTC-XMR
enabled = 1
[Bittrex-BTCXRP-Restful]
exchange = Bittrex
instmt_name = BTCXRP
instmt_code = BTC-XRP
enabled = 1
[Bittrex-BTCXVC-Restful]
exchange = Bittrex
instmt_name = BTCXVC
instmt_code = BTC-XVC
enabled = 1
[Bittrex-BTCXWC-Restful]
exchange = Bittrex
instmt_name = BTCXWC
instmt_code = BTC-XWC
enabled = 1
[Bittrex-BTCZEN-Restful]
exchange = Bittrex
instmt_name = BTCZEN
instmt_code = BTC-ZEN
enabled = 1
[Bittrex-ETHCRB-Restful]
exchange = Bittrex
instmt_name = ETHCRB
instmt_code = ETH-CRB
enabled = 1
[Bittrex-ETHGNO-Restful]
exchange = Bittrex
instmt_name = ETHGNO
instmt_code = ETH-GNO
enabled = 1
[Bittrex-ETHGUP-Restful]
exchange = Bittrex
instmt_name = ETHGUP
instmt_code = ETH-GUP
enabled = 1
[Bittrex-ETHQRL-Restful]
exchange = Bittrex
instmt_name = ETHQRL
instmt_code = ETH-QRL
enabled = 1
[Bittrex-ETHRLC-Restful]
exchange = Bittrex
instmt_name = ETHRLC
instmt_code = ETH-RLC
enabled = 1
[Bittrex-ETHTRST-Restful]
exchange = Bittrex
instmt_name = ETHTRST
instmt_code = ETH-TRST
enabled = 1

The follow error is generated after starting bitcoinexchangeFH:

Exception in thread Thread-2:
Traceback (most recent call last):
File "/home/obnauticus/.local/lib/python3.5/site-packages/befh/kdbplus_client.py", line 217, in insert
self.conn.sync(command)
File "/home/obnauticus/.local/lib/python3.5/site-packages/qpython/qconnection.py", line 303, in sync
response = self.receive(data_only = False, **options)
File "/home/obnauticus/.local/lib/python3.5/site-packages/qpython/qconnection.py", line 380, in receive
result = self._reader.read(**self._options.union_dict(**options))
File "/home/obnauticus/.local/lib/python3.5/site-packages/qpython/qreader.py", line 139, in read
message.data = self.read_data(message.size, message.is_compressed, **options)
File "/home/obnauticus/.local/lib/python3.5/site-packages/qpython/qreader.py", line 216, in read_data
return raw_data if self._options.raw else self._read_object()
File "/home/obnauticus/.local/lib/python3.5/site-packages/qpython/qreader.py", line 225, in _read_object
return reader(self, qtype)
File "/home/obnauticus/.local/lib/python3.5/site-packages/qpython/qreader.py", line 240, in _read_error
raise QException(self._read_symbol())
qpython.qtype.QException: b'insert'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/home/obnauticus/.local/lib/python3.5/site-packages/befh/exch_bittrex.py", line 236, in get_trades_worker
self.insert_trade(instmt, trade)
File "/home/obnauticus/.local/lib/python3.5/site-packages/befh/exchange.py", line 179, in insert_trade
is_commit=True)
File "/home/obnauticus/.local/lib/python3.5/site-packages/befh/kdbplus_client.py", line 219, in insert
raise Exception("Error in running insert statement (%s).\n%s" % (command, e))
Exception: Error in running insert statement (`exch_bittrex_btc1st_snapshot_20170622 insert (1296; 0.00068; 7.08377613; 0.0006782; 0.000678; 0.00067116; 0.00067; 0.00066199; 0.00068; 0.00068885; 0.00068886; 0.00068887; 0.0006931; 234.68564114; 61.47389469; 12.48536861; 25.52632037; 16.99421441; 209.68084468; 130.88266889; 57.17452034; 8.63; 19.22753368; 2017.06.22D14:11:39.304126000; 2017.06.22D14:11:39.544221000; 2)).
b'insert'

2017-06-22 15:11:41,584 - ERROR
[ExchGwBittrex]
Error in order book: Error in running insert statement (`exch_bittrex_btcgno_snapshot_20170622 insert (1308; 0.084; 0.04219615; 0.08219567; 0.08219566; 0.08219564; 0.08219561; 0.08219556; 0.08399999; 0.084; 0.08440694; 0.08440818; 0.08555794; 7.599; 5.97443015; 26.759; 0.43233853; 0.28534343; 26.82432278; 7.0; 2.46357776; 0.19752483; 7.459; 2017.06.22D14:11:41.581279000; 2017.06.22D14:09:31.360276000; 1)).
b'insert'

2017-06-22 15:11:43,758 - ERROR
[ExchGwBittrex]
Error in order book: Error in running insert statement (`exch_bittrex_btcfct_snapshot_20170622 insert (1321; 0.01168231; 0.76617289; 0.011521; 0.01152084; 0.01152018; 0.01152017; 0.01152005; 0.01168133; 0.0116815; 0.0116823; 0.01168231; 0.01168232; 0.07981868; 0.07981868; 263.3046; 590.16; 8.54815189; 0.08990821; 0.08972119; 0.090133; 1.53334305; 25.4497504; 2017.06.22D14:11:43.740125000; 2017.06.22D14:10:26.297787000; 1)).
b'insert'

I'm not exactly sure why, but new data doesn't seem to be streaming into the KDB+. Also, not all tickers in subscriptions.ini are being updated in resources.ini. Only about half are being created. Here's a screenshot of the tables which are created when I run BitcoinExchangeFH:

http://i.imgur.com/ZFZSM5z.png

Here's a full output log:

http://sprunge.us/YDEg

looking to have quote screen for users pulling from updates to mysql database..can you help? any skype?

the project is written in php so far
please email me at [email protected]

It seems that Quoine has quite a restrictive policy for anonymous querying, 4s interval is definitely too short and almost all queries fail with

HTTP Error 429: Too Many Requests

can we connect to the azure data lake ?

I want to move the data to a bigger disk so I can keep recording for a long period of time. Is it possible to move the sqlite and CSV files to a new location without interrupting?

The only solution I can think of is to start another instance in the new location then merge the files later.

Cheers

Handle the signals

SIGINT
SIGTERM

to exit cleanly.

To exit cleanly, the threads should be killed one by one.

Reference: https://pymotw.com/2/signal/

Currently some exchanges publish a bulk of trades in the first RESTful API query. There is no way to identify the trades are from recovery rather than normal trading.

Resolve it by filtering it out or adding a flag to indicate it.

Due to endless issues on socketio-client on the old versions.

This library did not work for me in python 2.7: 'module' object has no attribute 'Request'
In ./befh/restful_api_socket.py you try to import urllib as urlrequest but it seems that urllib does not have any Request attribute.

The python3 version worked.

Support feed Quoine

Every row has a trade price and a 5-level order book. Is this simply a record of every trade and the order book at that time? Does that mean it doesn't create a new record every time the order book (most importantly, the top bid/ask) changes but waits for a trade to happen first?

Thank you, I think it's a good idea to add that in the readme file. Thanks for your work tho, I think this is a really good project with a simple 5 minutes setup.

According to exchange.py:61, ZeroMQ is not allowed to handle instmt records, resulting only in exchange snapshots. Is there any reason for that?

The socket was not reconnected occasionally.

2017-01-04 20:30:42,222 - INFO
[ExchGwBitmexWs]
Socket error:
Connection is already closed.

2017-01-04 20:30:42,223 - INFO
[ExchGwBitmexWs]
Socket is closed.

2017-01-04 20:30:42,223 - INFO
[ExchGwBitmex]
Instrument XBTUSD is unsubscribed in channel BitMEX

2017-01-04 20:30:42,223 - INFO
[ExchGwBitmex]
Instrument XBTH17 is unsubscribed in channel BitMEX

2017-01-04 20:47:37,913 - ERROR
[ExchGwBtccFuture]
Error in order book: HTTP Error 504: GATEWAY_TIMEOUT

Some prices obtained from Kraken look strange:

q)select maxbid: max b1, minask: min a1 from exch_kraken_xbtusd_book
maxbid   minask 
----------------
1156.671 1155.14

q)select maxprice: max trade_price, minprice: min trade_price from exch_kraken_xbtusd_trades
maxprice minprice
-----------------
1198.801 122     
q)

Currently all timestamps are populated as

1. Exchange given timestamp if it is provided
2. Local timestamp if not

Set a configuration item so that the user can choose to

1. Remain the original schema
2. Record all timestamps in local timestamp

It is for backtesting arbitrage or some other strategies related to cross exchanges.

Compiling and installation of both BitcoinExchangeFH and ZeroGQ is a smith process. However I have issues with connection to ZeroMQ. Linking of the Python binding seems to work.
From the README.md it looks like ZeroMQ should be running as a free standing server

(bitcoinexchangefh -zmq -zmqdest "tcp://localhost:6001" -instmts subscription.ini)

With the above options I got this stack backtrace:

Zmq client is connecting to tcp://localhost:6001
Traceback (most recent call last):
  File "/usr/bin/bitcoinexchangefh", line 9, in <module>
    load_entry_point('BitcoinExchangeFH==0.1.3rc3', 'console_scripts', 'bitcoinexchangefh')()
  File "/usr/lib/python3.4/site-packages/befh/bitcoinexchangefh.py", line 93, in main
    db_client.connect(addr=args.zmqdest)
  File "/usr/lib/python3.4/site-packages/befh/zmq_client.py", line 28, in connect
    self.conn.bind(addr)
  File "zmq/backend/cython/socket.pyx", line 495, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:5653)
    # sockopts will still raise just the same, but it will be libzmq doing
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:10014)
    raise ZMQError(errno)
zmq.error.ZMQError: No such device

The documentation seems to lack some information on this part.
I hope for some information that shed some light upon this.
Runnung the nodejs client/server combo works without a hitch.
I am not a Pythom programmer, BTW

Thanks!

Added exchange gatecoin

Currently SQL is executed at every commit and file flushes at every insertion. It is observed during early start up of the process, it spent a lot time to processing all the recovery trades.

The change is to optimize the execution/flushing mechanism.

Hi. Thanks for this awesome project!
Is there any way to change the interval of when market data is pulled, say from Bittrex?
I was looking for something like getting json data say every 1 or 5 min.
Thanks!

In host machines, there are spam of error messages on Bitfinex gateway. Need to raise the log level to investigate the root cause.

Hello,
I tried running BitcoinExchangeFH with multiple exchanges as a publisher to ZeroMQ socket on windows 7 and Ubuntu 16.04.2 AWS EC2. See my simple ZeroMQ subscriber client below.
The market data looks OK on windows, but it seems to be completely jumbled on Ubuntu.
See examples below.

What could be the reason for this behavior?

Examples:

Windows:

LEONIDS@LEONIDS03 MINGW64 ~/Envs/BitcoinExchangeFH$ head -n 5 zmq_sub_client.win.log; tail -n 5 zmq_sub_client.win.log
Getting market feed...
Started...
b'{"exchange":"BitMEX","instmt":"XBTUSD","trade_px":0.0,"trade_volume":0.0,"b1":2609.1,"b2":2609.0,"b3":2608.9,"b4":2608.6,"b5":2608.3,"a1":2609.2,"a2":2610.0,"a3":2610.1,"a4":2610.7,"a5":2610.9,"bq1":45361.0,"bq2":270.0,"bq3":5000.0,"bq4":5000.0,"bq5":6000.0,"aq1":1260.0,"aq2":100.0,"aq3":8.0,"aq4":32.0,"aq5":96.0,"order_date_time":"20170705 21:59:57.368267","trades_date_time":"20000101 00:00:00.000000","update_type":1,"table":"exchanges_snapshot"}'
b'{"exchange":"BitMEX","instmt":"XBTUSD","trade_px":2609.2,"trade_volume":974,"b1":2609.1,"b2":2609.0,"b3":2608.9,"b4":2608.6,"b5":2608.3,"a1":2609.2,"a2":2610.0,"a3":2610.1,"a4":2610.7,"a5":2610.9,"bq1":45361.0,"bq2":270.0,"bq3":5000.0,"bq4":5000.0,"bq5":6000.0,"aq1":1260.0,"aq2":100.0,"aq3":8.0,"aq4":32.0,"aq5":96.0,"order_date_time":"20170705 21:59:57.368267","trades_date_time":"20170705 21:59:57.399267","update_type":2,"table":"exchanges_snapshot"}'
b'{"exchange":"BitMEX","instmt":"XBTUSD","trade_px":2609.2,"trade_volume":560,"b1":2609.1,"b2":2609.0,"b3":2608.9,"b4":2608.6,"b5":2608.3,"a1":2609.2,"a2":2610.0,"a3":2610.1,"a4":2610.7,"a5":2610.9,"bq1":45361.0,"bq2":270.0,"bq3":5000.0,"bq4":5000.0,"bq5":6000.0,"aq1":1260.0,"aq2":100.0,"aq3":8.0,"aq4":32.0,"aq5":96.0,"order_date_time":"20170705 21:59:57.368267","trades_date_time":"20170705 21:59:57.512267","update_type":2,"table":"exchanges_snapshot"}'
b'{"exchange":"Bitstamp","instmt":"BTCUSD","trade_px":2610.01,"trade_volume":0.07388939,"b1":2610.01,"b2":2610.0,"b3":2605.0,"b4":2603.65,"b5":2603.63,"a1":2611.6,"a2":2611.89,"a3":2611.95,"a4":2612.12,"a5":2612.3,"bq1":4.76708818,"bq2":22.22829399,"bq3":0.0022,"bq4":1.128,"bq5":3.8992,"aq1":0.241,"aq2":0.479,"aq3":2.3939244,"aq4":0.15518748,"aq5":7.09836726,"order_date_time":"20170705 22:00:23.408267","trades_date_time":"20170705 22:00:24.026267","update_type":2,"table":"exchanges_snapshot"}'
b'{"exchange":"Bitstamp","instmt":"BTCUSD","trade_px":2610.01,"trade_volume":0.07388939,"b1":2610.01,"b2":2610.0,"b3":2605.0,"b4":2603.65,"b5":2603.63,"a1":2611.6,"a2":2611.89,"a3":2612.12,"a4":2612.29,"a5":2612.3,"bq1":4.39451152,"bq2":22.22829399,"bq3":0.0022,"bq4":1.128,"bq5":3.8992,"aq1":0.241,"aq2":0.479,"aq3":0.15518748,"aq4":4.236222,"aq5":7.09836726,"order_date_time":"20170705 22:00:24.402267","trades_date_time":"20170705 22:00:24.026267","update_type":1,"table":"exchanges_snapshot"}'
b'{"exchange":"BitMEX","instmt":"XBTUSD","trade_px":2611,"trade_volume":20,"b1":2610.9,"b2":2610.7,"b3":2610.6,"b4":2610.5,"b5":2610.3,"a1":2611.0,"a2":2611.2,"a3":2611.4,"a4":2612.0,"a5":2612.2,"bq1":33811.0,"bq2":184.0,"bq3":5000.0,"bq4":5000.0,"bq5":150.0,"aq1":5363.0,"aq2":2150.0,"aq3":10000.0,"aq4":2000.0,"aq5":16.0,"order_date_time":"20170705 22:00:24.404267","trades_date_time":"20170705 22:00:09.463267","update_type":1,"table":"exchanges_snapshot"}'
b'{"exchange":"Gdax","instmt":"BTCUSD","trade_px":0.0,"trade_volume":0.0,"b1":2615.0,"b2":2614.76,"b3":2614.71,"b4":2614.2,"b5":2613.92,"a1":2615.09,"a2":2615.44,"a3":2615.59,"a4":2615.63,"a5":2615.96,"bq1":0.48,"bq2":1.91051,"bq3":0.10065071,"bq4":0.13,"bq5":0.01,"aq1":2.72,"aq2":0.01,"aq3":0.01,"aq4":0.1,"aq5":0.1,"order_date_time":"20170705 22:00:24.420267","trades_date_time":"20000101 00:00:00.000000","update_type":1,"table":"exchanges_snapshot"}'
b'{"exchange":"OkCoin","instmt":"SPOT_BTCUSD","trade_px":2697.24,"trade_volume":0.073,"b1":2702.84,"b2":2696.38,"b3":2695.69,"b4":2695.66,"b5":2693.01,"a1":2705.0,"a2":2709.27,"a3":2710.55,"a4":2712.53,"a5":2714.99,"bq1":0.036,"bq2":2.0,"bq3":0.05,"bq4":2.576,"bq5":0.04,"aq1":9.945,"aq2":0.02,"aq3":0.2,"aq4":0.012,"aq5":0.5,"order_date_time":"20170705 22:00:24.731267","trades_date_time":"20170705 22:00:00.616267","update_type":1,"table":"exchanges_snapshot"}'
LEONIDS@LEONIDS03 MINGW64 ~/Envs/BitcoinExchangeFH$

Ubuntu:

(BitcoinExchangeFH) ubuntu@ip-172-31-40-58:/home/ubuntu/envs/BitcoinExchangeFH$ head -n 5 zmq_sub_client.log; tail -n 5 zmq_sub_client.log
Getting market feed...
Started...
b'{"b4":2556.88,"a2":2558.57,"bq5":1.9,"trades_date_time":"20170705 08:32:47.084933","b1":2557.68,"aq3":0.01,"bq4":0.8,"bq1":0.01,"a5":2559.96,"trade_px":2558.5,"b2":2556.92,"a4":2559.66,"trade_volume":0.35015176,"update_type":1,"b3":2556.91,"a1":2558.5,"aq5":0.01,"exchange":"Gdax","aq4":1.0,"b5":2556.56,"a3":2559.2,"aq2":0.915188,"bq2":0.0081,"bq3":0.6276,"table":"exchanges_snapshot","order_date_time":"20170705 08:32:48.308532","instmt":"BTCUSD","aq1":6.97541503}'
b'{"b4":2538.77,"a2":2541.65,"bq5":4.22,"trades_date_time":"20170705 08:32:48.320541","b1":2538.9,"aq3":6.24038635,"bq4":4.3,"bq1":0.376,"a5":2544.0,"trade_px":2541.65,"b2":2538.88,"a4":2543.02,"trade_volume":0.09903745,"update_type":2,"b3":2538.87,"a1":2541.44,"aq5":1.0,"exchange":"Bitstamp","aq4":0.12345676,"b5":2537.89,"a3":2541.8,"aq2":6.24038635,"bq2":4.072297,"bq3":2.81107191,"table":"exchanges_snapshot","order_date_time":"20170705 08:32:47.275917","instmt":"BTCUSD","aq1":0.12147692}'
b'{"b4":2512.5,"a2":2515,"bq5":0.01,"trades_date_time":"20170705 08:32:45.470112","b1":2512.9,"aq3":1.2,"bq4":0.15,"bq1":6.19322842,"a5":2517.1,"trade_px":2513,"b2":2512.8,"a4":2516.2,"trade_volume":0.18464675,"update_type":1,"b3":2512.6,"a1":2513,"aq5":1.713,"exchange":"Bitfinex","aq4":0.01,"b5":2512,"a3":2515.5,"aq2":0.05,"bq2":1.2,"bq3":1.1872589,"table":"exchanges_snapshot","order_date_time":"20170705 08:32:48.414512","instmt":"BTCUSD","aq1":3.93626483}'
b'{"b4":2557.68,"a2":2561.31,"bq5":0.06,"trades_date_time":"20170705 08:33:50.242615","b1":2558.46,"aq3":3.45026,"bq4":0.8521,"bq1":0.01,"a5":2561.48,"trade_px":2558.51,"b2":2558.45,"a4":2561.37,"trade_volume":0.01,"update_type":1,"b3":2558.44,"a1":2558.51,"aq5":0.01,"exchange":"Gdax","aq4":0.5,"b5":2557.67,"a3":2561.34,"aq2":1.1,"bq2":0.01,"bq3":0.01,"table":"exchanges_snapshot","order_date_time":"20170705 08:33:53.966488","instmt":"BTCUSD","aq1":8.0}'
b'{"b4":2502.00000001,"a2":2509.0,"bq5":0.44963128,"trades_date_time":"20170705 08:33:53.582717","b1":2505.00003594,"aq3":3.88908198,"bq4":0.05150097,"bq1":0.03952096,"a5":2515.0,"trade_px":2507.0,"b2":2505.0,"a4":2514.99999999,"trade_volume":0.04,"update_type":1,"b3":2502.00000005,"a1":2507.0,"aq5":5.29567228,"exchange":"Bittrex","aq4":0.98411746,"b5":2502.0,"a3":2510.0,"aq2":0.88671901,"bq2":0.182,"bq3":1.751,"table":"exchanges_snapshot","order_date_time":"20170705 08:33:53.968146","instmt":"USDTBTC","aq1":0.02977037}'
b'{"b4":2610.5,"a2":2628.0,"bq5":2.0,"trades_date_time":"20170705 08:33:37.421357","b1":2610.78,"aq3":0.684,"bq4":0.3,"bq1":0.5,"a5":2631.95,"trade_px":2615.95,"b2":2610.77,"a4":2630.45,"trade_volume":0.1,"update_type":1,"b3":2610.75,"a1":2622.0,"aq5":0.02,"exchange":"OkCoin","aq4":1.943,"b5":2610.49,"a3":2630.42,"aq2":0.025,"bq2":0.6,"bq3":0.3,"table":"exchanges_snapshot","order_date_time":"20170705 08:33:54.028874","instmt":"SPOT_BTCUSD","aq1":0.09}'
b'{"b4":2521.59795,"a2":2563.05677,"bq5":2.14241,"trades_date_time":"20000101 00:00:00.000000","b1":2548.74,"aq3":0.14241,"bq4":0.61241,"bq1":4.87,"a5":2577.44969,"trade_px":0.0,"b2":2542.12,"a4":2577.44968,"trade_volume":0.0,"update_type":1,"b3":2535.83802,"a1":2548.74422,"aq5":0.19241,"exchange":"Quoine","aq4":15.0,"b5":2513.231,"a3":2571.4663,"aq2":0.11241,"bq2":9.75,"bq3":0.10241,"table":"exchanges_snapshot","order_date_time":"20170705 08:33:54.066095","instmt":"BTCUSD","aq1":0.10241}'
b'{"b4":2538.77,"a2":2541.88,"bq5":4.22,"trades_date_time":"20170705 08:33:10.693341","b1":2540.5,"aq3":6.24038635,"bq4":4.3,"bq1":2.91127733,"a5":2544.0,"trade_px":2538.98,"b2":2540.45,"a4":2543.02,"trade_volume":0.1,"update_type":1,"b3":2538.79,"a1":2541.84,"aq5":1.0,"exchange":"Bitstamp","aq4":0.12345676,"b5":2537.89,"a3":2542.62,"aq2":6.24038635,"bq2":4.11,"bq3":0.376,"table":"exchanges_snapshot","order_date_time":"20170705 08:33:54.273394","instmt":"BTCUSD","aq1":6.24038635}'
(BitcoinExchangeFH) ubuntu@ip-172-31-40-58:/home/ubuntu/envs/BitcoinExchangeFH$

Another run:

(BitcoinExchangeFH) ubuntu@ip-172-31-40-58:/home/ubuntu/envs/BitcoinExchangeFH$ head -n 5 zmq_sub_client.01.log; tail -n 5 zmq_sub_client.01.log
Getting market feed...
Started...
b'{"bq5":0.16310297,"instmt":"USDTBTC","order_date_time":"20170705 04:36:47.688285","aq2":1.63,"trade_px":0.0,"b3":2526.0,"a3":2538.0,"trades_date_time":"20000101 00:00:00.000000","bq1":1.44395489,"a4":2538.55,"b5":2525.98,"b1":2533.2,"bq3":0.81693282,"aq4":0.09872762,"b2":2526.08,"a1":2537.99999998,"bq4":0.46431003,"exchange":"Bittrex","trade_volume":0.0,"a2":2537.99999999,"table":"exchanges_snapshot","aq3":0.1,"update_type":1,"a5":2539.86,"aq1":3.11e-06,"b4":2525.99,"aq5":0.05,"bq2":0.3491967}'
b'{"bq5":5.0,"instmt":"BTCUSD","order_date_time":"20170705 04:36:48.086726","aq2":0.3,"trade_px":0.0,"b3":2555.5,"a3":2658.54046222,"trades_date_time":"20000101 00:00:00.000000","bq1":1.63005649,"a4":2674.9899,"b5":2539.7893,"b1":2570.0,"bq3":0.4877743,"aq4":1.0838,"b2":2556.0,"a1":2627.2,"bq4":0.51847133,"exchange":"Gatecoin","trade_volume":0.0,"a2":2640.0,"table":"exchanges_snapshot","aq3":0.01,"update_type":1,"a5":2674.99,"aq1":0.02,"b4":2550.0,"aq5":0.04,"bq2":0.51}'
b'{"bq5":0.01,"instmt":"BTCUSD","order_date_time":"20170705 04:36:48.478696","aq2":0.0145,"trade_px":0.0,"b3":2569.0,"a3":2569.77,"trades_date_time":"20000101 00:00:00.000000","bq1":2.899,"a4":2569.78,"b5":2568.32,"b1":2569.09,"bq3":3.12,"aq4":0.01,"b2":2569.08,"a1":2569.1,"bq4":0.02,"exchange":"Gdax","trade_volume":0.0,"a2":2569.12,"table":"exchanges_snapshot","aq3":0.01,"update_type":1,"a5":2569.79,"aq1":2.1481,"b4":2568.8,"aq5":0.01,"bq2":0.11}'
b'{"trades_date_time":"20170705 08:26:45.883629","b4":2514,"b2":2515.3,"instmt":"BTCUSD","aq4":1.3,"bq4":0.4,"bq1":2.20351999,"trade_px":2515.4,"order_date_time":"20170705 08:26:49.174633","aq5":0.09,"a5":2518,"table":"exchanges_snapshot","a2":2515.6,"aq2":9,"update_type":1,"bq5":1.01,"a1":2515.5,"b3":2514.7,"b5":2512.6,"aq1":2.07,"trade_volume":0.422837,"bq2":0.01,"a4":2517.2,"bq3":0.01,"aq3":0.329,"exchange":"Bitfinex","a3":2516.9,"b1":2515.4}'
b'{"trades_date_time":"20170705 08:26:45.883629","b4":2514,"b2":2515.3,"instmt":"BTCUSD","aq4":1.3,"bq4":0.4,"bq1":2.25738695,"trade_px":2515.4,"order_date_time":"20170705 08:26:49.554109","aq5":0.09,"a5":2518,"table":"exchanges_snapshot","a2":2515.6,"aq2":9,"update_type":1,"bq5":1.01,"a1":2515.5,"b3":2514.7,"b5":2512.6,"aq1":2.07,"trade_volume":0.422837,"bq2":0.01,"a4":2517.2,"bq3":0.01,"aq3":0.329,"exchange":"Bitfinex","a3":2516.9,"b1":2515.4}'
b'{"trades_date_time":"20170705 08:26:34.726610","b4":2537.63,"b2":2539.47,"instmt":"BTCUSD","aq4":1.128,"bq4":1.504,"bq1":1.128,"trade_px":2544.5,"order_date_time":"20170705 08:26:49.768011","aq5":4.4,"a5":2548.27,"table":"exchanges_snapshot","a2":2544.5,"aq2":5.02365222,"update_type":1,"bq5":4.076,"a1":2544.4,"b3":2538.91,"b5":2537.58,"aq1":6.24038635,"trade_volume":1.21673413,"bq2":4.79,"a4":2548.03,"bq3":4.45008538,"aq3":0.37037031,"exchange":"Bitstamp","a3":2547.67,"b1":2539.55}'
b'{"trades_date_time":"20170705 08:26:49.809470","b4":2512.0,"b2":2512.0222,"instmt":"USDTBTC","aq4":2.46244838,"bq4":0.05205908,"bq1":0.67769556,"trade_px":2513.001,"order_date_time":"20170705 08:26:42.948505","aq5":0.00028219,"a5":2516.7548,"table":"exchanges_snapshot","a2":2514.25405,"aq2":0.00028247,"update_type":2,"bq5":0.5277909,"a1":2513.5,"b3":2512.022,"b5":2511.99999993,"aq1":0.3439122,"trade_volume":0.01661362,"bq2":2.0,"a4":2516.34150594,"bq3":2.0,"aq3":0.2,"exchange":"Poloniex","a3":2516.0,"b1":2513.001}'
b'{"trades_date_time":"20170705 08:26:48.573898","b4":2609.09,"b2":2611.06,"instmt":"SPOT_BTCUSD","aq4":1.948,"bq4":0.5,"bq1":0.487,"trade_px":2623.17,"order_date_time":"20170705 08:26:49.836235","aq5":1.44,"a5":2628.49,"table":"exchanges_snapshot","a2":2623.17,"aq2":0.112,"update_type":1,"bq5":0.3,"a1":2620.0,"b3":2611.0,"b5":2609.08,"aq1":5.0,"trade_volume":0.01,"bq2":0.3,"a4":2624.97,"bq3":0.076,"aq3":1.2,"exchange":"OkCoin","a3":2624.53,"b1":2617.15}'
(BitcoinExchangeFH) ubuntu@ip-172-31-40-58:/home/ubuntu/envs/BitcoinExchangeFH$

Subscriber client code:

(BitcoinExchangeFH) ubuntu@ip-172-31-40-58:/home/ubuntu/envs/BitcoinExchangeFH$ cat zmq_sub_client.py

# http://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/patterns/pubsub.html
import sys
import zmq

port = "6001"
if len(sys.argv) > 1:
    port =  sys.argv[1]
    int(port)

# Socket to talk to server
context = zmq.Context()
socket = context.socket(zmq.SUB)

print("Getting market feed...")
socket.connect("tcp://127.0.0.1:%s" % port)

topicfilter = ""
socket.setsockopt_string(zmq.SUBSCRIBE, topicfilter)

print("Started...")
while True:
    ret = socket.recv()
    print(ret)

(BitcoinExchangeFH) ubuntu@ip-172-31-40-58:/home/ubuntu/envs/BitcoinExchangeFH$

Ubuntu configuration:

(BitcoinExchangeFH) ubuntu@ip-172-31-40-58:/home/ubuntu/envs/BitcoinExchangeFH$ python --version
Python 3.5.2
(BitcoinExchangeFH) ubuntu@ip-172-31-40-58:/home/ubuntu/envs/BitcoinExchangeFH$ pip --version
pip 9.0.1 from /home/ubuntu/envs/BitcoinExchangeFH/lib/python3.5/site-packages (python 3.5)
(BitcoinExchangeFH) ubuntu@ip-172-31-40-58:/home/ubuntu/envs/BitcoinExchangeFH$ pip freeze
BitcoinExchangeFH==0.1.3rc3
gevent==1.2.2
greenlet==0.4.12
numpy==1.13.0
PyMySQL==0.7.11
pyzmq==16.0.2
qPython==1.2.2
six==1.10.0
websocket==0.2.1
websocket-client==0.44.0
(BitcoinExchangeFH) ubuntu@ip-172-31-40-58:/home/ubuntu/envs/BitcoinExchangeFH$

ubuntu@ip-172-31-40-58:/home/ubuntu$ uname -a
Linux ip-172-31-40-58 4.4.0-1013-aws #22-Ubuntu SMP Fri Mar 31 15:41:31 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ip-172-31-40-58:/home/ubuntu$ sudo lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
ubuntu@ip-172-31-40-58:/home/ubuntu$

Currently, all the real time public order book updates are output into two tables:

• exch_..._order_book

• exch_..._trades

For quicker import of data for researchers, all the updates will be converted into a single table "exch_..._snapshots", which contains the following fields

• First 5 aggregated bid and ask prices and volume

• Last trade price and volume

• Last order book and trade update time

• Update type

Currently the port number input to mysql client is ignored.

Support multiple sinkers to write in different databases or PUB/SUB channel.

Currently, the application cannot do the two events - publishing the feed and storing data into the database, at the same time.

It is hard to trace back issue in trading system.

Hi @gavincyi,

You are developing an awesome system, great work!
I am looking forward to put your code to run 24/7 on some of my servers.

How can i to get a longer order book? I saw that some API can provide 100 records depth.

Is the actual orderbook table model good for this size of orderbook?

Thanks for the help 👍