Endless warming up about crypto-rl HOT 5 CLOSED

I'll look into this issue and try to re-create it on side. Can you please confirm that you're using the latest version of the repo?

from crypto-rl.

Thank you for the quick answer!
Yes, I am using the latest version.

from crypto-rl.

I have looked into this problem a little bit. Just some information, but maybe helpful.
I exported the ticks from db (2020-07-08 11:00:00.054661+00:00 - 2020-07-08 15:59:59.983510+00:00). It is a 780MB csv file, I can upload somewhere if it needed.

grep 'load_book' test.csv returns only one row

2020-07-08 15:51:09.993000+02:00,BTC-USD,,15141105974.0,,,,,,,,,,,,,load_book,BTC-USD,2020-07-08 13:51:09.993612+00:00

There is a trace of this in the logs:

[2020-07-08 15:50:48,493] Coinbase - BTC-USD is warming up
[2020-07-08 15:50:53,493] Coinbase - BTC-USD is warming up
[2020-07-08 15:50:58,494] Coinbase - BTC-USD is warming up
[2020-07-08 15:51:03,494] Coinbase - BTC-USD is warming up
[2020-07-08 15:51:06,559] coinbase: subscription exception WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason
[2020-07-08 15:51:06,559] coinbase: Retrying to connect... attempted #5
[2020-07-08 15:51:07,214] Requesting Book: {"type": "subscribe", "product_ids": ["BTC-USD"], "channels": ["full"]}
[2020-07-08 15:51:07,214] BOOK COINBASE: BTC-USD subscription request sent.
[2020-07-08 15:51:07,330] Coinbase Subscriptions successful for : BTC-USD
[2020-07-08 15:51:07,330] BTC-USD get_book request made.
[2020-07-08 15:51:08,563] BTC-USD's order book cleared.
[2020-07-08 15:51:09,993] BTC-USD get_book request completed in 2.662830 seconds.
[2020-07-08 15:51:19,409] BTC-USD added 100000 msgs to Arctic
[2020-07-08 15:51:20,399] BTC-USD: book loaded................in 10.405931 seconds
[2020-07-08 15:51:20,399] BTC-USD [done] has a stale tick: current 15141105974 | incoming 15141104655
[2020-07-08 15:51:20,400] BTC-USD [received] has a stale tick: current 15141105974 | incoming 15141104656
[2020-07-08 15:51:20,400] BTC-USD [open] has a stale tick: current 15141105974 | incoming 15141104657
[2020-07-08 15:51:20,400] BTC-USD [received] has a stale tick: current 15141105974 | incoming 15141104658
[2020-07-08 15:51:20,400] BTC-USD [open] has a stale tick: current 15141105974 | incoming 15141104659

After a bunch of stale tick there is again

[2020-07-08 15:51:20,744] BTC-USD [open] has a stale tick: current 15141105974 | incoming 15141105974
[2020-07-08 15:51:23,583] Coinbase - BTC-USD is warming up
[2020-07-08 15:51:28,584] Coinbase - BTC-USD is warming up
[2020-07-08 15:51:33,584] Coinbase - BTC-USD is warming up
[2020-07-08 15:51:38,586] Coinbase - BTC-USD is warming up
[2020-07-08 15:51:43,587] Coinbase - BTC-USD is warming up
[2020-07-08 15:51:48,588] Coinbase - BTC-USD is warming up
[2020-07-08 15:51:53,587] Coinbase - BTC-USD is warming up

1. 2020-07-08 15:23:25,345 there is a Exception in thread tBTCUSD (bitfinex), The reconnect is not successful, there is no reload_book tick entry. After that there should not be a connection to bitfinex, only to Coinbase.

2. After that the log is full with Coinbase - BTC-USD is warming up

3. But it seems there is no problem, the db still receives ticks.
   

4. 2020-07-08 15:51:06,559 a coinbase restart with a reload_book tick entry.

5. the db still receives ticks
   

6. The log still shows Coinbase - BTC-USD is warming up

Idk, maybe I should not worry about the log, until there is data coming in. BTW is there any indicator which tells where from which exchange the tick came from?

from crypto-rl.

Hi @epyx25 ,

Thanks for sharing your logs and details.

Re: endless warming up deep-dive

• It appears the LOGGER in recorder.py was printing a misleading statement, which caused confusion re: if the LOB is still warming up, although the DB was indeed still receiving and storing data. Moreover, this issue was due to the logger outputting the opposite exchange as 'warming up'.
• I've fixed this aesthetic bug in commit a1cdc2b0534c8a208e86f4013eb8161998d97ac6.

Re: point #1 - there is no load_book() call for Bitfinex, only Coinbase.

• This is due to the data feeds being different. Coinbase requires a REST GET to obtain the initial LOB snapshot, whereas Bitfinex sends their initial LOB snapshot upon a new WebSocket connection, hence why there is no load_book() call where the REST request is made.

Re: indicator for identifying exchange origination via tick

• Use the instrument name to identify which exchange the tick originated from (e.g., symbols that start with t belong to Bitfinex)

from crypto-rl.

Thanks for the commit, I am gonna try it out, and start to collect data as soon as I can.

from crypto-rl.