mestrace / orderbook Goto Github PK

Currently the performance of GetExchangeOrderBookAll is bad. Usually takes around 1 minutes to query all.

• Setup prometheus and grafana to monitor the time consumption and QPS

Improvements

• Concurrently query for symbols
• Rate limit the calls to blockchain.com apis

Cache results for each symbol as we need to query for each symbol.

Redis

• Cache symbol list ob:{{exchange_id}}:sl (order_book:{{exchange_id}}:symbol_list).
• Cache symbol items ob:{{exchange_id}}:si:{{symbol}}:a/b (order_book:{{exchange_id}}:symbol_item:{{symbol}}:ask/bid)
• Wrap OrderBookDAO with cache for query symbol list and symbol item order books.

Stretch Goal

• Local cache, can take the cache all at once
  • an data refresher thread that does that on the background
  • consider using lockfree data structure, can benchmark if have time.
• Should implement cache hit / miss rate #6

Add api level integration testing for all three endpoints.

• GetExchangeOrderBook
  • Can get 1 symbol, can filter symbol, can compute correctly
• GetExchangeOrderBookAll
  • Can get all symbol list, can filter asks and bid, can sort
• UpdateExchangeMetadata
  • Can process csv files
  • Returning correct error when error cases

The default prometheus handler from hertz provides little experiences.

• Override the prometheus handler from hertz to provide end-point name
• Monitor the calls to external services including external apis, database and redis
• Cache hit / miss rate.

Stretch Goal

• Container utility (CPU, Mem) (if containerisation)
• Infra utility (CPU, Mem) (need to customize db?)

Currently the api is returning the raw error code from each of the packages and some dynamic error code to be used. Need to have a standardised error code system.

• for each of the relied 3pp errors, split into 400 and 500 error codes, and use appropriate biz_code and err messages for the api.
• use static errors to replace or wrap the dynamic errors.