crypto-crawler / crypto-crawler-rs Goto Github PK

View Code? Open in Web Editor NEW

219.0 8.0 71.0 1.59 MB

A rock-solid cryptocurrency crawler library.

License: Apache License 2.0

Rust 99.75% C 0.25%

crawler cryptocurrency websocket

crypto-crawler-rs's Introduction

crypto-crawler-rs

A rock-solid cryprocurrency crawler.

Quickstart

Use the carbonbot binary to crawl data.

If you need more control and customization, use this library.

Architecture

crypto-crawler is the crawler library to crawl websocket and restful messages from exchanges
carbonbot is the main CLI tool to run crawlers.
crypto-ws-client is the underlying websocket client library, providing a set of universal APIs for different exchanges.
crypto-rest-client is the underlying RESTful client library, providing universal APIs to get public data from different exchanges.
crypto-markets is a RESTful library to retreive market meta data from cryptocurrency echanges.
crypto-client is a RESTful client library to place and cancel orders.
Support multiple languages. Some libraries support multiple languages, which is achieved by first providing a FFI binding, then a languge specific wrapper. For example, crypto-crawler provides a C-style FFI binding first, and then provides a Python wrapper and a C++ wrapper based on the FFI binding.

How to parse raw messages

Use the crypto-msg-parser library to parse raw messages.

Crawlers should always preserve the original data without any parsing.

crypto-crawler-rs's People

Contributors

Stargazers

Watchers

crypto-crawler-rs's Issues

Implement ftx.us

The https://ftx.us exchange is a new cryptocurrency exchange, which needs to be implemented.

Volunteers are always welcomed!

Implement binance.us

The https://www.binance.us exchange is a new cryptocurrency exchange, which needs to be implemented.

Volunteers are always welcomed!

Stack overflow on Hello World

rustc 1.60.0 (7737e0b5c 2022-04-04)

Window 11 10.0.22000 Build 22000

main.rs:

use crypto_crawler::{crawl_trade, MarketType};

#[tokio::main]
async fn main() {
    let (tx, rx) = std::sync::mpsc::channel();
    tokio::task::spawn(async move {
        for msg in rx {
            println!("{}", msg);
        }
    });

    crawl_trade("binance", MarketType::Spot, None, tx).await;
}

Cargo.toml:

[package]
name = "crypto_test"
version = "0.1.0"
edition = "2021"

[dependencies]
tokio = { version = "1", features = ["full"] }
crypto-crawler = "*"

> cargo run

thread 'main' has overflowed its stack
error: process didn't exit successfully: `target\debug\crypto_test.exe` (exit code: 0xc00000fd, STATUS_STACK_OVERFLOW)

Callstack

(665c.31a0): Stack overflow - code c00000fd (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
*** WARNING: Unable to verify checksum for crypto_test.exe
crypto_test!__chkstk+0x37:
00007ff7`4a62c887 41c60300        mov     byte ptr [r11],0 ds:0000008c`6af06000=00
0:000> k
 # Child-SP          RetAddr           Call Site
00 0000008c`6af08cd8 00007ff7`4992d3db crypto_test!__chkstk+0x37 [d:\a01\_work\43\s\src\vctools\crt\vcstartup\src\misc\amd64\chkstk.asm @ 109] 
01 0000008c`6af08cf0 00007ff7`49918e7d crypto_test!tungstenite::handshake::machine::HandshakeMachine<tokio_tungstenite::compat::AllowStd<enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > > >::single_round<tokio_tungstenite::compat::AllowStd<enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > >,http::response::Response<tuple$<> > >+0xb [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tungstenite-0.17.2\src\handshake\machine.rs @ 39] 
02 0000008c`6af08d00 00007ff7`4988bb82 crypto_test!tungstenite::handshake::MidHandshake<tungstenite::handshake::client::ClientHandshake<tokio_tungstenite::compat::AllowStd<enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > > > >::handshake<tungstenite::handshake::client::ClientHandshake<tokio_tungstenite::compat::AllowStd<enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > > > >+0x8d [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tungstenite-0.17.2\src\handshake\mod.rs @ 42] 
03 0000008c`6af0ccf0 00007ff7`49972638 crypto_test!tokio_tungstenite::client_async_with_config::async_fn$0::closure$0<http::request::Request<tuple$<> >,enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > >+0x282 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-tungstenite-0.17.1\src\lib.rs @ 102] 
04 0000008c`6af0e550 00007ff7`499712e7 crypto_test!tokio_tungstenite::handshake::impl$1::poll<tungstenite::handshake::client::ClientHandshake<tokio_tungstenite::compat::AllowStd<enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > > >,tokio_tungstenite::client_async_with_config::async_fn$0::closure_env$0<http::request::Request<tuple$<> >,enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > >,enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > >+0x278 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-tungstenite-0.17.1\src\handshake.rs @ 138] 
05 0000008c`6af11d70 00007ff7`499001f7 crypto_test!tokio_tungstenite::handshake::handshake::async_fn$0<tungstenite::handshake::client::ClientHandshake<tokio_tungstenite::compat::AllowStd<enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > > >,tokio_tungstenite::client_async_with_config::async_fn$0::closure_env$0<http::request::Request<tuple$<> >,enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > >,enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > >+0x2e7 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-tungstenite-0.17.1\src\handshake.rs @ 78] 
06 0000008c`6af16d60 00007ff7`49971a39 crypto_test!core::future::from_generator::impl$1::poll<tokio_tungstenite::handshake::handshake::async_fn_env$0<tungstenite::handshake::client::ClientHandshake<tokio_tungstenite::compat::AllowStd<enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > > >,tokio_tungstenite::client_async_with_config::async_fn$0::closure_env$0<http::request::Request<tuple$<> >,enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > >,enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > > >+0x87 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
07 0000008c`6af178f0 00007ff7`49901ca7 crypto_test!tokio_tungstenite::handshake::client_handshake::async_fn$0<tokio_tungstenite::client_async_with_config::async_fn$0::closure_env$0<http::request::Request<tuple$<> >,enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > >,enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > >+0x1f9 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-tungstenite-0.17.1\src\handshake.rs @ 100] 
08 0000008c`6af1c8b0 00007ff7`4988b7c0 crypto_test!core::future::from_generator::impl$1::poll<tokio_tungstenite::handshake::client_handshake::async_fn_env$0<tokio_tungstenite::client_async_with_config::async_fn$0::closure_env$0<http::request::Request<tuple$<> >,enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > >,enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > > >+0x87 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
09 0000008c`6af1d440 00007ff7`49901177 crypto_test!tokio_tungstenite::client_async_with_config::async_fn$0<http::request::Request<tuple$<> >,enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > >+0x290 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-tungstenite-0.17.1\src\lib.rs @ 104] 
0a 0000008c`6af21c30 00007ff7`498b2a3b crypto_test!core::future::from_generator::impl$1::poll<tokio_tungstenite::client_async_with_config::async_fn_env$0<http::request::Request<tuple$<> >,enum$<tokio_tungstenite::stream::MaybeTlsStream<tokio::net::tcp::stream::TcpStream> > > >+0x87 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
0b 0000008c`6af227c0 00007ff7`499049b7 crypto_test!tokio_tungstenite::tls::client_async_tls_with_config::async_fn$0<http::request::Request<tuple$<> >,tokio::net::tcp::stream::TcpStream>+0xdab [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-tungstenite-0.17.1\src\tls.rs @ 226] 
0c 0000008c`6af28cf0 00007ff7`4992fa8f crypto_test!core::future::from_generator::impl$1::poll<tokio_tungstenite::tls::client_async_tls_with_config::async_fn_env$0<http::request::Request<tuple$<> >,tokio::net::tcp::stream::TcpStream> >+0x87 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
0d 0000008c`6af29880 00007ff7`498ffda7 crypto_test!tokio_tungstenite::connect::connect_async_with_config::async_fn$0<str>+0x98f [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-tungstenite-0.17.1\src\connect.rs @ 51] 
0e 0000008c`6af2e2b0 00007ff7`4992ef5d crypto_test!core::future::from_generator::impl$1::poll<tokio_tungstenite::connect::connect_async_with_config::async_fn_env$0<str> >+0x87 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
0f 0000008c`6af2ee40 00007ff7`498ffb87 crypto_test!tokio_tungstenite::connect::connect_async::async_fn$0<str>+0x17d [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-tungstenite-0.17.1\src\connect.rs @ 22] 
10 0000008c`6af32a40 00007ff7`499555fd crypto_test!core::future::from_generator::impl$1::poll<tokio_tungstenite::connect::connect_async::async_fn_env$0<str> >+0x87 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
11 0000008c`6af335d0 00007ff7`499000e7 crypto_test!crypto_ws_client::common::connect_async::connect_async::async_fn$0+0x49d [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\crypto-ws-client-4.10.6\src\common\connect_async.rs @ 57] 
12 0000008c`6af3e9c0 00007ff7`49937d86 crypto_test!core::future::from_generator::impl$1::poll<crypto_ws_client::common::connect_async::connect_async::async_fn_env$0>+0x87 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
13 0000008c`6af3ec10 00007ff7`49900397 crypto_test!crypto_ws_client::common::ws_client_internal::impl$0::connect::async_fn$0<crypto_ws_client::clients::binance::BinanceMessageHandler>+0x2c6 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\crypto-ws-client-4.10.6\src\common\ws_client_internal.rs @ 48] 
14 0000008c`6af422d0 00007ff7`498697c8 crypto_test!core::future::from_generator::impl$1::poll<crypto_ws_client::common::ws_client_internal::impl$0::connect::async_fn_env$0<crypto_ws_client::clients::binance::BinanceMessageHandler> >+0x87 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
15 0000008c`6af42430 00007ff7`499033b7 crypto_test!crypto_ws_client::clients::binance::impl$0::new::async_fn$0<CONST$ad4fc55ace7c4c84>+0x258 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\crypto-ws-client-4.10.6\src\clients\binance.rs @ 84] 
16 0000008c`6af45aa0 00007ff7`499abf64 crypto_test!core::future::from_generator::impl$1::poll<crypto_ws_client::clients::binance::impl$0::new::async_fn_env$0<CONST$ad4fc55ace7c4c84> >+0x87 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
17 0000008c`6af45c10 00007ff7`498fe14f crypto_test!crypto_crawler::crawlers::utils::create_ws_client_internal::async_fn$0+0x15b4 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\crypto-crawler-4.2.2\src\crawlers\utils.rs @ 370] 
18 0000008c`6afbb3e0 00007ff7`499b0e97 crypto_test!core::future::from_generator::impl$1::poll<crypto_crawler::crawlers::utils::create_ws_client_internal::async_fn_env$0>+0x5f [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
19 0000008c`6afbb480 00007ff7`498ff8ff crypto_test!crypto_crawler::crawlers::utils::create_ws_client::async_fn$0+0x557 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\crypto-crawler-4.2.2\src\crawlers\utils.rs @ 495] 
1a 0000008c`6afc2fe0 00007ff7`499b20a0 crypto_test!core::future::from_generator::impl$1::poll<crypto_crawler::crawlers::utils::create_ws_client::async_fn_env$0>+0x5f [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
1b 0000008c`6afc3080 00007ff7`498fc461 crypto_test!crypto_crawler::crawlers::utils::crawl_event_one_chunk::async_fn$0+0x450 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\crypto-crawler-4.2.2\src\crawlers\utils.rs @ 679] 
1c 0000008c`6afc6cd0 00007ff7`499b479f crypto_test!core::future::from_generator::impl$1::poll<crypto_crawler::crawlers::utils::crawl_event_one_chunk::async_fn_env$0>+0x81 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
1d 0000008c`6afc6d80 00007ff7`49903499 crypto_test!crypto_crawler::crawlers::utils::crawl_event::async_fn$0+0x1b7f [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\crypto-crawler-4.2.2\src\crawlers\utils.rs @ 811] 
1e 0000008c`6afd3f00 00007ff7`498bf528 crypto_test!core::future::from_generator::impl$1::poll<crypto_crawler::crawlers::utils::crawl_event::async_fn_env$0>+0x59 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
1f 0000008c`6afd3f80 00007ff7`499006f9 crypto_test!crypto_crawler::crawlers::binance::crawl_trade::async_fn$0+0xa88 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\crypto-crawler-4.2.2\src\crawlers\binance.rs @ 38] 
20 0000008c`6afdb400 00007ff7`498858ea crypto_test!core::future::from_generator::impl$1::poll<crypto_crawler::crawlers::binance::crawl_trade::async_fn_env$0>+0x59 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
21 0000008c`6afdb480 00007ff7`498fd349 crypto_test!crypto_crawler::crawl_trade::async_fn$0+0x3ea [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\crypto-crawler-4.2.2\src\lib.rs @ 239] 
22 0000008c`6afeae00 00007ff7`49996227 crypto_test!core::future::from_generator::impl$1::poll<crypto_crawler::crawl_trade::async_fn_env$0>+0x59 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
23 0000008c`6afeae80 00007ff7`49900929 crypto_test!crypto_test::main::async_block$0+0x2c7 [d:\DEV\PROJ\crypto_test\src\main.rs @ 12] 
24 0000008c`6afeeff0 00007ff7`498c495a crypto_test!core::future::from_generator::impl$1::poll<crypto_test::main::async_block_env$0>+0x59 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\future\mod.rs @ 84] 
25 0000008c`6afef070 00007ff7`498c57a9 crypto_test!tokio::park::thread::impl$5::block_on::closure$0<core::future::from_generator::GenFuture<crypto_test::main::async_block_env$0> >+0x4a [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.18.2\src\park\thread.rs @ 263] 
26 0000008c`6afef0e0 00007ff7`499a9842 crypto_test!tokio::coop::with_budget::closure$0<enum$<core::task::poll::Poll<tuple$<> > >,tokio::park::thread::impl$5::block_on::closure_env$0<core::future::from_generator::GenFuture<crypto_test::main::async_block_env$0> > >+0xa9 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.18.2\src\coop.rs @ 102] 
27 0000008c`6afef170 00007ff7`499a962d crypto_test!std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget> >::try_with<core::cell::Cell<tokio::coop::Budget>,tokio::coop::with_budget::closure_env$0<enum$<core::task::poll::Poll<tuple$<> > >,tokio::park::thread::impl$5::block_on::closure_env$0<core::future::from_generator::GenFuture<crypto_test::main::async_block_env$0> > >,enum$<core::task::poll::Poll<tuple$<> > > >+0xa2 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\std\src\thread\local.rs @ 413] 
28 0000008c`6afef230 00007ff7`498c4753 crypto_test!std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget> >::with<core::cell::Cell<tokio::coop::Budget>,tokio::coop::with_budget::closure_env$0<enum$<core::task::poll::Poll<tuple$<> > >,tokio::park::thread::impl$5::block_on::closure_env$0<core::future::from_generator::GenFuture<crypto_test::main::async_block_env$0> > >,enum$<core::task::poll::Poll<tuple$<> > > >+0x2d [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\std\src\thread\local.rs @ 389] 
29 (Inline Function) --------`-------- crypto_test!tokio::coop::with_budget+0x2e [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.18.2\src\coop.rs @ 95] 
2a (Inline Function) --------`-------- crypto_test!tokio::coop::budget+0x90 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.18.2\src\coop.rs @ 72] 
2b 0000008c`6afef280 00007ff7`4996769b crypto_test!tokio::park::thread::CachedParkThread::block_on<core::future::from_generator::GenFuture<crypto_test::main::async_block_env$0> >+0x253 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.18.2\src\park\thread.rs @ 263] 
2c 0000008c`6aff1480 00007ff7`4987d09d crypto_test!tokio::runtime::enter::Enter::block_on<core::future::from_generator::GenFuture<crypto_test::main::async_block_env$0> >+0x7b [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.18.2\src\runtime\enter.rs @ 151] 
2d 0000008c`6aff3580 00007ff7`49996706 crypto_test!tokio::runtime::thread_pool::ThreadPool::block_on<core::future::from_generator::GenFuture<crypto_test::main::async_block_env$0> >+0x7d [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.18.2\src\runtime\thread_pool\mod.rs @ 81] 
2e 0000008c`6aff5680 00007ff7`498eb252 crypto_test!tokio::runtime::Runtime::block_on<core::future::from_generator::GenFuture<crypto_test::main::async_block_env$0> >+0x106 [C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.18.2\src\runtime\mod.rs @ 477] 
2f 0000008c`6aff9780 00007ff7`4988fd7b crypto_test!crypto_test::main+0xf2 [d:\DEV\PROJ\crypto_test\src\main.rs @ 12] 
30 0000008c`6afffa60 00007ff7`4996c66b crypto_test!core::ops::function::FnOnce::call_once<void (*)(),tuple$<> >+0xb [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\ops\function.rs @ 227] 
31 0000008c`6afffaa0 00007ff7`498651a1 crypto_test!std::sys_common::backtrace::__rust_begin_short_backtrace<void (*)(),tuple$<> >+0x1b [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\std\src\sys_common\backtrace.rs @ 125] 
32 0000008c`6afffae0 00007ff7`4a5f8baf crypto_test!std::rt::lang_start::closure$0<tuple$<> >+0x11 [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\std\src\rt.rs @ 145] 
33 (Inline Function) --------`-------- crypto_test!core::ops::function::impls::impl$2::call_once+0xb [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\core\src\ops\function.rs @ 259] 
34 (Inline Function) --------`-------- crypto_test!std::panicking::try::do_call+0xb [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\/library\std\src\panicking.rs @ 492] 
35 (Inline Function) --------`-------- crypto_test!std::panicking::try+0xb [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\/library\std\src\panicking.rs @ 456] 
36 (Inline Function) --------`-------- crypto_test!std::panic::catch_unwind+0xb [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\/library\std\src\panic.rs @ 137] 
37 (Inline Function) --------`-------- crypto_test!std::rt::lang_start_internal::closure$2+0xb [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\/library\std\src\rt.rs @ 128] 
38 (Inline Function) --------`-------- crypto_test!std::panicking::try::do_call+0xb [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\/library\std\src\panicking.rs @ 492] 
39 (Inline Function) --------`-------- crypto_test!std::panicking::try+0xb [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\/library\std\src\panicking.rs @ 456] 
3a (Inline Function) --------`-------- crypto_test!std::panic::catch_unwind+0xb [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\/library\std\src\panic.rs @ 137] 
3b 0000008c`6afffb20 00007ff7`4986516f crypto_test!std::rt::lang_start_internal+0x11f [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\/library\std\src\rt.rs @ 128] 
3c 0000008c`6afffc00 00007ff7`498eb346 crypto_test!std::rt::lang_start<tuple$<> >+0x2f [/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\std\src\rt.rs @ 144] 
3d 0000008c`6afffc60 00007ff7`4a62cabc crypto_test!main+0x16
3e (Inline Function) --------`-------- crypto_test!invoke_main+0x22 [d:\a01\_work\43\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 78] 
3f 0000008c`6afffc90 00007ffc`319254e0 crypto_test!__scrt_common_main_seh+0x10c [d:\a01\_work\43\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288] 
40 0000008c`6afffcd0 00007ffc`3232485b KERNEL32!BaseThreadInitThunk+0x10
41 0000008c`6afffd00 00000000`00000000 ntdll!RtlUserThreadStart+0x2b

It seems between frame 17 and 18 a bunch of state data (500Kb) is allocated on stack instead of heap or something. Don't have much time to investigate, sorry.

The development schedule of crypto-client

Hi Soulmachine,

crypto-client is important to trade in the exchanges.

Async Client is desired.

Any schedule?

try to got data from huobi fail

below is detail logs thanks !

// crawl_trade("huobi", MarketType::Spot, symbols_opt, tx).await;
// crawl_l2_event("huobi", MarketType::Spot, symbols_opt, tx).await;
// crawl_l2_topk("huobi", MarketType::Spot, None, tx).await;
//crawl_bbo("huobi", MarketType::Spot, None, tx).await;
// crawl_candlestick("huobi", MarketType::Spot, None, tx).await;
// crawl_funding_rate("huobi", MarketType::Spot, None, tx).await;

thread 'main' panicked at 'Failed to connect to websocket: Io(Os { code: 60, kind: TimedOut, message: "Operation timed out" })', /Users/admin/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-ws-client-4.11.4/src/common/ws_client_internal.rs:50:14
stack backtrace:
0: rust_begin_unwind
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:584:5
1: core::panicking::panic_fmt
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:143:14
2: core::result::unwrap_failed
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/result.rs:1749:5
3: core::result::Result<T,E>::expect
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/result.rs:1022:23
4: crypto_ws_client::common::ws_client_internal::WSClientInternal::connect::{{closure}}
at /Users/admin/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-ws-client-4.11.4/src/common/ws_client_internal.rs:48:40
5: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
6: crypto_ws_client::clients::huobi::HuobiWSClient<_>::new::{{closure}}
at /Users/admin/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-ws-client-4.11.4/src/clients/huobi.rs:98:14
7: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
8: crypto_crawler::crawlers::huobi::crawl_l2_event::{{closure}}
at /Users/admin/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-crawler-4.4.0/src/crawlers/huobi.rs:34:89
9: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
10: crypto_crawler::crawl_l2_event::{{closure}}
at /Users/admin/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-crawler-4.4.0/src/lib.rs:260:77
11: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
12: test::main::{{closure}}
at ./crypto-test/src/test.rs:95:63
13: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
14: tokio::park::thread::CachedParkThread::block_on::{{closure}}
at /Users/admin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.19.2/src/park/thread.rs:263:54
15: tokio::coop::with_budget::{{closure}}
at /Users/admin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.19.2/src/coop.rs:102:9
16: std::thread::local::LocalKey::try_with
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/thread/local.rs:413:16
17: std::thread::local::LocalKey::with
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/thread/local.rs:389:9
18: tokio::coop::with_budget
at /Users/admin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.19.2/src/coop.rs:95:5
19: tokio::coop::budget
at /Users/admin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.19.2/src/coop.rs:72:5
20: tokio::park::thread::CachedParkThread::block_on
at /Users/admin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.19.2/src/park/thread.rs:263:31
21: tokio::runtime::enter::Enter::block_on
at /Users/admin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.19.2/src/runtime/enter.rs:151:13
22: tokio::runtime::thread_pool::ThreadPool::block_on
at /Users/admin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.19.2/src/runtime/thread_pool/mod.rs:90:9
23: tokio::runtime::Runtime::block_on
at /Users/admin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.19.2/src/runtime/mod.rs:482:43
24: test::main
at ./crypto-test/src/test.rs:95:5
25: core::ops::function::FnOnce::call_once
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

Add support for Bittrex

Bittrex is a cryptocurrency exchange in US, and this library needs to support it.

Volunteers are always welcomed!

Compile failed due to rust nightly bug

CI is failing https://github.com/crypto-crawler/crypto-crawler-rs/runs/5685232312?check_suite_focus=true

due to this bug rust-lang/rust#95267

Need to wait for a newer nightly release

Migrate from `f32` and `f64` to rust_decimal::Decimal

Abstract

f32 and f64 has some precision issues¹, which may be critical for finance calculation². You can find it with this interactive converter.

crypto-crawler currently converts all the decimal float numbers (String) to binary float numbers (f32/f64), and stores values in Float/f32/f64. It makes the numbers inaccurate and may introduce issues when using this in production (for example, the wrong determination and time point to buy and sell³.)

Solution

To prevent the potential issue, I propose to use rust_decimal::Decimal instead of f32 or f64. It stores float numbers in decimal and thus prevents infinite precision due to the binary representation.

I've tried this library in my project (though has not been in production yet)⁴, and I can help rewrite this to Decimal.

TBDs (To Be Discussed)

It introduces breaking changes. A safer non-breaking way is to write a feature flag and manually enable Decimal; however, it complexes the codebase.
Error handling when we failed to convert a number to Decimal or convert a Decimal to a number. Note that Decimal::to_f64(self) returns a Option<f64>, and the Option may be None if f64 is not enough to represent this Decimal.

(Chinese) https://medium.com/starbugs/see-why-floating-point-error-can-not-be-avoided-from-ieee-754-809720b32175 ↩
(Chinese, ⚠️ Content Farm) https://www.twblogs.net/a/5eec07e211be511c38938805 ↩
(Chinese) https://zh.wikipedia.org/zh-tw/捨入誤差#举例 ↩
wmjtyd/libstock (1) wmjtyd/libstock (2) ↩

After 3 days websocket disconnected (Failed to read, error: IO error: Connection reset by peer (os error 104)) and did not automatically reconnect after

I left the basic README websocket example running for a few days, and then ECONNRESET 104 Connection reset by peer occurred and websocket disconnected. After that ping/pongs were failing but there seems to be no auto-reconnection mechanism. I reproduced this several times.
I think it should reconnect on ERROR crypto_ws_client::common::connect_async] Failed to read, error: IO error: Connection reset by peer (os error 104) and also on failed ping sends.
The log at the moment of the disconnect 2023-01-13T12:44:15Z follows:

[2023-01-13T12:44:09Z INFO  qk] {"exchange":"kucoin","market_type":"spot","msg_type":"trade","received_at":1673613849587,"json":"{\"type\":\"message\",\"topic\":\"/market/match:SOL-USDT\",\"subject\":\"trade.l3match\",\"data\":{\"makerOrderId\":\"63c152190f2fb20001bd63ba\",\"price\":\"16.569\",\"sequence\":\"1184114405824513\",\"side\":\"buy\",\"size\":\"5.4773\",\"symbol\":\"SOL-USDT\",\"takerOrderId\":\"63c1521917f8760001e12e2b\",\"time\":\"1673613849515000000\",\"tradeId\":\"1184114405824513\",\"type\":\"match\"}}"}
[2023-01-13T12:44:10Z INFO  qk] {"exchange":"kucoin","market_type":"spot","msg_type":"trade","received_at":1673613850830,"json":"{\"type\":\"message\",\"topic\":\"/market/match:SOL-USDT\",\"subject\":\"trade.l3match\",\"data\":{\"makerOrderId\":\"63c15219d686c500014277ae\",\"price\":\"16.57\",\"sequence\":\"1184114486564865\",\"side\":\"buy\",\"size\":\"6.5612\",\"symbol\":\"SOL-USDT\",\"takerOrderId\":\"63c1521af109f00001e76310\",\"time\":\"1673613850760000000\",\"tradeId\":\"1184114486564865\",\"type\":\"match\"}}"}
[2023-01-13T12:44:12Z DEBUG crypto_ws_client::common::ws_client_internal] Instant { tv_sec: 80631244, tv_nsec: 862016328 } sending ping {"type":"ping", "id": "crypto-ws-client"}
[2023-01-13T12:44:12Z DEBUG crypto_ws_client::common::ws_client_internal] Received {"id":"crypto-ws-client","type":"pong"} from kucoin, reset num_unanswered_ping to 0
[2023-01-13T12:44:15Z INFO  qk] {"exchange":"kucoin","market_type":"spot","msg_type":"trade","received_at":1673613855208,"json":"{\"type\":\"message\",\"topic\":\"/market/match:SOL-USDT\",\"subject\":\"trade.l3match\",\"data\":{\"makerOrderId\":\"63c1521efe6ad80001d821ae\",\"price\":\"16.569\",\"sequence\":\"1184114833381377\",\"side\":\"buy\",\"size\":\"2.1725\",\"symbol\":\"SOL-USDT\",\"takerOrderId\":\"63c1521fbdf44300018a2cdc\",\"time\":\"1673613855040000000\",\"tradeId\":\"1184114833381377\",\"type\":\"match\"}}"}
[2023-01-13T12:44:15Z INFO  qk] {"exchange":"kucoin","market_type":"spot","msg_type":"trade","received_at":1673613855209,"json":"{\"type\":\"message\",\"topic\":\"/market/match:SOL-USDT\",\"subject\":\"trade.l3match\",\"data\":{\"makerOrderId\":\"63c1521efe6ad80001d821ae\",\"price\":\"16.569\",\"sequence\":\"1184114833643521\",\"side\":\"buy\",\"size\":\"0.6579\",\"symbol\":\"SOL-USDT\",\"takerOrderId\":\"63c1521f6a048f000144ef4c\",\"time\":\"1673613855040000000\",\"tradeId\":\"1184114833643521\",\"type\":\"match\"}}"}
[2023-01-13T12:44:15Z ERROR crypto_ws_client::common::connect_async] Failed to read, error: IO error: Connection reset by peer (os error 104)
[2023-01-13T12:44:15Z DEBUG tokio_tungstenite] websocket start_send error: IO error: Broken pipe (os error 32)
[2023-01-13T12:44:15Z DEBUG crypto_ws_client::common::ws_client_internal] Instant { tv_sec: 80631247, tv_nsec: 621612513 } sending ping {"type":"ping", "id": "crypto-ws-client"}
[2023-01-13T12:44:15Z ERROR crypto_ws_client::common::ws_client_internal] Error sending ping channel closed
[2023-01-13T12:44:43Z DEBUG crypto_ws_client::common::ws_client_internal] Instant { tv_sec: 80631275, tv_nsec: 862016328 } sending ping {"type":"ping", "id": "crypto-ws-client"}
[2023-01-13T12:44:43Z ERROR crypto_ws_client::common::ws_client_internal] Error sending ping channel closed
[2023-01-13T12:44:46Z DEBUG crypto_ws_client::common::ws_client_internal] Instant { tv_sec: 80631278, tv_nsec: 621612513 } sending ping {"type":"ping", "id": "crypto-ws-client"}
[2023-01-13T12:44:46Z ERROR crypto_ws_client::common::ws_client_internal] Error sending ping channel closed
[2023-01-13T12:45:14Z DEBUG crypto_ws_client::common::ws_client_internal] Instant { tv_sec: 80631306, tv_nsec: 862016328 } sending ping {"type":"ping", "id": "crypto-ws-client"}
[2023-01-13T12:45:14Z ERROR crypto_ws_client::common::ws_client_internal] Error sending ping channel closed
[2023-01-13T12:45:17Z DEBUG crypto_ws_client::common::ws_client_internal] Instant { tv_sec: 80631309, tv_nsec: 621612513 } sending ping {"type":"ping", "id": "crypto-ws-client"}
[2023-01-13T12:45:17Z ERROR crypto_ws_client::common::ws_client_internal] Error sending ping channel closed
[2023-01-13T12:45:45Z DEBUG crypto_ws_client::common::ws_client_internal] Instant { tv_sec: 80631337, tv_nsec: 862016328 } sending ping {"type":"ping", "id": "crypto-ws-client"}
[2023-01-13T12:45:45Z ERROR crypto_ws_client::common::ws_client_internal] Error sending ping channel closed
[2023-01-13T12:45:48Z DEBUG crypto_ws_client::common::ws_client_internal] Instant { tv_sec: 80631340, tv_nsec: 621612513 } sending ping {"type":"ping", "id": "crypto-ws-client"}                                                                :
[2023-01-13T12:45:48Z ERROR crypto_ws_client::common::ws_client_internal] Error sending ping channel closed

Is level 3 orderbook websocket implemented?

Apologies for the beginner question. I went through the code but couldn't answer it myself.

Does this build the level 3 orderbook (full book) for any of the exchanges and in particular Coinbase? I don't think it does

`seq_id` and `prev_seq_id` are not being setup for gateio

seq_id and prev_seq_id are not being setup for gateio

It should probably be ws_msg.U and ws_msg.u but not sure I understand the definition of seq_id and prev_seq_id.

Gateio orderbook reference

keeping books in sync

Is there any logic in cryptocrawler to keep books in sync with the exchange?

It seems that there is some support for that but no sure if there is some additional logic like for example here.

Prevent using `.unwrap()` and `.expect()` in codebase

Abstract

crypto-crawler panics when the data format is invalid, which may have better recoverable way to indicate instead of just aborting the whole tasks.

Solution

Replace all of them to Result<T, SimpleError>:

ws_msg.data.B.parse::<f64>().map_err(|_| SimpleError::new("Failed to parse ws_msg.data.B"));

Besides, I recommended replacing SimpleError to anyhow. It allows us wrapping all the error with a generic Error:

fn parse() -> anyhow::Result<T> {
  // Even simpler than `.unwrap()`. Besides, the `parse()` is recoverable now!
  ws_msg.data.B.parse::<f64>()?;

  // If you need context:
  ws_msg.data.B.parse::<f64>().context("Failed to parse ws_msg.data.B")?;
}

TBDs (To Be Discussed)

Use anyhow or remain SimpleError? I prefer former.
Migrate it with less breaking changes (or just make it without writing too many “runtime” to unwrap)

carbonbot: specify symbols

first I have to thank you so much for providing this awesome piece of software! it really backs me out <3

I'd like to be in more control of the symbols ingested by carbonbot, since blindly subscribing to everything an exchange provides generates a tremendous amount of data which I don't need.

features I'd like implement:

sort the symbols by CMC rank, then truncate to a user supplied length (= "top X symbols by CMC rank").
filter the symbols by a list of user supplied substring matches (allows manual control)

These additional "configuration" parameters could be supplied as cmd args, env vars or via a config file, please let me know whatever works best for you. If you are interested I can open a pull-request.

Example to reconstruct full order book from updates

It would be great to have an example on how to keep a full order book from updates.
Related gateio example

crypto-ws-client example does not work; MissingOrMalformedExtensions thrown by webpki

I've been trying to run the example for the websocket client, but unfortunately, I keep getting the following error:

thread 'tokio-runtime-worker' panicked at 'Failed to connect to wss://ws-feed.exchange.coinbase.com, error: TLS error: webpki error: MissingOrMalformedExtensions', C:\Users\jsvanv\.cargo\registry\src\github.com-1ecc6299db9ec823\crypto-ws-client-4.12.6\src\common\ws_client_internal.rs:76:22

I get the same error on another Windows 10 machine. I've tried a small variety of exchanges and symbols, but webpki always complains.

Exchanges tried: Binance, Kraken, Coinbase Pro
Symbols tried: ETHUSDT, ETHUSD, BTCUSDT, BTCUSD
OS: Windows 10 64-bit
Targets: x86_64-pc-windows-gnu and x86_64-pc-windows-msvc
Rust version: 1.64.0

Deadlock when many exchanges are crawled

I guess that the problem leaves in

pub(crate) fn create_conversion_thread(
    exchange: String,
    msg_type: MessageType,
    market_type: MarketType,
    tx: Sender<Message>,
) -> Sender<String> {
    let (tx_raw, rx_raw) = std::sync::mpsc::channel();
    tokio::task::spawn(async move {
        for json in rx_raw {
            let msg = Message::new(exchange.clone(), market_type, msg_type, json);
            if tx.send(msg).is_err() {
                break; // break the loop if there is no receiver
            }
        }
    });
    tx_raw
}

And exactly in

    tokio::task::spawn(async move {

I think that's because std::sync::mpsc::channel block the thread until data is available and official tokio documentation tells that this is not allowed (https://tokio.rs/tokio/tutorial/channels).

I see two options to fix this:

To use 'tokio::task::spawn_blocking(move || {'
change from usage of std::sync::mpsc::channel to tokio::sync::mpsc

crawl more than one exchange?

Hi thanks for the great library! I was wondering if there is some way to use crypto_crawler::crawl_l2_event but for several exchanges? (rust noobie)

Anyone has the trade data from 2022-06-22 to 2022-06-28?

I accidentally deleted all trade data from 2022-06-22 to 2022-06-28 recently. Anybody has thetrade data during this period? Thanks

The root cause is that rclone sync deletes files silently in the destination fold, while rsync doesn't. Previously I thought rclone sync is compatible with rsync.

I used rclone sync to merge two trade folders, then the tragedy happened 😢

If anyone happens to have this data, please leave a comment here, Thanks!

is proxy can be supported?

Some area can't access the exchange directly, they need proxy for websocket ，so,can you provide a interface or api used for setting proxy?

coinbase_pro exchange doesn't work any more

https://docs.cloud.coinbase.com/exchange/reference/exchangerestapi_getproducts

The properties base_max_size, base_min_size, max_market_funds were [removed on June 30 (https://docs.cloud.coinbase.com/exchange/docs/changelog#2022-jun-30).

src/exchanges/coinbase_pro.rs

fn fetch_spot_markets_raw() -> Result<Vec<SpotMarket>> {
  let txt = http_get("https://api.pro.coinbase.com/products", None)?;
  let markets = serde_json::from_str::<Vec<SpotMarket>>(&txt)?;
  Ok(markets)
}

crypto_msg_parser::parse_trade() : Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.

I am using the following code (collected from your examples):

 1 use crypto_crawler::{crawl_trade, MarketType};
  2
  3 #[tokio::main(flavor = "multi_thread")]
  4 async fn main() {
  5     let (tx, rx) = std::sync::mpsc::channel();
  6
  7     tokio::task::spawn(async move {
  8         crawl_trade(
  9             "kucoin",
 10             MarketType::LinearSwap,
 11             Some(&["SOLUSDTM".to_string()]),
 12             /*
 13             MarketType::Spot,
 14             Some(&["SOL-USDT".to_string()]),
 15             */
 16             tx,
 17         )
 18         .await
 19     });
 20     for msg in rx {
 21         //https://github.com/crypto-crawler/crypto-crawler-rs/blob/main/crypto-crawler/src/msg.rs
 22         let trades =
 23             crypto_msg_parser::parse_trade(&msg.exchange, msg.market_type, &msg.json).unwrap();
 24         let trade = &trades[0];
 25         println!("{:#?}", trade);
 26     }
 27 }
~

Only for MarketType::LinearSwap (but not for MarketType::Spot because https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L198 never gets called for Spot) I get the following panic with the debug build (this never happens with a release build, i think it's something debug-specific? with blocking http_get("https://api-futures.kucoin.com/api/v1/contracts/active") reqwest in crypto-contract-value-1.7.7/src/exchanges/kucoin.rs:181 ).

alecm@ark2:~/qkt$ ./target/debug/qk
thread 'main' panicked at 'Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.', /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/blocking/shutdown.rs:51:21
stack backtrace:
   0: std::panicking::begin_panic
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/std/src/panicking.rs:588:12
   1: tokio::runtime::blocking::shutdown::Receiver::wait
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/blocking/shutdown.rs:51:21
   2: tokio::runtime::blocking::pool::BlockingPool::shutdown
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/blocking/pool.rs:261:12
   3: <tokio::runtime::blocking::pool::BlockingPool as core::ops::drop::Drop>::drop
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/blocking/pool.rs:278:9
   4: core::ptr::drop_in_place<tokio::runtime::blocking::pool::BlockingPool>
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ptr/mod.rs:487:1
   5: core::ptr::drop_in_place<tokio::runtime::runtime::Runtime>
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ptr/mod.rs:487:1
   6: reqwest::blocking::wait::enter
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.13/src/blocking/wait.rs:76:21
   7: reqwest::blocking::wait::timeout
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.13/src/blocking/wait.rs:13:5
   8: reqwest::blocking::client::ClientHandle::new
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.13/src/blocking/client.rs:1038:15
   9: reqwest::blocking::client::ClientBuilder::build
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.13/src/blocking/client.rs:103:9
  10: crypto_contract_value::exchanges::utils::http_get
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/exchanges/utils.rs:7:18
  11: crypto_contract_value::exchanges::kucoin::fetch_linear_multipliers
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/exchanges/kucoin.rs:181:22
  12: crypto_contract_value::exchanges::kucoin::LINEAR_CONTRACT_VALUES::{{closure}}
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/exchanges/kucoin.rs:152:23
  13: core::ops::function::FnOnce::call_once
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ops/function.rs:251:5
  14: core::ops::function::FnOnce::call_once
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ops/function.rs:251:5
  15: once_cell::sync::Lazy<T,F>::force::{{closure}}
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1275:28
  16: once_cell::sync::OnceCell<T>::get_or_init::{{closure}}
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1085:57
  17: once_cell::imp::OnceCell<T>::initialize::{{closure}}
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/imp_std.rs:82:23
  18: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ops/function.rs:297:13
  19: once_cell::imp::initialize_or_wait
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/imp_std.rs:206:20
  20: once_cell::imp::OnceCell<T>::initialize
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/imp_std.rs:78:9
  21: once_cell::sync::OnceCell<T>::get_or_try_init
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1126:13
  22: once_cell::sync::OnceCell<T>::get_or_init
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1085:19
  23: once_cell::sync::Lazy<T,F>::force
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1274:13
  24: <once_cell::sync::Lazy<T,F> as core::ops::deref::Deref>::deref
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1335:13
  25: crypto_contract_value::exchanges::kucoin::get_contract_value
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/exchanges/kucoin.rs:198:40
  26: crypto_contract_value::get_contract_value
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/lib.rs:22:21
  27: crypto_msg_parser::exchanges::utils::calc_quantity_and_volume
             at /home/alecm/.cargo/git/checkouts/crypto-msg-parser-aaa16a2660930560/8bc9eb7/crypto-msg-parser/src/exchanges/utils.rs:41:9
  28: crypto_msg_parser::exchanges::kucoin::kucoin_swap::parse_trade
             at /home/alecm/.cargo/git/checkouts/crypto-msg-parser-aaa16a2660930560/8bc9eb7/crypto-msg-parser/src/exchanges/kucoin/kucoin_swap.rs:64:62
  29: crypto_msg_parser::exchanges::kucoin::parse_trade
             at /home/alecm/.cargo/git/checkouts/crypto-msg-parser-aaa16a2660930560/8bc9eb7/crypto-msg-parser/src/exchanges/kucoin/mod.rs:118:9
  30: crypto_msg_parser::parse_trade
             at /home/alecm/.cargo/git/checkouts/crypto-msg-parser-aaa16a2660930560/8bc9eb7/crypto-msg-parser/src/lib.rs:97:21
  31: qk::main::{{closure}}
             at ./src/main.rs:23:13
  32: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/future/mod.rs:91:19
  33: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/park.rs:283:63
  34: tokio::runtime::coop::with_budget
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/coop.rs:102:5
  35: tokio::runtime::coop::budget
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/coop.rs:68:5
  36: tokio::runtime::park::CachedParkThread::block_on
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/park.rs:283:31
  37: tokio::runtime::context::BlockingRegionGuard::block_on
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/context.rs:295:13
  38: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/scheduler/multi_thread/mod.rs:66:9
  39: tokio::runtime::runtime::Runtime::block_on
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/runtime.rs:284:45
  40: qk::main
             at ./src/main.rs:20:5
  41: core::ops::function::FnOnce::call_once
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ops/function.rs:251:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

PS. With the release build, i let it run for several days, but the websocket stopped getting new messages after 2 days. I rebuilt as a debug and ran into the panic above. Should I file a separate Issue on the stuck websocket (unsure how to debug that, strace showed waiting for some FUTEX)?

crypto-crawler / crypto-crawler-rs Goto Github PK

crypto-crawler-rs's Introduction

crypto-crawler-rs

Quickstart

Architecture

How to parse raw messages

crypto-crawler-rs's People

Contributors

Stargazers

Watchers

Forkers

crypto-crawler-rs's Issues

Abstract

Solution

TBDs (To Be Discussed)

Footnotes

Abstract

Solution

TBDs (To Be Discussed)

Recommend Projects

Recommend Topics

Recommend Org