crypto-toolbox / btfxwss Goto Github PK

View Code? Open in Web Editor NEW

285.0 29.0 125.0 293 KB

Bitfinex Websocket API Client written in Python3

License: MIT License

Python 100.00%

cryptocurrency bitfinex-websocket-api python bitfinex-wss exchange-api

btfxwss's Introduction

BTFXWSS

Client for Bitfinex Websocket API written in Python

Currently supports all public endpoints; authenticated channels are a work in progress, but are supported.

Offers graceful exception handling of common server errors. Make sure you check the log messages and have proper logging enabled, as there are no exceptions thrown.

Data is stored within BtfxWss as Queues. There are convenience methods available to retrieve a queue for a given type. Consult the code for more documentation.

Please note that you must take care of handling data in the queues yourself! Not doing so will eventually result in MemoryErrors, since the queues do not have a maximum length defined.

Installation

Via pip:

pip install btfxwss

Usage

import logging
import time
import sys

from btfxwss import BtfxWss
    
log = logging.getLogger(__name__)

fh = logging.FileHandler('test.log')
fh.setLevel(logging.DEBUG)
sh = logging.StreamHandler(sys.stdout)
sh.setLevel(logging.DEBUG)

log.addHandler(sh)
log.addHandler(fh)
logging.basicConfig(level=logging.DEBUG, handlers=[fh, sh])
    

wss = BtfxWss()
wss.start()

while not wss.conn.connected.is_set():
    time.sleep(1)

# Subscribe to some channels
wss.subscribe_to_ticker('BTCUSD')
wss.subscribe_to_order_book('BTCUSD')

# Do something else
t = time.time()
while time.time() - t < 10:
    pass

# Accessing data stored in BtfxWss:
ticker_q = wss.tickers('BTCUSD')  # returns a Queue object for the pair.
while not ticker_q.empty():
    print(ticker_q.get())

# Unsubscribing from channels:
wss.unsubscribe_from_ticker('BTCUSD')
wss.unsubscribe_from_order_book('BTCUSD')

# Shutting down the client:
wss.stop()

Your help is required

If you find any bugs, error or have feature requests, please don't hesitate to open an issue. Be as descriptive as possible, and I'll look into the matter as soon as I can.

Looking for a REST client?

Checkout BitEx, a python-based Framework for connecting to a variety of Bitcoin exchanges, including Bitfinex!

Thanks

A big thanks to the devs providing the websocket-client <https://github.com/websocket-client/websocket-client> library, as well ekulyk for providing the PythonPusherClient, which I used as a reference for the connection class. And finally, a big thanks to all the people submitting issues, discussing solutions and simply starring the project - you all help me stay excited and motivated for this project! Cheers to you.

btfxwss's People

Contributors

Stargazers

Watchers

Forkers

litch leobouloc tangshuran crazyheex billcamel rodrigo-m-lopez mademin emilam kramer65 bekir-arslan deromka meercodemaster leokudrik lastagile dzhlovegithub s5s zhiyuzhang86 henrytw lucasb-eyer lucasb davkhech yury-pachin rterbush theodleif kay0822 kimms213 thlo7777 auto-d1dact snth so2liu declanholliday tyzjames voxelru rchiossi digicur runz0rd aceofall junw3i samholt tux-00 sevikkk milodu sergshabal farshidgit tonypius bion1cle csrocha tarcil ghitodin andressommerhoff anthbs neoleo001 niladell iforiq blockcoingroup j0k bestlinkxx 171767313 shenggaozhu renochen jrhee17 depresto spox03 chethan123 rcshadman keeganmccallum lucyking real0day edwardburgin mannguyen0107 ryan-leung sanderbrauwers klen1n stopium landwei gavinong10 baxeico dennysvf zer0hedge louis-cai yenlinlai philsippl yiylin qichwi robert-verkuil usama2762 petr-fedorov lange88 ethanedelson waltervolodenkov360 lihon9 gadasin-darling alxgrh neohanju digasic martinh85 tm-89 liuchangyu1988 novacuum 1337guys

btfxwss's Issues

websocket._exceptions.WebSocketConnectionClosedException: socket is already closed.

I setup the script collecting some candles,

After chewing through around 6700 candles worth of data... I got this error..

2017-08-21 12:18:16,802 -- INFO -- connection 179 -- Connection Error - Connection is already closed.

I got quite a few of these..

2017-08-21 13:31:32,821 -- INFO -- connection 121 -- Attempting to connect again in 10 seconds.
2017-08-21 13:31:43,501 -- INFO -- connection 179 -- Connection Error - Handshake status 525
2017-08-21 13:31:43,508 -- INFO -- connection 168 -- Connection closed

a few of these ...

2017-08-21 13:33:37,691 -- INFO -- connection 173 -- Connection opened
2017-08-21 13:33:47,819 -- INFO -- connection 121 -- Attempting to connect again in 10 seconds.
2017-08-21 13:33:58,525 -- INFO -- connection 179 -- Connection Error - Handshake status 520
2017-08-21 13:33:58,533 -- INFO -- connection 168 -- Connection closed

And a few of these...

2017-08-21 14:19:28,177 -- INFO -- connection 168 -- Connection closed
2017-08-21 14:19:28,183 -- INFO -- connection 121 -- Attempting to connect again in 10 seconds.
2017-08-21 14:19:38,764 -- INFO -- connection 179 -- Connection Error - Handshake status 302

Ultimately the script after 2 hours of trying produced ..

2017-08-21 15:25:25,719 -- INFO -- connection 173 -- Connection opened
Exception in thread Thread-21958:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.6/threading.py", line 1182, in run
    self.function(*self.args, **self.kwargs)
  File "/usr/local/apache2/htdocs/Bots/btfxwss/connection.py", line 218, in send_ping
    self.conn.send(json.dumps({'event': 'ping'}))
  File "/usr/local/lib/python3.6/site-packages/websocket/_app.py", line 117, in send
    if not self.sock or self.sock.send(data, opcode) == 0:
  File "/usr/local/lib/python3.6/site-packages/websocket/_core.py", line 234, in send
    return self.send_frame(frame)
  File "/usr/local/lib/python3.6/site-packages/websocket/_core.py", line 259, in send_frame
    l = self._send(data)
  File "/usr/local/lib/python3.6/site-packages/websocket/_core.py", line 423, in _send
    return send(self.sock, data)
  File "/usr/local/lib/python3.6/site-packages/websocket/_socket.py", line 113, in send
    raise WebSocketConnectionClosedException("socket is already closed.")
websocket._exceptions.WebSocketConnectionClosedException: socket is already closed.

Test for initial connection state should be inside BtfxWss.start()

This snippet of code at Usage section in README.md

while not wss.conn.connected.is_set():
    time.sleep(1)

Should be inside BtfxWss.start():

    def start(self):
        """Start the client.
        
        :return: 
        """
        self.conn.start()
        self.queue_processor.start()
        while not self.conn.connected.is_set():
            time.sleep(.1)

This would avoid end user to care about this.

Restart() Method throws error

@GenesisGupta submitted the following issue

I am getting an error while restarting :

Exception in thread Controller Thread:
Traceback (most recent call last):
  File "D:\Program Files\Python35\lib\threading.py", line 914, in
_bootstrap_inner
    self.run()
  File "D:\Program Files\Python35\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "D:\Program Files\Python35\lib\site-packages\btfxwss\classes.py",
line 174, in _controller
    self.restart(soft=True)
  File "D:\Program Files\Python35\lib\site-packages\btfxwss\classes.py",
line 323, in restart
    self.start()
  File "D:\Program Files\Python35\lib\site-packages\btfxwss\classes.py",
line 1103, in start
    super(BtfxWssRaw, self).start()
  File "D:\Program Files\Python35\lib\site-packages\btfxwss\classes.py",
line 255, in start
    self.conn = create_connection(self.addr, timeout=3)
  File "D:\Program Files\Python35\lib\site-packages\websocket\_core.py",
line 487, in create_connection
    websock.connect(url, **options)
  File "D:\Program Files\Python35\lib\site-packages\websocket\_core.py",
line 214, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "D:\Program
Files\Python35\lib\site-packages\websocket\_handshake.py", line 65, in
handshake
    status, resp = _get_resp_headers(sock)
  File "D:\Program
Files\Python35\lib\site-packages\websocket\_handshake.py", line 122, in
_get_resp_headers
    raise WebSocketBadStatusException("Handshake status %d", status)
websocket._exceptions.WebSocketBadStatusException: Handshake status 429
Exception in thread Controller Thread:
Traceback (most recent call last):
  File "D:\Program Files\Python35\lib\threading.py", line 914, in
_bootstrap_inner
    self.run()
  File "D:\Program Files\Python35\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "D:\Program Files\Python35\lib\site-packages\btfxwss\classes.py",
line 174, in _controller
    self.restart(soft=True)
  File "D:\Program Files\Python35\lib\site-packages\btfxwss\classes.py",
line 323, in restart
    self.start()
  File "D:\Program Files\Python35\lib\site-packages\btfxwss\classes.py",
line 1103, in start
    super(BtfxWssRaw, self).start()
  File "D:\Program Files\Python35\lib\site-packages\btfxwss\classes.py",
line 255, in start
    self.conn = create_connection(self.addr, timeout=3)
  File "D:\Program Files\Python35\lib\site-packages\websocket\_core.py",
line 487, in create_connection
    websock.connect(url, **options)
  File "D:\Program Files\Python35\lib\site-packages\websocket\_core.py",
line 214, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "D:\Program
Files\Python35\lib\site-packages\websocket\_handshake.py", line 65, in
handshake
    status, resp = _get_resp_headers(sock)
  File "D:\Program
Files\Python35\lib\site-packages\websocket\_handshake.py", line 122, in
_get_resp_headers
    raise WebSocketBadStatusException("Handshake status %d", status)
websocket._exceptions.WebSocketBadStatusException: Handshake status 429

attaching the code that i am running :

from btfxwss import BtfxWss,BtfxWssRaw
import logging
import sys
import time


logging.basicConfig(level=logging.DEBUG, filename='test.log')
log = logging.getLogger(__name__)

fh = logging.FileHandler('test.log')
fh.setLevel(logging.DEBUG)
sh = logging.StreamHandler(sys.stdout)
sh.setLevel(logging.DEBUG)

log.addHandler(sh)
log.addHandler(fh)

apikey = 'xxxxxxxxxxxxxxxx'
apisecret = 'yyyyyyyyyyyyyyyy'

wss = BtfxWssRaw(key=apikey, secret=apisecret,output_dir='./',
rotate_to='./')
wss.start()
time.sleep(1)  # give the client some prep time to set itself up.

# wss.authenticate()

# Subscribe to some channels
wss.ticker('BTCUSD')
wss.order_book('BTCUSD', prec='P3',len="100")
wss.ohlc('BTCUSD')

# Send a ping - if this returns silently, everything's fine.
wss.ping()

# Do something else
t = time.time()

while time.time() - t < 100:
    #print(time.time())
    #print(t)
    pass

if time.time() - t > 100:
    wss.restart(soft=False)
    wss.ticker('BTCUSD')
    wss.order_book('BTCUSD', prec='P3',len="100")
    wss.ohlc('BTCUSD')

# wss.stop()

#print(wss.tickers['BTCUSD'])
#print(wss.books['BTCUSD'].bids())  # prints all current bids for the
BTCUSD order book
#print(wss.books['BTCUSD'].asks())  # prints all current asks for the
BTCUSD order book

#wss1 = BtfxWssRaw(key=apikey,secret=apisecret,output_dir='data/',
#                  rotate_at='22:35',rotate_to='var/')
#wss1 = BtfxWssRaw(key=apikey,secret=apisecret,
#                  rotate_at='13:59')
#wss1.start()

Regards,
Nitin Gupta
9731423808

On Thu, Apr 27, 2017 at 7:29 PM, Nils [email protected] wrote:

Closed #7 #7.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/Aam05txakxQh4aLjxiW4vg4mg7BeXp4mks5r0J9UgaJpZM4NFSvU
.

Add state to Connection object

Currently, if the Connection hasn't been established yet, a
websocket._exceptions.WebSocketConnectionClosedException is thrown and the program crashes. Instead, this should be handled gracefully:

Either Prevent calls to the API while the Connection is still attempting to connect

Return a False value or similar to indicate failure.

Once connection is broken it won't reconnect

Hi,

I have this issue where by once a connection is broken (Restarted Bitfinex end or cloudflare error or something)... the connection is closed.. it trys reconnecting.. over and over again.. endlessly.. but never does.... just waits 10 seconds then connection closed then retrys.. over and over....

IF I kill the script then restart it it reconnects fine and works up until the next time it dies for some reason then again unable to restart...

Because the script doesn't actually die... just endlessly tries to reconnect.. I can't really run a bash script to check if its still running all the time..

Are you familiar with this issue ?

Order book & Raw Order books not working

This is due to an error within the Btfx API and/or their documentation.

Responses for these endpoints are identical, which makes it impossible to differentiate between their subscription confirmations, hence the data receveived for them ends up in the wrong method.

A ticket has been sent to bitfinex support to resolve this issue.

wrong plus/minus sign of volume

Sometime plus/minus sign of volume on trades method response data is wrong.

Evidence:
In this case, -4 should be +4.
(['te', [69742359, 1506137906000, -0.24, 3613.6]], 1506137906.7423086)
(['te', [69742360, 1506137906000, -0.00762595, 3613.5]], 1506137906.8536527)
(['tu', [69742359, 1506137906000, -0.24, 3613.6]], 1506137906.8545558)
(['tu', [69742360, 1506137906000, -0.00762595, 3613.5]], 1506137906.8882198)
(['te', [69742367, 1506137916000, -4, 3615.6]], 1506137916.9646268) <--- this volume should be +4
(['tu', [69742367, 1506137916000, -4, 3615.6]], 1506137916.9763083) <--- this volume should be +4

snapshot on https://cryptowat.ch/bitfinex/btcusd/1m at the same time:

Example Usage Errors in README.md

When trying to run the provided code in the example, the following line of code seems to cause issues:

wss.conn.wait():

If I try to run the code in it's origina form I get SyntaxError: invalid syntax.

If I remove the ' : ', I get AttributeError: 'WebSocketConnection' object has no attribute 'wait

When I comment it out, I get the following stack trace error:
Traceback (most recent call last):
File "C:/Users/Nietzsche/Google Drive/Project Cryptic/bitfinex_test.py", line 33, in
ticker_q = wss.tickers('BTCUSD') # returns a Queue object for the pair.
File "C:\Users\Nietzsche\Google Drive\Project Cryptic\py_venv\crypticpy\lib\site-packages\btfxwss\client.py", line 228, in tickers
raise KeyError(pair)
KeyError: 'BTCUSD'

I am trying to store btfxwssraw for storing data but it is not working

wss1 = BtfxWssRaw(key=apikey,secret=apisecret,output_dir='data/')
wss1.start()

using the following code , i am trying to store data ,
It created all the csv files , but it is somehow not storing anything.

Error in Readme example execution

I get the following error when running the example in the README:

>>> wss.conn.wait()
Traceback (most recent call last):

  File "<ipython-input-6-972ef3cfd4a6>", line 1, in <module>
    wss.conn.wait()

AttributeError: 'WebSocketConnection' object has no attribute 'wait'

If I replace this line by a time.sleep(60), i get the following error:

>>> wss.subscribe_to_ticker('BTCUSD')
ERROR:btfxwss.client:Cannot call subscribe_to_ticker() on unestablished connection!
INFO:btfxwss.connection:Connection Error - [Errno 110] Connection timed out
INFO:btfxwss.connection:Connection closed
INFO:btfxwss.connection:Attempting to connect again in 10 seconds.
INFO:btfxwss.connection:Connection opened
INFO:btfxwss.connection:Connection closed

Any ideas ?

Config settings not supported

Current methods have not been tested with custom config settings set via the config event.

An expected issue is that string values may be incompatible with methods that expect decimals instead.

Also, enabling sequencing may result in unexpected behaviour and/or errors

Authentication Not working

While the basic sign method has been written, authentication isn't working as of now.

Deploy dev to pypi

Release current dev branch to pypi.

Deploy Beta of next release to PyPi
Enable nightly builds

Auth handler

Hi
It looks like QueueProcessor has no auth handler, is it so? And does it mean, that currently I can not obtain auth data: my open orders, wallet and so on?

Write Tests to avoid Deployment Fuck-uppery

As detected by @rafaelcapucho in issue #18, there has been an error on pip install since the last release. Furthermore several other issues were raised, which shouldn't happen.

Therefore, basic testing should be deployed to keep shit from hitting the fan for each release.

AssertEach Endpoint subscribes and returns Data
Assert pip install does not fail
Assert Authentification works properly

Lack of state of authentication and reconnection

The current library lacks the state of authentication. It is still fine until the library reconnects to bitfinex without letting the rest of the code know. Is there any way to notify the other side of the program to re-issue authentication? Maybe a queue containing reconnection signals?

I am happy to implement it and make a PR if you are not available to do so. Let's agree on a way to achieve this if there isn't an existing way. Many thanks!

error code 10100 (auth: invalid)

Hello!,

When receiving this error, connection keeps working but only for unauthenticated channels.
Patched it like this:

=== modified file 'connection.py'
--- connection.py       2017-09-18 00:47:35 +0000
+++ connection.py       2017-09-18 01:55:39 +0000
@@ -47,6 +47,7 @@
 
         # Connection Handling Attributes
         self.connected = Event()
+        self.needsreauth = Event()
         self.disconnect_called = Event()
         self.reconnect_required = Event()
         self.reconnect_interval = reconnect_interval if reconnect_interval else 10
@@ -389,6 +390,7 @@
         """
         errors = {10000: 'Unknown event',
                   10001: 'Unknown pair',
+                  10100: 'Auth invalid',
                   10300: 'Subscription Failed (generic)',
                   10301: 'Already Subscribed',
                   10302: 'Unknown channel',
@@ -401,6 +403,8 @@
             # Unknown error code, log it and reconnect.
             self.log.error("Received unknown error Code in message %s! "
                            "Reconnecting..", data)
+        if data['code'] == 10100:
+            self.needsreauth.set()
 
     def _data_handler(self, data, ts):
         """Handles data messages by passing them up to the client.

So it is possible to check and solve from BtfxWss instance if wanted/necessary:

if self.conn.needsreauth.is_set():
  self.authenticate()

After reconnect, all keeps smooth again.

Best Regards,

python 3.4 compatibility?

Hi :)

for what python version is your code?
It seems 3.4 is not supported, I get:

entry = (*data, ts,)
                ^

SyntaxError: can use starred expression only as assignment target

Searching for this error shows that this is something that was fixed in 3.5, but not in 3.4.
What would be the 3.4 equivalent of it?

edit:
And a question: How exactly and where is the data stored? From your description it sounds like that somewhere a file with the orderbook information is stored. But where? And is it necessary?
I would like to know where, so I can delete it after closing websocket connection. Or is this not necessary?

Can't get orders, positions or wallets

Hi,
I can place orders fine but the code:
wss = BtfxWss(key, secret)
wss.start()
wss.authenticate()
wss.orders.empty()

Always returns True, how can I get my orders, positions and wallets using btfxwss?

wss.candles(pair, timeframe) adds timeframe to key...

wss.subscribe_to_candles('BTCUSD', '1m')
leads to
defaultdict(<class 'queue.Queue'>, {('candles', 'BTCUSD'): <queue.Queue object at 0x000002749A73E208>})
calling
wss.candles('BTCUSD', '1m')
leads to a key of
('candles', 'BTCUSD', '1m')
which does not match the queue's key...

DEBUG mode requires more detailed DEBUG messages

as can be seen in #22 , the debug log calls in the code aren't helpful - which isn't surprising, since the code barely has any log calls.

Error "cafile, capath and cadata cannot be all omitted"

When running this:

import logging
import sys
import time

from btfxwss import BtfxWss

logging.basicConfig(level=logging.DEBUG, filename='test.log')
log = logging.getLogger(__name__)

fh = logging.FileHandler('test.log')
fh.setLevel(logging.DEBUG)
sh = logging.StreamHandler(sys.stdout)
sh.setLevel(logging.DEBUG)

log.addHandler(sh)
log.addHandler(fh)

apikey = '....'
apisecret = '....'

wss = BtfxWss(key=apikey, secret=apisecret)
wss.start()
time.sleep(2)  # give the client some prep time to set itself up.

# wss.authenticate()

# Subscribe to some channels
wss.ticker('BTCUSD')
wss.order_book('BTCUSD')

# Send a ping - if this returns silently, everything's fine.
wss.ping()

# Do something else
t = time.time()
while time.time() - t < 10:
    pass

print(wss.tickers['BTCUSD'])
print(wss.books['BTCUSD'].bids())  # prints all current bids for the BTCUSD order book
print(wss.books['BTCUSD'].asks())  # prints all current asks for the BTCUSD order book

I got these:

/usr/bin/python3.6 /home/kepler/Workspace/FinexMonitor/main.py
Traceback (most recent call last):
  File "/home/kepler/Workspace/FinexMonitor/main.py", line 29, in <module>
    wss.ticker('BTCUSD')
AttributeError: 'BtfxWss' object has no attribute 'ticker'

Process finished with exit code 1

Log:

INFO:btfxwss.connection:Connection Error - cafile, capath and cadata cannot be all omitted
INFO:btfxwss.connection:Connection closed
INFO:btfxwss.connection:Attempting to connect again in 10 seconds.

might it be related to websocket-client/websocket-client#227 ?

Thx.

websocket._exceptions.WebSocketConnectionClosedException: socket is already closed.

After updating to the latest dev branch, after about 2 days of running i got this error...

Exception in thread Thread-3091435:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.6/threading.py", line 1182, in run
    self.function(*self.args, **self.kwargs)
  File "/usr/local/apache2/htdocs/Bots/btfxwss/connection.py", line 232, in send_ping
    self.socket.send(json.dumps({'event': 'ping'}))
  File "/usr/local/lib/python3.6/site-packages/websocket/_app.py", line 117, in send
    if not self.sock or self.sock.send(data, opcode) == 0:
  File "/usr/local/lib/python3.6/site-packages/websocket/_core.py", line 234, in send
    return self.send_frame(frame)
  File "/usr/local/lib/python3.6/site-packages/websocket/_core.py", line 259, in send_frame
    l = self._send(data)
  File "/usr/local/lib/python3.6/site-packages/websocket/_core.py", line 423, in _send
    return send(self.sock, data)
  File "/usr/local/lib/python3.6/site-packages/websocket/_socket.py", line 113, in send
    raise WebSocketConnectionClosedException("socket is already closed.")
websocket._exceptions.WebSocketConnectionClosedException: socket is already closed.

SyntaxError: can use starred expression only as assignment target

Traceback (most recent call last):
File "contrib/live_data_feeders/bitfinex_data_feeder.py", line 1, in
from btfxwss import BtfxWss
File "/home/arajguru/work/CCPlatform_Private/venvpython3/lib/python3.4/site-packages/btfxwss/init.py", line 1, in
from btfxwss.client import BtfxWss
File "/home/arajguru/work/CCPlatform_Private/venvpython3/lib/python3.4/site-packages/btfxwss/client.py", line 9, in
from btfxwss.connection import WebSocketConnection
File "/home/arajguru/work/CCPlatform_Private/venvpython3/lib/python3.4/site-packages/btfxwss/connection.py", line 266
self.q.put((event, data, *args))
^
SyntaxError: can use starred expression only as assignment target

Unable to Subscribe to Candle Channel

test_is_connected_decorator_works_as_expected (__main__.BtfxWssTests) ... ok
test_subscribing_to_data_works (__main__.BtfxWssTests) ... ERROR

======================================================================
ERROR: test_subscribing_to_data_works (__main__.BtfxWssTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\bxty9\Desktop\client_tests.py", line 27, in test_subscribing_to_data_works
    wss.tickers('BTCUSD').get(block=False)
  File "C:\ProgramData\Anaconda3\lib\site-packages\btfxwss\client.py", line 228, in tickers
    raise KeyError(pair)
KeyError: 'BTCUSD'

----------------------------------------------------------------------
Ran 2 tests in 13.095s

FAILED (errors=1)

Sleep time after `start` affects behaviour

I tried the sample code in the README. My issue is with the effect of setting sleep time in this line:

time.sleep(1)  # give the client some prep time to set itself up.

When keeping 1 second sleep time, I could receive server responses. When I then set sleep time to 5, values in wss.tickers and wss.books would no longer update no matter how long I waited:

print(wss.tickers['BTCUSD'])
print(wss.books['BTCUSD'].bids())  # prints all current bids for the BTCUSD order book
print(wss.books['BTCUSD'].asks())  # prints all current asks for the BTCUSD order book
[]
[]
[]

This behaviour seems like it should not be expected.

Also, NB: there is a ' missing in print(wss.tickers['BTCUSD])) in the README.

Thank you for sharing your code.

Nonce small

Hi
After wss.authenticate() I got the following traceback:

DEBUG:btfxwss.connection:_on_message(): Received new message {"event":"auth","status":"FAILED","chanId":0,"code":10114,"msg":"nonce: small"} at 1503428552.745091
DEBUG:btfxwss.connection:_system_handler(): Received a system message: {'chanId': 0, 'msg': 'nonce: small', 'event': 'auth', 'code': 10114, 'status': 'FAILED'}
DEBUG:btfxwss.connection:_system_handler(): Distributing {'chanId': 0, 'msg': 'nonce: small', 'code': 10114, 'status': 'FAILED'} to _response_handler..
DEBUG:btfxwss.connection:_response_handler(): Passing {'chanId': 0, 'msg': 'nonce: small', 'code': 10114, 'status': 'FAILED'} to client..
DEBUG:btfxwss.connection:_start_timers(): Resetting timers..
DEBUG:btfxwss.connection:_stop_timers(): Timers stopped.
ERROR:btfxwss.queue_processor:Dtype 'auth' does not have a response handler!

After changing the 'nonce = str(int(time.time() * 1000))' to 'nonce = str(int(time.time() * 1000000))' in client.py (line 467) all works fine

Improve design of `client` class methods

As pointed out in issue #55, the method signatures for the Client class are odd to look at.
The entire setup of methods appears a bit bolted on and redundant, and should be improved.

automatic restart when no heartbeat?

After few minutes I get the message:
WARNING:main:BtfxWss.heartbeats: Channel ('book', {'freq': 'F0', 'prec': 'P0', 'pair': 'BTCUSD', 'symbol': 'tBTCUSD'}) hasn't sent a heartbeat in 10.06259298324585 seconds!

And the orderbook does not update anymore.

Maybe implement an automate restart?

And can you imagine why I get this error? I started it two times, but both times after ~5 minutes I got no heartbeat anymore.

All I do is waiting and print best bid and best ask from time to time. I want to test for few hours, if orderbook is still consistent. Maybe I need to send a ping from time to time?

PS:
I commented out the two conflicting parts (python 3.4), but since they do not affect orderbook, everything should still work I guess.

edit:
I tried it now with sending a ping every 1 minute.
But according to logfile, after few minutes the "TimeoutError: Ping Command timed out!"
And then I also got no heartbeat anymore... hmm..

If it is relevant:
I did not install this module. I copied the classes.py and exception.py in two new python files, adjusted the exception import and added your example code:

if name == "main":

import sys
import traceback

logging.basicConfig(level=logging.INFO, filename='bitfinex_ws.log')
log = logging.getLogger(__name__)

fh = logging.FileHandler('bitfinex_ws.log')
fh.setLevel(logging.INFO)
log.addHandler(fh)

wss = BtfxWss(key='my_api_key', secret='my_api_secret')
wss.start()
time.sleep(1)  # give the client some prep time to set itself up.

# Subscribe to some channels
wss.order_book('BTCUSD')

# Send a ping - if this returns silently, everything's fine.
wss.ping()


count = 0
while True:
    time.sleep(2)
    bid = wss.books['BTCUSD'].bids[0]
    ask = wss.books['BTCUSD'].asks[0]
    if bid[0] > ask[0]:
        print("BID > ASK")
    count += 1
    if count>30:
        print("bid: {} ... ask: {}".format(bid,ask))
        wss.ping()
        count = 0
wss.stop()

Queue or LifoQueue

It would be nice to have a LifoQueue ticker to get always the most recent information.

Spell Check all the things!

There's likely a boat full of spelling errors and grammatical atrocities in the code - log messages, doc strings and comments all need to be reviewed.

Feel free to contribute!

connection time out

can you help me with problem of connection timeout, it does not connect to server, and can not get any data except pong message!

Review and Implement Changes of ABitfinex API Update

As described here:

http://docs.bitfinex.com/v2/docs/changelog

These go live today (31.03.17), but should only affect the client in a minor way.

order book processing

Hi,
It looks like current version of BtfxWss.subscribe_to_order_book and BtfxWss.unsubscribe_from_order_book does not process at all such parameters as precision, frequency and length. Of course they can be transferred via **kwargs. But, for example, BtfxWss.subscribe_to_raw_order_book and BtfxWss.unsubscribe_from_raw_order_book have parameter "prec".
I am not sure it is a bug but all these discrepancies between those procedures look strange from design point of view.

Minor issue with example in the README

wss.conn.wait(): --> this line is not valid in python

Also, I would suggest adding the imports of the required standard libraries so that a simple copy/paste is enough to run the code:

import logging
import time
import sys

send new_order but can not get return message

I am not clear the specific rules, when I add some function based on your code to trade, I send new order message, I can not get any message about the order. but how can I get the return message which i can known the status of my order. thanks!

pls change the following line in classes.py : line 1089

current line : date = time.strftime('%Y-%m-%d_%H:%M:%S')

new line : time.strftime('%Y-%m-%d_%H_%M_%S')

new line does not throw an error , old line does

Authenticated channels? (specifically trade)

First off, thank you so much for creating such an amazing and useful program - This has been enormously useful and an amazing resource.

I see that you noted that authenticated channels are a work in progress - what stage of development are they currently and is it possible to place orders with the current code?

Problem spotted getting Wallet balance for second time

Hello,
There is a problem with this code:

wss = BtfxWss(key=key, secret=secret) 
wss.start()
wss.authenticate()
while True:
  if not wss.wallets.empty():
    print(wss.wallets.get())
  else:
    print("No wallet update received")

Actual Results

First loop iteration: Revice correct wallet balance
Every other loop iteration, even after the balance change on BitFinex site: "No wallet update recived"

Expected Results

Well, the wallet should receive an update when it is changed

** EDIT**
So, basically no post request is being done to get the new wallet balance.
As @nlsdfnbch explained below there is no need to make a post request as the nature of the web-socket is to send updates automatically

API Endpoint for candle data needs renaming

Currently, subscribing to the endpoint candles requires calling the method BtfxWss.ohlc(), which is counter intuitive, since all other method names reflect their API counterpart.

Hence, renaming the methods is requested.

Relationship to nlsdfnbch/bitex?

Hi, after stalking you a little, I noticed your other project nlsdfnbch/bitex. It seems to be, or at least become a superset of this one. What are your plans? Why not link to the other one in the README? Anyways, thanks for both projects!

Cheers!

I try to create new order but nothing happens

I run this code:

socket = BtfxWss(key, secret)
socket.start()
time.sleep(1)
socket.authenticate()
order = {
"cid": 12345,
"type": "MARKET",
"symbol": "tBTCUSD",
"amount": "1.0",
"hidden": 0
}
socket.new_order(**order)
socket.stop()

And the order isn't created. How can I create new order using btfxwss?

Re-Subscribe to channels after Connection Time-Out

As pointed out by @8ick in #29:

I may have a related problem, when trying to establish a long-running websocket connection. I'm not sure if it is intentional or not, but at the moment it seems that the client does not resubscribe to the same channels it was connected to before a timeout or disconnect happened. No subscribed channel leads to endless reconnects due to timeouts. Is this behavior intended or do you consider an automatic renewal of subscriptions?

This should be handled properly.

Connection is not reopening once closed

Hello @nlsdfnbch :)

Ok, It ran on my VPS, it worked directly for 26 hours, then it died.

[root@arch-production phoenixsystem]# cat test.log | grep wss
INFO:btfxwss.connection:Connection opened
DEBUG:btfxwss.client:_subscribe: {'event': 'subscribe', 'channel': 'ticker', 'symbol': 'BTCUSD'}
DEBUG:btfxwss.client:_subscribe: {'event': 'subscribe', 'channel': 'ticker', 'symbol': 'IOTUSD'}
DEBUG:btfxwss.client:_subscribe: {'event': 'subscribe', 'channel': 'ticker', 'symbol': 'ETHUSD'}
DEBUG:btfxwss.client:_subscribe: {'event': 'subscribe', 'channel': 'ticker', 'symbol': 'LTCUSD'}
INFO:btfxwss.queue_processor:Subscription succesful for channel ('ticker', 'BTCUSD')
INFO:btfxwss.queue_processor:Subscription succesful for channel ('ticker', 'IOTUSD')
INFO:btfxwss.queue_processor:Subscription succesful for channel ('ticker', 'ETHUSD')
INFO:btfxwss.queue_processor:Subscription succesful for channel ('ticker', 'LTCUSD')
INFO:btfxwss.connection:Connection closed    (26 hous later, I know by looking at the graph)

So, the problem still exists in dev since it didn't even try to reconnect this time.

It is represented by the last "missing part" of the following 30m graph:

Are there any other info that might help you? Just let me know, Thanks!

Errors in `BtfxWss()._handle_book` and `Orders().call`

I put both errors in the same issue as I am unsure how both are linked and how one fix affects the other...

When running the sample code in the documentation, I get the following error:

Exception in thread Processing Thread:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/XXX/.local/lib/python3.5/site-packages/btfxwss/classes.py", line 399, in process
    self.handle_data(ts, data)
  File "/home/XXX/.local/lib/python3.5/site-packages/btfxwss/classes.py", line 605, in handle_data
    self.channels[chan_id](ts, chan_id, data)
  File "/home/XXX/.local/lib/python3.5/site-packages/btfxwss/classes.py", line 645, in _handle_book
    price, count, amount = order
ValueError: too many values to unpack (expected 3)

This fix in _handle_data seems to avoid having an error but I don't know if this is the behavior you intended:

    def _handle_book(self, ts, chan_id, data):
        """
        Updates the order book stored in self.books[chan_id]
        :param ts: timestamp, declares when data was received by the client
        :param chan_id: int, channel id
        :param data: dict, tuple or list of data received via wss
        :return:
        """
        label = self.channel_labels[chan_id][1]['pair']
        assert len(data) == 1
        data = data[0]
        if isinstance(data[0], list):
            # snapshot
            
            for order in data:
                price, count, amount = order
                side = (self.books[label].bids if amount > 0
                        else self.books[label].asks)
                side[str(price)] = (price, amount, count, ts)
        else:
            # update
            price, count, amount = data
            side = (self.books[label].bids if amount > 0
                    else self.books[label].asks)
            if count == 0:
                # remove from book
                try:
                    side.pop(str(price))
                except KeyError:
                    # didn't exist, move along
                    pass
            else:
                # update in book
                side[str(price)] = (price, amount, count, ts)

Using this fix, I then get the following error, in wss.books['BTCUSD'].bids():

>>> time.sleep(5); wss.books['BTCUSD'].bids()
Traceback (most recent call last):

  File "<ipython-input-107-35d8e9dec676>", line 1, in <module>
    wss.books['BTCUSD'].bids()

  File "/home/leo/.local/lib/python3.5/site-packages/btfxwss/classes.py", line 45, in __call__
    reverse=self._reverse)]

  File "/home/leo/.local/lib/python3.5/site-packages/btfxwss/classes.py", line 44, in <listcomp>
    return [self._orders[i]() for i in sorted(self._orders.keys(),

TypeError: 'tuple' object is not callable

In this case, changing self._orders[i]() to self._orders[i] seems to remove the error.

Shows syntaxError trying to run for the first time.

from btfxwss.classes import BtfxWss, BtfxWssRaw
  File "/home/username/dev/src/bitfinex_wss/btfxwss/classes.py", line 637
    entry = (*data, ts,)
                    ^
SyntaxError: can use starred expression only as assignment target

Maybe I'm doing something wrong? any idea why it giving this error? thanks.

Candles fetch error

I'm trying to fetch candles with timeframe more than 1m.
wss.candles(pair="BTCUSD", timeframe="15m")

With '1m' its ok, but if I try to use other timeframes like "15m" I have an Error:

Traceback (most recent call last): File "main.py", line 192, in <module> candles[kind]['queue'] = wss.candles(pair=PAIR, timeframe=kind) File "/Users/anton/anaconda/lib/python3.5/site-packages/btfxwss/client.py", line 278, in candles raise KeyError(pair) KeyError: 'BTCUSD'

I also try to fetch this data from Websocket Playground https://bitfinex.readme.io/v2/reference#ws-public-candle there is no problem.

orderbook validity

Hi nlsdfnbch,

Hope you are well.
I have been working on the python API implementation of several exchanges. But recently I ran into some trouble with exchanges like Bitfinex and Gemini. The problem is that when we try to maintain a real-time order book, we need all the messages and we need them in correct order. But websocket client sometimes could miss messages or receive out-of-order messages. Websocket API of Bitfinex and Gemini does not offer ways to verify the message, so our local order book could be wrong but we don't know it. (by the way GDAX API does not have this problem).
I'm just writing to see what do you think about this problem.

Blessings,
Seratna

Error during instalation, README not found

Hi, I'm getting this, thanks.

$ sudo pip3 install btfxwss
[sudo] password for user: 
Collecting btfxwss
  Downloading btfxwss-1.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-byx62y6k/btfxwss/setup.py", line 8, in <module>
        long_description=open('README.rst').read())
    FileNotFoundError: [Errno 2] No such file or directory: 'README.rst'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-byx62y6k/btfxwss/

The last version, download here directly seems to install, could you update the package on pypi if possible?

$ sudo pip3 install /tmp/btfxwss-master.zip 
[sudo] password for user: 
Processing /tmp/btfxwss-master.zip
Requirement already satisfied: websocket-client in /usr/lib/python3.6/site-packages (from btfxwss==1.0.1)
Requirement already satisfied: six in /usr/lib/python3.6/site-packages (from websocket-client->btfxwss==1.0.1)
Installing collected packages: btfxwss
  Running setup.py install for btfxwss ... done
Successfully installed btfxwss-1.0.1

Multi threading

Hi,
I'm newbie in multi threading and I want to achieve the behavior of the parallel channels processing in one class (with common fields). But when I use the common BtfxWss in different threads I get the following:

File "c:\python35\Lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "c:\python35\Lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "c:\python35\Lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "c:\python35\Lib\multiprocessing\popen_spawn_win32.py", line 66, in init
reduction.dump(process_obj, to_child)
File "c:\python35\Lib\multiprocessing\reduction.py", line 59, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects

So, sorry for so stupid question, but is this correct to use the same BtfxWss object in different processes or should I create new instance for each process?
Most likely I do not fully understand how multithreading works, but let me start with this question