streetyoga / atapi Goto Github PK

Normaly i would do:

from binance.spot import Spot as Client
client = Client()
exchange_info = client.exchange_info()
symbols = set()
for s in exchange_info['symbols']:
    if s['symbol'].endswith('USDT'):
        symbols.add(s['symbol'])

But as I already use the circulating supply data, I will go with that. it has 80 pairs less, which seem only be some leveraged and very exotic tokens pairs

Name could be changed to ata - Alogorithmic Trading API, or atapi, (like the AT Attachment Packet Interface)
Considerations:
short names like ata could mean less typing , but something like import atapi as at could also work...

• getpass (Python Standard Library) https://docs.python.org/3/library/getpass.html
• cryptography https://pypi.org/project/cryptography/
  already dependency of python-binance
• passlib https://passlib.readthedocs.io/en/stable/
• as json, i.e. atapi.json:

{"key" : "your_api_key","secret":"your_secret_key"}

• export as environment variables:

  export BINANCE_API_KEY=<your_api_key>
  export BINANCE_API_SECRET=<your_secret_key>

• python-dotenv
  standard for Flask, don't see the point atm, as all it does is reading an .env file and setting it as environment variables?
• python-decouple, same, from Django

Check all formulas and implement how to calculate with those DataFrames.
For now I limited calculations with [:3] to the first 3 symbols as those have no NaN values.
Possible solutions:

• limit calculations to 1 year into the past
• calculate individualy from first price, i.e. dropna()
• don't concatenate DataFrames and find another solution

Frontends with pyproject.toml Support

• pip
• build

Backends which read pyproject.toml:

• Flit
  Simple
  For pure python projects without extension modules (C/C++)
• setuptools

Other Build Tools

• waf
• A-A-P: only C automation? But written by Bram Moolenaar :)
• Bitbake
• Buildout

CI

• github actions (research build options)
• buildbot
• toxi https://github.com/tox-dev/tox
• https://docs.python-guide.org/scenarios/ci/
• travis-ci.com

Testing

• pytest: has also CI and benchmarks as plugins

Linting

• flake8

Everyhing should be nicely grouped as atapi.algo. and within other future instance classes.
Seperated data fetch into own class.

Evaluation has to be made which plugins will be future proof, all candidates are updated regularly.
Research future prospects on new functionality like trading NFT's...

• Binance public API connector
  Advantages: Official from Binance, easy to understand
  Disadvantages: lightweight More work has to be done. Only Spot trading, no futures, but could be built upon...
  binance API docu
  This Documentation includes info about NFT endpoints:
  API Documentation Site

• python-binance
  Advantages: Very mature and robust, almost all features from binance API implemented
  Disadvantages: Third party

• CCXT
  Advantages: Supports a streamlined API for hundreds of exchanges, only small adjustments for ticker-names needed,
  futureproof for arbitrage trading
  Disadvantages: may be slow with implementing new features like NFT trades on binance

Table looks good.
investigate coins with 0 supply and remove.
remove NaN fiat rows.

symbols and marketcap limited with [:3] due to formulas having to be adapted for NaN values.

Algorithms used in Algorithmic Trading

• A* Search
• Simple Moving Average (SMA)
  Momentum Trading or Trend following:
  When moving average 50 falls below moving average of 200, sell (and short futures), otherwise, buy and go long.

import pandas as pd  
import numpy as np
pd.DataFrame.rolling(window = 7, min_periods = 7).mean()
pd.DataFrame.rolling(window = 25).mean()
pd.DataFrame.rolling(window = 99).mean()

# Common Movin Averages:
price.rolling(window = 50)
price.rolling(window = 200)

price.rolling(window = 5)
price.rolling(window = 10)
price.rolling(window = 20)
price.rolling(window = 100)

# Fibonacci:
price.rolling(window =  (5, 8, 13, 21 ...) )

month_ret = pd.DataFrame.resample('M', kind = 'period').last().pct_change().dropna()
for years in [1, 3, 5, 10, 20]:
      # Anualized Mean Returns For Last ... Years.
      pd.DataFrame['month_ret'].rolling(years * 12).mean() * 12 
      # Anualized Risk For Last ... Years.
      pd.DataFrame['month_ret'].rolling(years * 12).std() * np.sqrt(12) 

• Exponential Moving Average (EMA)
  reacts faster to recent price changes
  Typical short term are 12 and 26 days, long term 50 and 200 days

price.ewm(span = 10, min_periods= 10).mean()

• Double Exponential Moving Average (DEMA)
  DEMA = 2*EMA - EMA(EMA)
• Expanding Windows
  Fixed lower bound
  Mean of price up to this point in time

price.expanding().mean()

• Correlation

returns.corr()

• Rolling Correlation

returns['BTC'].rolling(36).corr(returns['ETH'])

Logarithmic Returns

mean_log_ret =  np.log(df / df.shift(1)).dropna().mean()

I.e. price is 130, 110, 120, 90, 80:

returns are -15%, 9%, -25%, -11%
mean: -12.13%

# Actual Price
investment * np.exp(periods * mean_log_ret) 

130 * np.exp(4*-0.1213) = 80

Geometric Average

[(1+R1)(1+R2 )(1+R3)…*(1+Rn)]**(1/n)-1
where:
R=Return
n=Count of the numbers in the series

I.e. returns are respectively
30%, 10%, 20%, -10%, and -80%

(1.3 *1.1 * 1.2 * 0.9 * 0.8)**(1/5)-1

The geometric average annual return == 4.3%

Other Statistic Measures

• Confidence Interval
  https://en.wikipedia.org/wiki/Confidence_interval

Fix.
some ideas:

name = "user"
say = input("Say: ")
sys.stdout.write("\033[F")
print (name, "said:", say)

from getpass import getpass

name = "user"
say = getpass(prompt="")
print (name, "said:",say)

AF class and ATFShell into seperate file, merge if possible.

Different testsuites to consider:

External:

• pytest
  using atm, also supports unittest
• nose (unittest wrapper)

Standard Library

• unittest
• doctest tests inside functions as docstring interactive python sessions.

Write test conditions for every single method/function accessed in the cmd-interface, saves time in the long run

This script can do it but needs elevated privileges:

import time
import os

try:
    import ntplib
    client = ntplib.NTPClient()
    response = client.request('pool.ntp.org')
    os.system('date ' + time.strftime('%m%d%H%M%Y.%S',time.localtime(response.tx_time)))
except:
    print('Could not sync with time server.')

print('Done.')

Only sums up symbols in use, change to all polled symbols from binance.

Caches results forever, check all methods and eventually replace with one of this options:

classmethod not needad as class is instantiated anyway for interpreter access with atapi.algo. So no problem that Python 3.11 removed chaining classmethod descriptors has been removed. Initial loading time slower without? Could be rate limiting because of testing cli and module at same time. Time it and check if true.

@functools.lru_cache(maxsize=2) # caches 2 calls

from cachetools import cached, TTLCache
@cached(cache=TTLCache(maxsize=2, ttl=900)) # caches for 15min.

Check if setter property could be useful, i.e.:

@property
def assets_close(self):
    return ...

@assets_close.setter
def assets_close(self, ...):
    ...
    
assets_close = ...

Other means to to speed up code execution:
Numbas jit decorator:

from numba import jit
@jit(nopython=True)

instead of having two methods, return dataframe as tuple before and after creating column. and access by index.

As bare minimum it should include:

• Class Diagram
• Sequence Diagram

For consistency, all other outputs have to be sorted.

Not sure by which criteria:

• MarketCap
• Weight of Portfolio
• Alphabetically

as they are of no use atm.
Mainnet balance still a long way to go.
API keys not needed as a lot of work has to be done with statistics.

Instructions on how to setup a your API key for the spot testnet:

Binance Spot API Key

Export as environment variables:

Bash/Sh/Zsh:

export BINANCE_API_KEY='your_api_key'           
export BINANCE_API_SECRET='your_secret_key'      

# Check
echo $BINANCE_API_KEY
echo $BINANCE_API_SECRET

Powershell:

$env:BINANCE_API_KEY = 'your_api_key'          
$env:BINANCE_API_SECRET = 'your_secret_key'    

# Check
$env:BINANCE_API_KEY
$env:BINANCE_API_KEY

from comp/algo.py:

    @staticmethod
    def balance():
        """Balance and kline fields for selected assets."""
        return pd.json_normalize(client_test.account()['balances'])

from __main__.py

    @staticmethod
    def do_balance(arg):
        """Balance and kline fields for selected assets."""
        table = Table('Assets Balance')
        table.add_row(algo_class.balance().astype({'free': 'float', 'locked': 'float'}).to_string(
            float_format=lambda _: '{:.2f}'.format(_), index=False))
        console.print(table)

At a later stage, Jupyter Notebooks will be abandoned and the script will be converted to a Python module.
There will be a transitioning time, where the convenient Jupyter interface will be kept, but afterwards development will focus on pleasant functionality in Python's REPL, possibly with rich support, kind of like shutil
Object oriented loading of modules, calling various algorityms.
requirements.txt provided, but maybe at the begining dependencies as %pip install pandas for the notebook

Binance has no market cap data through their public API.
Custom requests query has to be written.

Here is a workaround:
Howto Calculate Market Cap Data with Binance API

Originally posted by @miron in #16 (comment)

For annual probabilistic sharpe ratio, a dataframe sample of yearly returns (yearly log returns) over at least 2 years is needed:
Algo().assets_close().resample('Y')
Consider daily/weekly/monthly returns.

pct_change() returns to log returns:
np.log1p(df['Yearly Returns']

@staticmethod
def psr(df, benchmark=0):
    sharpe = stats['Sharpe']
    skew = scipy.stats.skew(df['Yearly Returns'])
    kurtosis = scipy.stats.kurtosis(df['Yearly Returns'])
    n = len(df)
    sigma_sr = np.sqrt(
        (1/(n-1)) * (1 + 0.5*sharpe**2 - skew*sharpe+(kurtosis/4)*sharpe**2))
    ratio = (sharpe - benchmark) / sigma_sr
    psr = scipy.stats.norm.cdf(ratio)
    return psr

All annualized stats don't make sense on testnet, as testnet resets once a month.
Consideration to switch to api.binance.com for data that does not need API key.

py -m setuptools_scm :
Automatically increments to next version, dev2, dev3, for every push, until you tag it for release (git tag v0.1.3)
(0.1.2.dev2+g8d01dfa), needs to be resolved before pushing new build to pypi

• you have documented stats, correlation, returns but what about the rest?
  clearer indication where to find thouroughfull documentation
  indicate that excessive documentation can be found right in the interface.
  Documentation of endpoints at / API call

• what are the inputs we need to feed to the API for each method
  CLI (line interpreter) on top of the JSON API,
  calculations: Algo(),
  Flask (or other framework) API logic: API(), Data().

• To get started, you need to have a clear idea of what to test
  when you present your project, specify exactly what you need to be tested.
  Portfolio Maker, Crypto Portfolio Maker, CPM
  project to categorize cryptocurencies
  A tool that uses moderm portfolio theory that helps to categorize cryptocurencies based on those metrics.
  Rebranding as crypto lexicon that educates thoroughly but in nontechnical terms about those properties and builds your portfolio in a short time, but long enough to make an informed decision.
  Pulls data from Binance and calculates modern portfolio theory metrics like alpha, beta, sharpe ratio, annualized risk/return, categorizes them into defensive, non-cyclical coins according to market needs,..
  Feedback from professionals is very important, also from laypersons not knowing anything but would like to invest a little, learn where to start, educate themselves just enough to make an informed decision, all in one interface.
  Binance is too complicated for majority.

• Coinpanion tries, but:

  • Not flexible enough, third party,
  • Markets diversification but offers strongly correlated portfolios ('theme based portfolios') .
  • Cautious portfolio is the only one with a loss 🤔.
  • Landing page goes straight to KYC.
  • You don't own any crypto, withdrawal only in euros, same as revolut. Uses every single buzzword but does not explain
  • Dubious proprietary crypto score.
  • Claim weekly rebalancing with 13 cryptocurrencies, makes 676 transactions a year per portfolio, calculated with 0.1% per transaction, that's 67.6%, yet they only charge 2% annual fee.

• Focus on requests and binance, ccxt for arbitrage trading with support over hundred exchanges is too much, trading comes later but in simple form just buying the uncorrelated portfolio, one click solution, i.e. invest 100$, diversify in 10 coins a 10$.

• There are too many trading frameworks and TA , see forks, QuantConnect i.e. works even with Binance

• A simple program for people who don't know anything about crypto coughing up 10 sorted different coins sharing the same universe, by specific criteria of which you choose ONE.

• Oracle which asks you about some properties, multiple choice, and then coughs up the top 10-15 with that properties. Also educate about the importance of diversification, so recommend other uncorrelated properties. Build up to a portfolio of 10-15 uncorrelated crypto currencies.
  Sort for the best weights, sharpe, risk, returns,
  Categorize by consensus mechanisms like POW, POS, DPOS, PTOS, DeFi, NFT, ERC-20, BEP-20, eosio.token...

• I wanted to take a list simple list of crypto pairs but could understand how i would input them

  • By correlation:
    Let user choose from 10 pairs from 10 groups. every member of a specific group is strongly correlated but minimaly correlated to other groups
    Run SLSQP algo over 10 out of 300 user chosen binance coins/USDT annualized over 1 year, show optimal weights and comparison of the tangency portfolio with market cap portfolio, equal weighed, capital weighted and price weighted solutions.
  • By performance metrics:
    Top 10 Coins with:
    highest sharpe, least risk, highest returns, highest/lowest var, sysvar, unsysvar, beta, CAPM, alpha
    Top 10 by optimal weights if portfolio would consist of all 300
  • By Consensus Mechanism:
    POW, POS, DPOS. POTS
  • Capital Metrics:
    Volume, Most Expensive per token, Dollar Volume
  • Other:
    lineage, Defi, NFT, Tokens vs Coins

• so no, did not proceed to test
  make people interested enough to install that damn thing :)

• I think you should make a “How to” step by step in readme
  Emphasize the availability of the CLI help with meaningful examples
  Emphasize the availability of documentation in the WIKI, or find a less hidden solution.
  Make the readme more interesting, less details, more up to go and how to use it quick. Simple and catchy elevator pitch.

• TI (coarse filter needs to be implemented before expensive computation)
  above 200 day SMA
  NVT ratio (Network valute to transaction, MarcetCap/24hVolume, P/E-like metric for Crypto)

  • https://woobull.com/introducing-nvt-ratio-bitcoins-pe-ratio-use-it-to-detect-bubbles/

• 10-15 good uncorrelated-return streams (Ray Dalio)

• You can do websockets in Flask too: Flask SocketIO

• Flasks own request instance of Request class and requests

• Rewrite of cmd to client that gets data from API endpoints instead of importing Algo class.

• API documentation in get request at root, i.e. api.street.yoga/
  Returns all endpoints and servertime from class, but split it up to appropriate resource classes and save to database.
  check postman Code snippets for http.client or requests payload
  See this list of http-clients, it includes asynchronious libraries and even one that offers HTTP 2.0 support.
  Using one of the asynchronious ones is a strong consideration.

• Websockets under Windows: pyOpenSSL expects trusted root set from twisted but gets none:

  In PowerShell:

  binance/binance-connector-python#57 (comment)

  PS C:\> $env:SSL_CERT_FILE=(py -m certifi); py yourscript.py

  In Script:

  binance/binance-connector-python#57 (comment)

  import os, certifi, win32api
  from binance.websocket.spot.websocket_client \
      import SpotWebsocketClient as WebsocketClient
  os.environ['SSL_CERT_FILE'] = certifi.where()
  ws_client = WebsocketClient()
  
  def message_handler(message):
      print(message)
  
  ws_client.start()
  
  ws_client.mini_ticker(
      symbol='bnbusdt',
      id=1,
     callback=message_handler,
  )
  
  # Combine selected streams
  ws_client.instant_subscribe(
     stream=['bnbusdt@bookTicker', 'ethusdt@bookTicker'],
      callback=message_handler,
  )
  
  win32api.SetConsoleCtrlHandler(lambda _: ws_client.stop(), True)

  Cryptographic Authority wants to kill pyopenssl, avoid, as well as frameworks that depend on it, like twisted
  python-trio/trio#1145 (comment)

• An alternative to Flask is FastAPI, uses asyncio

• Removed binance-connector because being dependant on twisted

• Pyramid and Django probably overkill

Full Interpreters

• python interpreter https://github.com/python/cpython
• ptpython https://github.com/prompt-toolkit/ptpython
• IPython https://github.com/ipython/ipython
• bpython https://github.com/bpython/bpython

Interactive

• cmd https://docs.python.org/3/library/cmd.html
• Python Prompt Toolkit https://github.com/prompt-toolkit/python-prompt-toolkit

TUI

• Urwid https://github.com/urwid/urwid

CLI

• argparse https://github.com/python/cpython/blob/main/Lib/argparse.py
• Python Fire https://github.com/google/python-fire
• Click https://github.com/pallets/click
• Typer https://github.com/tiangolo/typer
  Some basic prompt features
• docopt https://github.com/docopt/docopt
• cement https://github.com/datafolklabs/cement
• cliff https://github.com/openstack/cliff
  There is an interactive Demo App included

GUI

• Gooey https://github.com/chriskiehl/Gooey
  Zero effort, just @Gooey decorator, works with argparse
  wxpython based, which is broken atm.
• cliff https://github.com/openstack/cliff

split into methods so that not everything is run alltogether.

drop and set as multi row index, now i select with .iloc[:, 6]

Discussed in #16

Exact same configuration of test project passes test, by collecting __init__.py
Some more confrontation with pytest is needed