snth / numismatic Goto Github PK
View Code? Open in Web Editor NEWCollector for digital assets
License: MIT License
Collector for digital assets
License: MIT License
Brave new coin (for example) requires authentication via tokens that are passed as a custom HTTP header. The requests library therefore need the ability to use custom headers
Some exchanges use BTC others use XBT. Similarly for MNR and XMR.
See this comment: #56 (comment)
That should make it more clear to get an overview of what's available for new contributors and help avoid merge conflicts.
How about this folder structure (this is just a suggestion, please alter):
This is just a starting point for conversation...
See #56 (comment)
A good example is CryptoCompare - it has many methods that are specific to itself (for example, get_historical_hour
or get_info
. The latter is surprising, but I could not find it on brave new coin.
What we need to do is discuss what specific methods we need (see #35) and move those to the base.
For methods specific, we should add the feed/exchange name as a prefix. For example, cryptocompare_get_info
. The trick would come in when we want to run this on the cli. So running coin -f cryptocompare info
would run cryptocompare_get_info
, and coin -f bravenewcoin xyz
would run bravenewcoin_get_xyz
In particular I would like to describe how to use the ~/.coinrc
file.
What is the modern way of easily adding documentation? I would prefer to write in markdown rather than rst.
Command "python setup.py egg_info" failed with error code 1 in /private/tmp/pip-install-14pvn2be/toolz/
I have no idea what could it be
The output_stream writer writes every single event. The raw stream does have a batch size but this is of a fixed size. This should use the Stream.timed_window
operation but that requires running an event loop in a separate thread.
This issue is for creating that separate thread event loop and then having constant time interval flushes to disk.
Right now, we have two concepts:
Both of these names seem non-optimal:
Honestly, I am not even sure if there is a better naming, but lets start the debate shall we?
We need to determine what methods are common to every exchange and/or feed. Only those methods should be in the base class. This will take understanding which will come about after we implement a few exchanges/feeds.
The use of symbol
isn't consistent across exchanges, e.g. BTCUSD
vs BTC-USD
vs USDT_BTC
. To avoid this I think we should rather record asset
and currency
separately.
Each symbol
may also be better referred to as pair
.
These are used together in the same place so can probably be combined or split in a different way.
The following fails:
$ coin collect --help
Traceback (most recent call last):
File "/opt/venv-numismatic/bin/coin", line 11, in <module>
load_entry_point('numismatic', 'console_scripts', 'coin')()
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 1085, in invoke
allow_interspersed_args=False)
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 621, in make_context
self.parse_args(ctx, args)
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 880, in parse_args
value, args = param.handle_parse_result(ctx, opts, args)
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 1404, in handle_parse_result
self.callback, ctx, self, value)
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 78, in invoke_param_callback
return callback(ctx, param, value)
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 809, in show_help
echo(ctx.get_help(), color=ctx.color)
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 496, in get_help
return self.command.get_help(self)
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 830, in get_help
self.format_help(ctx, formatter)
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 845, in format_help
self.format_options(ctx, formatter)
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 859, in format_options
rv = param.get_help_record(ctx)
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 1608, in get_help_record
rv = [_write_opts(self.opts)]
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 1605, in _write_opts
rv += ' ' + self.make_metavar()
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 1298, in make_metavar
metavar = self.type.get_metavar(self)
File "/opt/venv-numismatic/lib/python3.6/site-packages/click/types.py", line 140, in get_metavar
return '[%s]' % '|'.join(self.choices)
TypeError: sequence item 0: expected str instance, NoneType found
Currently the output_stream writing is handled in cli:collect. The cli module should just be a thin wrapper to expose the API and the actual business logic should be handled elsewhere. There is similar logic in handling the raw_output streams. This should be factored out into common code in a collectors module.
The syntax for auto_envvar_variables is obscure to find but I eventually got it as PREFIX_FUNCTION_VARIABLE
from values-from-environment-variables
e.g. the function "info" with flag "--assets" expects a value like $ export NUMISMATIC_INFO_ASSETS=ETH
And you do have correct syntax in cli.py, if one wanted to use the auto prefix feature.
coin(auto_envvar_prefix=ENVVAR_PREFIX)
Unfortunately, the feature is restrictive in that it requires the function name in the middle i.e. NUMISMATIC_INFO_ASSETS, when we just want NUMISMATIC_ASSETS for use in info and prices.
Since it doen't work neatly for assets and currencies, then you could remove the feature, provided you don't need it for any other variables (where the function name is the middle is appropriate). That would be a single line code change to just have coin()
at the end. And you can keep using your f-string style throughout as envvar=f'{ENVVAR_PREFIX}_ASSETS'
, which is the most economical in this situation without using the auto_envvars_prefix
feature..
Let me know what you think.
Currently the writes to disk use blocking io operations. This will likely be a problem on high volume feeds.
Change this to use the aiofiles
package.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.