Giter Club home page Giter Club logo

bitshares-explorer-api's Introduction

BitShares Explorer REST API

BitShares Explorer REST API allows your programs to query the blockchain.

https://explorer.bitshares-kibana.info/apidocs/

Installation

The following procedure will work in Debian based Linux, more specifically the commands to make the guide were executed in Ubuntu 18.04 with Python 3.7.

Elasticsearch node

Some API calls make use of elasticsearch plugins for Bitshares. This plugins are elasticsearch and es-objects.

For elasticsearch installation and usage tutorial please go to: https://github.com/bitshares/bitshares-core/wiki/ElasticSearch-Plugin.

To avoid installation the API comes with public elasticsearch node that can be updated from config.

BitShares node

This API backend connects to a BitShares witness_node to get data. Additionally from elasticsearch API makes use of the following bitshares plugins:

  • market_history
  • grouped_orders

The node must have asset_api and orders_api enabled.

api-access.json:

{
   "permission_map" :
   [
      [
         "*",
         {
            "password_hash_b64" : "*",
            "password_salt_b64" : "*",
            "allowed_apis" : ["database_api", "network_broadcast_api", "history_api", "asset_api", "orders_api"]
         }
      ]
   ]
}

To install a bitshares node please refer to: https://github.com/bitshares/bitshares-core/blob/master/README.md

You can use/change public bitshares API nodes for this by updating the config.

Install BitShares Explorer API and dependencies

Install python and pip if you dont have them:

apt-get install -y python python-pip

Clone the app:

git clone https://github.com/bitshares/bitshares-explorer-api
cd bitshares-explorer-api/

Install virtual environment and setup:

pip install virtualenv 
virtualenv -p python3 wrappers_env/ 
source wrappers_env/bin/activate

Deactivate with:

deactivate

Install dependencies in virtual env activated:

pip install -r requirements/production.pip

Note: If you have errors in the output about websocket you may need to also do:

apt-get install python-websocket

Note: If you see a problem similar to:

 WARNING:connexion.options:The swagger_ui directory could not be found.
    Please install connexion with extra install: pip install connexion[swagger-ui]
    or provide the path to your local installation by passing swagger_path=<your path>

You need to execute:

pip install connexion[swagger-ui]

Usage

Simple running

In order to simply test and run the backend api you can do:

export FLASK_APP=app.py
flask run --host=0.0.0.0

Then go to apidocs with your IP:

http://127.0.0.1:5000/apidocs/

Nginx and uwsgi

In a production environment, when multiple requests start to happen at the same time, flask alone is not enough to handle the load. Nginx and uwsgi are alternatives to host a production backend.

Install nginx:

apt-get install nginx

Install uwgsi:

pip install uwsgi

Create config file in /etc/nginx/sites-available:

server {
    listen 5000;
    server_name 185.208.208.184;
    location / {
        include uwsgi_params;
        uwsgi_pass unix:/tmp/app.sock;
    }
}

Create symbolic link to sites-enabled and restart nginx:

ln -s /etc/nginx/sites-available/api /etc/nginx/sites-enabled/api
/etc/init.d/nginx restart

Now api can be started with:

(wrappers) root@oxarbitrage ~/bitshares/bitshares-explorer-api # uwsgi --ini app.ini

Profiler

To activate profiler use:

PROFILER_ENABLED=true flask run

Then you will be able to access profiling data at http://localhost:5000/profiler.

By default the profiler is not protected, to add basic authentification add username and password in config.py or using environment variables PROFILER_USERNAME and PROFILER_PASSWORD.

Development

To run the server in development mode to have an auto reload on code change:

FLASK_ENV=development flask run

Run all tests:

PYTHONPATH=. pytest

This will also run API tests (using Tavern), that needs an local server to run, so make sure your development server is started.

To run one specific test:

PYTHONPATH=. pytest -k test_ws_request

Or for API tests:

PYTHONPATH=. py.test tests/test_api_explorer.tavern.yaml -k get_asset_holders_count

You can run API tests on a non localhost server using the command:

PYTHONPATH=. py.test tavern-global-cfg=your_customized_environment.yaml tests/test_api_explorer.tavern.yaml

See tests/local_urls.yaml to see how to define a new environment.

And for non regression see non_reg/README.md

bitshares-explorer-api's People

Contributors

ety001 avatar knaperek avatar oxarbitrage avatar zapata avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

bitshares-explorer-api's Issues

/committee_votes does not return voting result for non voting proxies.

On testnet data:

$ curl 'http://localhost:5005/committee_votes' | jq '.'
[
  [
    "committee-member-1",
    "1.5.11",
    "1.2.29:Y",
    "1.2.6:-",
    "1.2.4986:-",
    "1.2.3299:-",
    0,
    0,
    0,
    0,
    0,
    0
  ],
  [
    "committee-member-2",
    "1.5.12",
    "1.2.29:Y",
    "1.2.6:-",
    "1.2.4986:-",
    "1.2.3299:-",
    0,
    0,
    0,
    0,
    0,
    0
  ],
  [
    "committee-member-4",
    "1.5.14",
    "1.2.29:Y",
    "1.2.6:-",
    "1.2.4986:-",
    "1.2.3299:-",
    0,
    0,
    0,
    0,
    0,
    0
  ],
  [
    "committee-member-3",
    "1.5.13",
    "1.2.29:Y",
    "1.2.6:-",
    "1.2.4986:-",
    "1.2.3299:-",
    0,
    0,
    0,
    0,
    0,
    0
  ],
  [
    "committee-member-7",
    "1.5.17",
    "1.2.29:Y",
    "1.2.6:-",
    "1.2.4986:-",
    "1.2.3299:-",
    0,
    0,
    0,
    0,
    0,
    0
  ],
  [
    "committee-member-6",
    "1.5.16",
    "1.2.29:Y",
    "1.2.6:-",
    "1.2.4986:-",
    "1.2.3299:-",
    0,
    0,
    0,
    0,
    0,
    0
  ],
  [
    "committee-member-5",
    "1.5.15",
    "1.2.29:Y",
    "1.2.6:-",
    "1.2.4986:-",
    "1.2.3299:-",
    0,
    0,
    0,
    0,
    0,
    0
  ],
  [
    "init2",
    "1.5.2",
    "1.2.29:Y",
    "1.2.6:-",
    "1.2.4986:-",
    "1.2.3299:-",
    0,
    0,
    0,
    0,
    0,
    0
  ],
  [
    "init0",
    "1.5.0",
    "1.2.29:Y",
    "1.2.6:-",
    "1.2.4986:-",
    "1.2.3299:-",
    0,
    0,
    0,
    0,
    0,
    0
  ],
  [
    "init1",
    "1.5.1",
    "1.2.29:Y",
    "1.2.6:-",
    "1.2.4986:-",
    "1.2.3299:-",
    0,
    0,
    0,
    0,
    0,
    0
  ],
  [
    "init10",
    "1.5.10",
    "1.2.29:Y",
    "1.2.6:-",
    "1.2.4986:-",
    "1.2.3299:-",
    0,
    0,
    0,
    0,
    0,
    0
  ]
]                                                                                                                                                                           

There should not be 0.

Response `schema` not specified in YAML files

In the Swagger YAML files, response schema specs are not given. For example, for /account_name:

"/account_name":
get:
description: Get account name by ID
operationId: api.explorer.get_account_name
parameters:
- in: query
name: account_id
default: 1.2.0
type: string
required: true
description: Account ID (1.2.X)
responses:
'200':
description: 1 account name
'500':
description: Error processing parameters
tags:
- api
- account

In responses, it gives only description for each status code. The schema spec (https://swagger.io/docs/specification/2-0/describing-responses/) is missing.

This may not be a big issue in some use cases. However, we will not be able to use the API with some Swagger clients who rely on the schema to unmarshal JSON (like Bravado). This makes it harder for non-technical data analysis people to retrieve and make use of the API.

Implement a 'get_historic_feed_price' call

Requesting: Verbose keys for 'get_most_active_markets' and 'top_proxies'

Hey,

I was looking to integrate both 'get_most_active_markets' and 'top_proxies' into the Bitshares Google Assistant, but the keys for each of the items are simply 0,1,2,3,4 etc, and I don't know what they're representing.

Would we be able to provide informative key strings please? It's OK to not have the key strings for say 'top_uias' because it's just the uia name & the amount traded, but for functions which return many rows it's not immediately clear what they're supposed to represent.

Thanks for all the effort you've put into developing this repo, much appreciated & keep up the great work! ๐Ÿ‘

Optimizations for api_explorer_get_top_proxies

The current list of proxies, api_explorer_get_top_proxies, is considered to be used in the UI. However, this command today pulls a lot of data and seems to take a while to run.

It would require some optimization.

  • Pagination, only return 20 results per page

Create command api_explorer_get_top_proxies_count to return the total amount.

Related issue bitshares/bitshares-ui#2932

fix readme installation instructions

two ways to install:

  • manual
  • docker

for manual need to fix current README, is old and need some adjustments.

for docker need to make docs full, the installation is having some issues.

/accounts pagination

/accounts endpoint need to accept start and limit so we can get paginated results. node call to holders already support this.

Null op_object

When using /get_account_history sometimes entry['operation_history']['op_object'] is null (None), I got this only on ES wrapper v2. Example:

curl -X GET --header 'Accept: application/json' 'https://wrapper.elasticsearch.bitshares.ws/get_single_operation?operation_id=1.11.1000908945'

Create api to convert operation_type number to name

It is common when developing an app using this api that you end up with an operation type, this can come from a get_block call, from elasticsearch, etc.

A new api call can be created that match id with name, should be something as:

get operation_type_name(0) returns: transfer
get operation_type_name(1) returns: limit_order_create_operation
...

Operation names can be obtained from: https://github.com/bitshares/bitshares-core/blob/master/libraries/chain/include/graphene/chain/protocol/operations.hpp#L50-L105 where transfer_operation is 0.

If created, will save some time to the application developer avoiding the creation of this map on its own app.

operation_type filter not working properly

Try get 0 operation:

curl -X GET --header 'Accept: application/json' 'http://185.208.208.184:5000/es/account_history?account_id=1.2.25010&operation_type=0&size=3&from_date=2015-10-10&to_date=now&sort_by=block_data.block_time&type=data&agg_field=operation_type'

Results contains different operation types, not only 0.

Cache /top_proxies

/top_proxies is used intensively in the account page of the explorer, and it takes lot of time to compute. But as we load holders and accounts information only every XXX minutes, the computed data could be cached instead of recomputed.

add nginx to docker

The dockerfile currently run the service by flask. this is ok for development but on production flask cant handle large amounts of traffic so service need to run behind nginx. i think it is possible but not sure. what do you think @knaperek ?

for reference here is my nginx file:

root@alfredo:~/bitshares-munich/explorer/repo/bitshares-python-api-backend# cat /etc/nginx/sites-available/myapp 
server {
    listen 5000;
    server_name 23.94.69.140;
    location / {
        include uwsgi_params;
        uwsgi_pass unix:/tmp/api.sock;
    }
}
root@alfredo:~/bitshares-munich/explorer/repo/bitshares-python-api-backend# 

and the command i am using to start the service inside a screen session(probably not the best):

root@alfredo:~/bitshares-munich/explorer/repo/bitshares-python-api-backend# uwsgi --ini api.ini

Migrate to Python 3

Python 2.7 will reach end of life in January 1st, 2020, we should move to Python 3+.

/top_proxy does not takes open orders into account

top_proxy, computes the voting weight from the object-balance ES index, which does not include open orders. However the voting weight of a proxy takes into account all the balance + open order of the delegated accounts.

Ideally voting_weight should be retrieved from core node as the computation may change (see voting decay BSIP).

refactor get_objects to use ES when object is op

current implementation of lastnetworkops will call _get_object : https://github.com/oxarbitrage/bitshares-python-api-backend/blob/master/api.py#L522 with operation_id

this situation makes non possible to run all the explorer backend with 1 node as we need plugin elasticsearch and account_history activate at the same time. this is not possible in 1 node.

need to change _get_object to get data from ES when object type is operation or make new call for this purpose.

Imported data is inconsistent.

The way the data is imported in database is inconsistent, as there are blocks ingested by the core while all the assets/accounts/.. are retrieved.

Either the core should stop synchronize while the export is done, or the data should be loaded from a snapshot.

schema.txt is not up to date: column and table missing

Traceback (most recent call last):
File "./postgres/import_referrers.py", line 33, in
cur.execute(query)
psycopg2.ProgrammingError: relation "referrers" does not exist
LINE 1: SELECT rid FROM referrers ORDER BY rid DESC LIMIT 1

Proposal for API V2.

Currently there are many possible improvements to the API.

Here is a placeholder to regroup all those changes, to propose a new /v2/ base path.

  • /operation* return a list, it should not, just the operation
  • /account, /full_account returns a list, it should not, just the account
  • /get_assets, /get_asset_and_volume returns a list, it should not, just return the asset
  • /get_object returns a list, it should not, just the object
  • /get_settle_orders is wierd, parameters do not match core doc
  • /getlastblocknumbher typo, to be coherent is should be /get_last_block_number
  • /top_*: should returns objects with named fields instead of array of array (see also https://github.com/oxarbitrage/bitshares-explorer-api/issues/5)
  • /*_votes: should return objects with named fields instead of array of array
  • /get_witnesses: actually return an array of array of one object, should just be array of object
  • get_most_active_markets: should return objects with named fields instead of array of array (see also https://github.com/oxarbitrage/bitshares-explorer-api/issues/5)
  • (es_wrapper) /get_account_history: operation_type should be an int10
  • (es_wrapper) /get_single_operation: should return object instead of array
  • /get_account_history_pager_elastic is incoherent with /get_operation_full_elastic (following https://github.com/oxarbitrage/bitshares-explorer-api/pull/28)
  • (es_wrapper) do json.load on op and result fields
  • put all udf calls under /udf

In order to avoid breaking current api the proposal is to make a new /v2 endpoint. If clients could be updated we can stay on the current endpoint.

Allow get_asset_holders calls with pagination

Right now the call
http://23.94.69.140:5000/get_asset_holders?asset_id=1.3.1453
lists the top 20 holders. The backend supports pagination in the form of start and amount, that could be userfull to add here as well, e.g.
http://23.94.69.140:5000/get_asset_holders?asset_id=1.3.1453&start=20,amount=10

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.