Giter Club home page Giter Club logo

bds's Introduction

BDS

logo

Introduction

JD Cloud Blockchain Data Service (BDS) is a realtime data aggregating, analyzing and visualization service for chain-like unstructured data from all kinds of 3rd party Blockchains

Splitter is the key module of Blockchain Data Service (BDS) and provides data analysis capability.

Splitter is responsible for consuming blockchain data from message queue (kafka) and inserting data into persistent data storage services (relational database, data warehouse, etc.) for further processing

Architecture

Architecture

Environment Deployment

Install BDS

Environment initialization

Before compiling and running BDS, you must install go's compilation environment locally: go install

Install Splitter steps

  1. Set the path of project : $GOPATH/src/github.com/jdcloud-bds/bds/
  2. Inputgo build -v github.com/jdcloud-bds/bds/cmd/bds-splitter,compile to get executable file bds-splitter
  3. Build new configuration file splitter.conf, see /config/splitter_example.conf configuration file template
  4. Run program ./bds-splitter -c splitter.conf

Install confluent and kafka

Install kafka

See kafka

Modify config/server.properties
  • message.max.bytes=1048576000

Install confluent

see confluent

Unzip the confluent package and run Confluent REST Proxy

Modify /etc/kafka-rest/kafka-rest.properties
  • max.request.size = 1048576000
  • buffer.memory = 1048576000
  • send.buffer.bytes = 1048576000

Database

Database we now support SQL Server, PostgreSQL, you can choose one as a data storage method.

SQL Server

Buy JCS For SQL Server

PostgreSQL

Buy JCS For PostgreSQL

After you run the database, you need to manually create new database and use the database name initialization splitter.conf.

Install Grafana

See Grafana Official

Source code

Splitter Modules

Development Steps

  1. Define the data structure of Kafka messages.
  2. Define table structure.
  3. Analyze Kafka message and store data in database.

Contributing

Contributing guide

License

Apache License 2.0

Project Demonstration

Blockchain Data Service

bds's People

Contributors

pangang1992 avatar sallery-x avatar tibelf avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bds's Issues

panic: Reuse of exported var name: eth

(NOTE: Please use this template while reporting a bug and provide as much info as possible)

Bug description
when add JustitiaSplitter to bsd/splitter/spliter.go file, compilation can be passed, running will generate an painc error: Reuse of exported var name: eth

Expected behavior
Justitia splitter code runs normally and can parse block information

Steps to reproduce the bug
Create a new chain of splitters, added to the bds/splitter/splitter.go file Run() function,start and run .

Environment:go

  • Version:1.11.1
  • Cloud provider or hardware configuration:
  • OS :Mojave 10.14.5
  • Others:

``

configfile:
#占用cup核心数
[global]
max_process = 1

#堆栈监控
[profiling]
enable = true
host = 0.0.0.0
port = 6062

#内部数据监控
[metric]
enable = false
host = 0.0.0.0
port = 6063
path = /metrics

=============================== justitia ==================================

[justitia]
#是否开启 justitia 数据splitter
enable = true
#是否开启数据库
database_enable = true
#数据库worker缓存大小
database_worker_buffer = 8192
#数据库worker数量
database_worker_number = 1
#一次请求区块链数据的最大块数,400000块之前可以设置大一些,比如300
max_batch_block = 30
#justitia 全节点的地址
endpoint = http://127.0.0.1:47768
#运行 justitia 全节点设置的用户名
user = 0xc3b5e92111484503b98c0283206ff2f4239fb06b
#运行 justitia 全节点设置的密码
password = 123
#justitia 数据校验规则文件地址
json_schema_file = /etc/bds-splitter/schema/justitia.json↩
#justitia 数据校验是否开启
json_schema_validation_enable = false

#justitia 定时任务配置
[cron.justitia]
update_meta_expr = @every 1m

#justitia kafka 配置
[kafka.justitia]
enable = true
#topic = justitia
topic = justitia

kafka 客户端标示

client_id = justitia-client-1

kafka 消费组标示

group_id = justitia-group

kafka 服务的地址

broker_list = localhost:9092
buffer_size = 1000
return_errors = true

#justitia 数据库配置
[database.justitia]
#数据库类型,sql server为mssql,postgre为postgres
type = postgres
#数据库的访问地址
host = 127.0.0.1
#数据库的端口信息
port = 5432
#数据库的库名,需要初始化好,创建表和导入数据用
database = justitia
#数据库的访问账号
user = postgres
#数据库的访问账号密码信息
password = 123
timezone = Asia/Shanghai
max_open_conns = 500
max_idle_conns = 100
sql_log_file = /var/log/bds-splitter/justitia-sql.log
debug = false

max_idle_conns = 100
sql_log_file = /var/log/bds-splitter/justitia-sql.log
debug = false

=============================== log ==================================

#普通日志配置
[logging_normal]
writers = console,file
log_file = /var/log/bds-splitter/normal.log
log_level = debug
caller_level_skip = 5
format = short
max_size = 100m
daily = true

#错误日志配置
[logging_detail]
writers = console,file
log_file = /var/log/bds-splitter/detail.log
log_level = debug
caller_level_skip = 5
format = long
max_size = 100m
daily = true

eth->transaction 表中的 ‘V’ 字段 ‘char(2)’ 不够

(NOTE: Please use this template while reporting a bug and provide as much info as possible)

Bug description
启动eth私链测试时,‘V‘ 的长度为3,导致 eth transaction 写入失败
Expected behavior

Steps to reproduce the bug

Environment: eth私链

  • Version:当前最新bds测试
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
  • Others:

support xlm

(NOTE: Please use this template for submitting new feature)

Describe the feature request:
add xlm data
Why is this needed:

Additional context:

无法消费数据

(NOTE: Please use this template while reporting a bug and provide as much info as possible)

Bug description
splitter conf 中配置btc全节点运行地址,若为*...*:8333则会报错,若为kafka rest proxy的端口,则不报错,但kafka rest中报错。
ERROR Unhandled exception (io.confluent.rest.exceptions.KafkaExceptionMapper:123)
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of java.lang.String out of START_ARRAY token

Expected behavior
数据库仅生成表,没有数据插入
Steps to reproduce the bug
数据写入kafka正常,开启splitter无效,找不到区块block
Environment:

  • Version:
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
  • Others:

splitter无法读取kafka数据的疑问

您好,我刚开始学习区块链这方面的知识。按照您提供的文档我已经搭建了一套环境,比特币节点启动,kafka也受到了消息。启动splitter后,数据库中也自动生成了相关表项。但是没有数据,这是我很疑惑,希望您给予指点!谢谢!
××××××××××××××××××××××KafkaRESTProxy里的信息只有POST,没GET××××××××××××
[2020-03-22 21:05:20,259] INFO 192.168.*.** - - [22/三月/2020:13:05:20 +0000] "POST /topics/btc HTTP/1.0" 200 120 2 (io.confluent.rest-utils.requests:62)

××××××××××××××××××××splitter的错误日志一直报这个错××××××××××××××××××××
2020-03-22 21:02:43.436 ERROR [github.com/jdcloud-bds/bds/splitter/btc.(*BTCSplitter).CheckBlock:117] - Post http://192.168.*.*:8333:8332: invalid URL port "8333:8332"

×××××××××××××××××××信息日志里报如下的warning××××××××××××××××××××××××
2020-03-22 21:02:43.384 DEBUG - splitter btc: checking block 101105
2020-03-22 21:02:43.385 WARN - splitter btc: can not find previous block 101104
2020-03-22 21:02:43.385 DEBUG - splitter btc: get latest block -1 from database

××××××××××××××××××××××××××这是我splitter.conf的比特币相关部分×××××××××××××

=============================== btc ==================================

[btc]
#是否开启 btc 数据splitter
enable = true
#是否开启数据库
database_enable = true
#数据库worker缓存大小
database_worker_buffer = 8192
#数据库worker数量
database_worker_number = 1
#一次请求区块链数据的最大块数,400000块之前可以设置大一些,比如300
max_batch_block = 300
#btc全节点的地址
endpoint = http://192.168.*.*:8333:8332
#运行 btc 全节点设置的用户名
#user = [rpc 访问账号]
#运行 btc 全节点设置的密码
#password = [rpc 访问密码]
#btc数据校验规则文件地址
json_schema_file = /etc/bds-splitter/schema/btc.json↩
#btc数据校验是否开启
json_schema_validation_enable = false

#btc定时任务配置
[cron.btc]
update_meta_expr = @every 1m

#btc kafka 配置
[kafka.btc]
enable = true
topic = btc

kafka 客户端标示

client_id = btc-client-1

kafka 消费组标示

group_id = btc-group

kafka 服务的地址

broker_list = 192.168..:9092
buffer_size = 1000
return_errors = true

务必烦请您百忙之中答疑解惑。这对我真的很重要!!万分感谢!!!!

消费数据解析太慢的问题

我是消费btc的数据,一开始应该是因为交易少,所以数据处理非常快。
消息保留时间是168个小时,现在消费到了140000个块,24小时勉强消费10000个block,这样的速度太慢了,不知有什么方法可以解析快一些。
期待您的回复。

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.