-
The nodes are deployed with Terraform and run in Kubernetes. Refer to these for more details:
- Template repository for deploying Terraform configurations: https://github.com/blockchain-etl/blockchain-terraform-deployment
- Terraform configuration files for running blockchain nodes: https://github.com/blockchain-etl/blockchain-terraform
- Kubernetes manifests for running blockchain nodes: https://github.com/blockchain-etl/blockchain-kubernetes
-
The blockchain data is polled periodically from the nodes and pushed to Google Pub/Sub. Refer to these for more details:
- Article explaining how to subscribe to public blockchain data in Pub/Sub: https://medium.com/google-cloud/live-ethereum-and-bitcoin-data-in-google-bigquery-and-pub-sub-765b71cd57b5
- Streaming blockchain data to Google Pub/Sub or Postgres in Kubernetes: https://github.com/blockchain-etl/blockchain-etl-streaming
- CLI tools for polling blockchain data from nodes: https://github.com/blockchain-etl/ethereum-etl, https://github.com/blockchain-etl/bitcoin-etl, https://github.com/blockchain-etl/eos-etl.
-
Airflow DAGs export and load blockchain data to BigQuery daily. Refer to these for more details:
- Article explaining how the DAGs work: https://cloud.google.com/blog/products/data-analytics/ethereum-bigquery-how-we-built-dataset.
- Airflow DAGs for exporting, loading, and parsing blockchain data: https://github.com/blockchain-etl/ethereum-etl-airflow, https://github.com/blockchain-etl/bitcoin-etl-airflow, https://github.com/blockchain-etl/eos-etl-airflow.
-
The blockchain data is pulled from Pub/Sub, transformed and streamed to BigQuery. Refer to these for more details:
- Dataflow pipelines for connecting Pub/Sub topics with BigQuery tables: https://github.com/blockchain-etl/blockchain-etl-dataflow.
-
Various applications of the public blockchain data:
- Blockchain streaming analytics: https://github.com/blockchain-etl/blockchain-streaming-analytics.
- Parsing Ethereum smart contract data: https://towardsdatascience.com/how-to-get-any-ethereum-smart-contract-into-bigquery-in-8-mins-bab5db1fdeee.
- Blockchain analytics in BigQuery: https://github.com/blockchain-etl/awesome-bigquery-views.
- Clustering Ethereum addresses: https://towardsdatascience.com/clustering-ethereum-addresses-18aeca61919d.
- Twitter bot posting anomalous transactions: https://twitter.com/BlockchainETL.
- ...
blockchain-etl / blockchain-etl-architecture Goto Github PK
View Code? Open in Web Editor NEWBlockchain ETL Architecture
License: MIT License