Giter Club home page Giter Club logo

eosio_memento's Introduction

Memento: lightweight EOSIO history solution

In essence, Memento is a MariaDB database containing transaction traces from an EOSIO blockchain (such as EOS, Telos or WAX), and providing convenient indexes for quick searching.

A typical use case for Memento is tracking incoming and outgoing payments for a specific account, and checking their reversibility.

The database may contain traces for all contracts on the blockchain, or it may be limited to a specific subset of accounts. Dapps can set up their own instances, while public infrastructure providers may offer the service to a wider audience.

The tool supports active-standby redundancy with two writer processes: the standby process records its data in BKP_TRACES table, and monitors the master activity. Once it notices that the master stopped updating the database, it takes over the active role and continues writing the blockchain events. The old master switches to standby mode in this case.

The standby writer is also cleaning up transactions that are older than a specified retention period. If you are running Memento in a single-writer mode, you can launch the memento_dataguard service that will clean the old transactions.

Memento needs one or two EOSIO state history feeds. It uses Chronicle software for decoding the state history.

Installation

Installation instructions for an Ubuntu 20.04 host. In this example, a single writer is taking a WAX state history feed.

## Build Chronicle

apt install -y gnupg software-properties-common
wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | apt-key add -
apt-add-repository 'deb https://apt.kitware.com/ubuntu/ focal main'
add-apt-repository -y ppa:ubuntu-toolchain-r/test
apt update && apt install -y git g++-8 cmake libssl-dev libgmp-dev zlib1g-dev
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 800 --slave /usr/bin/g++ g++ /usr/bin/g++-8

mkdir /opt/src
cd /opt/src

wget https://boostorg.jfrog.io/artifactory/main/release/1.67.0/source/boost_1_67_0.tar.gz
tar -xvzf boost_1_67_0.tar.gz
cd boost_1_67_0
./bootstrap.sh
nice ./b2 install -j10


cd /opt/src
git clone https://github.com/EOSChronicleProject/eos-chronicle.git
cd eos-chronicle
git submodule update --init --recursive
mkdir build
cd build
cmake ..
nice make -j 10 install

cp /opt/src/eos-chronicle/systemd/chronicle_receiver\@.service /etc/systemd/system/
systemctl daemon-reload


## Database engine

apt-get update && apt-get install -y mariadb-server mariadb-client

## Perl modules for the writer

apt install -y cpanminus libjson-xs-perl libjson-perl libmysqlclient-dev libdbi-perl libwww-perl make gcc
cpanm --notest Net::WebSocket::Server
cpanm --notest DBD::MariaDB

## Memento writer

git clone https://github.com/cc32d9/eosio_memento.git /opt/eosio_memento
cd /opt/eosio_memento
cp systemd/*.service /etc/systemd/system/
systemctl daemon-reload

## initialize the MariaDB users
sh sql/create_db_users.sh

## create the database for WAX history
sh sql/create_memento_db.sh memento_wax

## writer service. Here 8806 is the TCP port is where Chronicle will send its output.
## You need to pick the port number that is free on your host.
## The first writer has always id=1, and the second writer should have id=2 if you
## run it in dual-writer mode.
echo 'DBWRITER_OPTS="--id=1 --port=8806 --database=memento_wax --keepdays=1"' >/etc/default/memento_wax1
systemctl enable memento_dbwriter@wax1
systemctl start memento_dbwriter@wax1


## Chronicle initialization. Here host and port point to the EOSIO state history source
mkdir -p /srv/memento_wax1/chronicle-config
cat >/srv/memento_wax1/chronicle-config/config.ini <<'EOT'
host = 10.0.3.1
port = 8080
mode = scan
skip-block-events = true
plugin = exp_ws_plugin
exp-ws-host = 127.0.0.1
exp-ws-port = 8806
exp-ws-bin-header = true
skip-table-deltas = true
skip-account-info = true
EOT

# You need to initialize the Chronicle database from the first block
# in the state history archive. See the Chronicle Tutorial for more
# details. You may point it to some other state history source during
# the initialization. Here we launch it in scan-noexport mode for faster initialization.
/usr/local/sbin/chronicle-receiver --config-dir=/srv/memento_wax1/chronicle-config \
 --data-dir=/srv/memento_wax1/chronicle-data \
 --host=my.ship.host.domain.com --port=8080 \
 --start-block=186332760 --mode=scan-noexport --end-block=186332800

# Once it stops at the end block, launch the service
systemctl enable chronicle_receiver@memento_wax1
systemctl start chronicle_receiver@memento_wax1

# the database will catch up to the head block, and will be ready fort use.
journalctl -u memento_dbwriter@wax1 -f

If you want to launch Memento in dual-writer mode, you need to add a line into SYNC table and start another writer with --id=2 option. It makes sense to run the second writer on an independent host, with an independent state history source.

# adding the second writer
mysql memento_wax --execute="INSERT INTO SYNC (sourceid, block_num, block_time, irreversible, is_master, last_updated) values (2,0, '2000-01-01',0, 0, '2000-01-01')"

If you run the service in single-writer mode, use the dataguard to delete old traces:

# enabling dataguard for data pruning
echo 'DBWRITER_OPTS="--database=memento_wax --keepdays=7"' >/etc/default/memento_dataguard_wax
systemctl enable memento_dataguard@wax
systemctl start memento_dataguard@wax

Example queries

# get the current Last irreversible block number (LIR)
SELECT MIN(irreversible) FROM SYNC;


# get all system token transfers related to the account,
# starting from specific block number, but not newer than LIR
SELECT seq, ACTIONS.block_num, block_time, trx_id, trace from TRANSACTIONS
JOIN ACTIONS USING(seq) JOIN RECEIPTS USING (seq) JOIN TRACES using (seq)
WHERE ACTIONS.contract='eosio.token' AND ACTIONS.action='transfer' AND
  RECEIPTS.block_num>186201662 AND
  RECEIPTS.block_num < (SELECT MIN(irreversible) FROM SYNC) AND
  RECEIPTS.account_name='arenaofglory';


# get all transactions related to a specific account, starting from
# specified sequence number (including reversible transactions which might
# get dropped or reordered afterwards)
SELECT seq, TRANSACTIONS.block_num, block_time, trx_id, trace from TRANSACTIONS
JOIN RECEIPTS USING (seq) JOIN TRACES using (seq)
WHERE 
  RECEIPTS.account_name='ysjki.c.wam' AND
  RECEIPTS.seq > 50662607733;  

Public access

See MEMENTO_PUBLIC_ACCESS.md.

Acknowledgments

This work was sponsored by EOS Amsterdam block producer.

Copyright 2022 [email protected]

eosio_memento's People

Contributors

cc32d9 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.