fatihbakir / blockstore Goto Github PK

CMake 0.75% TypeScript 52.75% C++ 24.01% JavaScript 10.73% HTML 5.46% Shell 5.23% CSS 1.08%

blockstore's Introduction

Blockstore

A distributed key-value storage system built on top of blockchain technology™

About

The system is built on nodejs using TypeScript. The lower-level elements of the code such as the blockchain functions are implemented in C++17, thus we use packages that allow for an interface between the node event loop and executing C++ functions. src/app contains a demo application consisting of creating posts, and comments on those posts, which sends HTTP requests to instances of Blockstore server nodes located in src/bs/ts

Getting Started (Linux/Ubuntu)

Clone the repo

git clone [email protected]:FatihBAKIR/blockchain.git

Install the node packages

cd src/app
npm install
cd ../bs/cpp
npm install
cd ../ts
npm install

If not already installed, install cmake-js. NOTE: This will install it globally

sudo npm install -g cmake-js

If not already installed, install cmake version 3.9 or greater. NOTE: If your Linux version is a LTS version, the apt package may be too old. In this case, you'll need to download the latest stable version of cmake and build it from source

# Non-LTS Linux version
sudo apt-get install cmake

# LTS Linux version
# If you accidentaly installed the apt package of cmake but ended up needing to build it from source, remove the package
sudo apt remove cmake
# Download the latest stable release (https://cmake.org/download/), unpack the file, and cd into it
./configure
make -j8
sudo make install

Cmake will need a compiler supporting C++17 features. Check your gcc version, and if it isn't 7+, search for a package supporting gcc-7. If no results are found, Use a ppa to build it from source

# Check your gcc version
gcc --version

# If it isn't 7+, search to see if an apt package is available
apt search gcc-7

# If no package was avaialble, use this PPA to discover and install 7+
sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
sudo apt-get update
apt search gcc-7
sudo apt install g++-7 -y
export CXX=g++-7
export CC=gcc-7

Compile the C++ code. NOTE: If you previously attempted to build it (and failed), you'll need to remove your build/ directory

rm -rf build/
cmake-js build

Install TypeScript. NOTE: This will install it globally

sudo npm install -g typescript

Compile the TypeScript code to JavaScript

tsc

Use node to start the demo application server (app/build/index.js) or an instance of the Blockstore server (bs/ts/build/index.js)

node app/build/index.js
node bs/ts/build/index.js

blockstore's People

Contributors

Stargazers

Watchers

Forkers

blockspacer

blockstore's Issues

Signing

The app or BS servers should sign the blocks or individual operations

Config file

Broadcast it along with the chain to a new node so it can configure itself accordingly
Add some mechanism to broadcast changes to the config to everyone else (changes should be rare so may no be needed for a limited implementation)

Node replication via Playback

When a new node joins the network and makes first contact with some node in good health/standing, the original node should send the new node the blockchain (either its exact copy or its copy minus the 6 most recent blocks). From this, the new node should replay the history of the blockchain locally, which will effectively build its copy of the KV store from scratch to reflect the current state of data in the system

Blockchain persistence

We should store the blockchain on the disc

Garbage collection/blockchain compression

Since we care about the payload of old blocks (and thus can't rely on merckle trees to reduce chain size), we need to implement some form of garbage collection. One approach to this would be:

Pick a block in the chain as the "checkpoint". You'd want this to be a block sufficiently committed ie 6 deep
Make a snapshot of the current state of the KV store
Starting from block 0 until the checkpoint block, go through each operation of each block and compare the operation's info to the state of the KV store to see if it is still relevant. Some examples:
- PUT: Check to see if its key still exists; if it doesn't, we know it must have eventually been deleted so we can throw this operation away, but if it does keep it
- UPDATE: Check to see if its key still exists; if it doesn't, we know it must have eventually been deleted so we can throw this operation away. If it does, compare the values; if they're different, a later update must have occurred to change the value of this one so throw this operation away, but if they're the same then keep it
- DELETE: Throw away deletes outright, since their PUTS and UPDATES will have also been thrown away
Pack as many operations as possible according to the config parameters into each block, then rehash the block and move onto the next one
Once done, broadcast the new compressed chain (as well as which block was its checkpoint block) to everyone else to update their chains

There's definitely some subtleties to consider here like the long-term uniqueness of the keys used (the approach above assumes keys will always be unique). This approach also has the implication that, while before we could see that a data entry was created, maybe updated, then deleted even though it isn't reflected from just the KV store at present time, after this we will no longer know that that data ever existed and it will be truly unrecoverable.

One nice thing is that while this node goes into GC mode to do all this, the other nodes can continue doing work uninterrupted, then once they get the <new_chain, checkpoint> packet, simply cut off the old chain before the checkpoint, point the prevHash of the checkpoint block at the end of the new chain, and continue about things. The GC node then just needs to hear from someone about the work that was done while it was GC'ing and catch up.