Giter Club home page Giter Club logo

cpp-filecoin's Introduction

Filecoin (cpp-filecoin)

C++17 implementation of blockchain based digital storage

Filecoin is a decentralized protocol described in spec

Minimal hardware requirements

Node minimal parameters:

Hard disk space: at least 200 GB
RAM: 8 GB
OS: Linux(Ubuntu), macOS. Other operating systems builds are not supported yet, so running on them may be unstable.

Miner minimal parameters:

CPU: 8+ cores
Hard Disk space: 256 GiB of very fast NVMe SSD memory space + 1 TiB of slow HDD memory space
RAM: 16 GB
GPU: GPU is highly recommended, it will speed up computations, but Mixing AMD CPUs and Nvidia GPUs should be avoided.

You can also read about lotus minimal requirements in filecoin-docs

Dependencies

All C++ dependencies are managed using Hunter. It uses cmake to download required libraries and do not require downloading and installing packages manually.

Target C++ compilers are:

  • GCC 9.3.0
  • Clang 9.0.1
  • AppleClang 12.0.0

Lotus CLI

fuhon-node supports subset of lotus CLI commands.
You may download pre-built binaries or build lotus from source.
Lotus CLI introduction.

Rust

filecoin-ffi provides pre-built binaries for some platforms.
If they are unavailable, you need Rust compiler to build them.
Rust installation instruction.

Build

# clone project
git clone --recursive https://github.com/filecoin-project/cpp-filecoin
# configure cmake
cmake cpp-filecoin -B cpp-filecoin/build
# build and install fuhon-node and fuhon-miner
cmake --build cpp-filecoin/build --target install
# check that fuhon-node and fuhon-miner are now available
fuhon-node --help
fuhon-miner --help

Usage

Interopnet node

Create the following fuhon-interopnet/config.cfg file

# use interopnet profile, corresponds to "make interopnet" lotus target
profile=interopnet

# enable debug logs to see sync progress
log=d

# bootstrap peers from https://github.com/filecoin-project/lotus/blob/master/build/bootstrap/interopnet.pi
bootstrap=/dns4/bootstrap-0.interop.fildev.network/tcp/1347/p2p/12D3KooWLGPq9JL1xwL6gHok7HSNxtK1Q5kyfg4Hk69ifRPghn4i
bootstrap=/dns4/bootstrap-1.interop.fildev.network/tcp/1347/p2p/12D3KooWFYS1f31zafv8mqqYu8U3hEqYvaZ6avWzYU3BmZdpyH3h

Start node

fuhon-node --repo fuhon-interopnet --genesis docker/interopnet/genesis.car
# you can omit --genesis flag after first run
fuhon-node --repo fuhon-interopnet

To use lotus CLI add --repo flag

lotus --repo fuhon-interopnet net peers

Mainnet node (from snapshot)

Download mainnet snapshot (docs).

LATEST_SNAPSHOT=$(curl -sI https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car | perl -ne '/x-amz-website-redirect-location:\s(.+\.car)/ && print $1')
curl -o mainnet-snapshot.car $LATEST_SNAPSHOT

Create following fuhon-mainnet/config.cfg file

# use downloaded snapshot file (do not delete that file)
use-snapshot=mainnet-snapshot.car

# bootstrap peers from https://github.com/filecoin-project/lotus/blob/master/build/bootstrap/mainnet.pi
bootstrap=/dns4/node.glif.io/tcp/1235/p2p/12D3KooWBF8cpp65hp2u9LK5mh19x67ftAam84z9LsfaquTDSBpt

Start node (first run may take some time)

fuhon-node --repo fuhon-mainnet --genesis cpp-filecoin/core/docker/mainnet/genesis.car

Docker-compose example

docker-compose up

CodeStyle

We follow CppCoreGuidelines.

Please use clang-format 11.0.0 with provided .clang-format file to autoformat the code.

Maintenance

Maintainers: @zuiris, @turuslan, @Elestrias, @ortyomka, @wer1st, @Alexey-N-Chernyshov

Tickets: Can be opened in GitHub Issues.

Hunter cache upload

If you have access and want to upload to hunter-binary-cache, you need to add your GitHub token with read:packages and write:packages permissions.
To generate GitHub token follow the instructions.

export GITHUB_HUNTER_USERNAME=<github account name>
export GITHUB_HUNTER_TOKEN=<github token>

cpp-filecoin's People

Contributors

ahmedelkashev avatar alexey-n-chernyshov avatar art-gor avatar deni64k avatar eigendude avatar ekovalev avatar elestrias avatar harrm avatar igkrsh avatar kamilsa avatar markuu-s avatar masterjedy avatar mingela avatar ortyomka avatar sergkaprovich avatar turuslan avatar warchant avatar zuiris avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cpp-filecoin's Issues

Restore deals in case of failure.

If deal failure is not critical - it should be continued.
The task is to decide what failure is critical and how many times it should be retried.
Also deal flow should be reworked in a such way that it restores state and tries to continue the flow.

Testing: Miner scenarios

  • Window PoSt block
  • Blocks with window post should be accepted by others nodes. It means that our sectors are accepted by the net.
  • Winning PoSt block
  • Pledge sector
  • 32 GiB sector sealing
  • Turn down the node, turn it on and continue sealing
  • Storage Deal and Retrieval Deal
  • One storage and one retrieval deal
  • Many simultaneous deals
  • Several in a row (in one sector)
  • Stress test - many in one moment, check where the boundaries are
  • Fast Retrieval
  • Check if miner works correctly with "Fast retrieval" flag

cpp-ledger

The library provides access to hardware USB ledgers

Adapt Fuhon API to work with Ledger

First of all we need to adapt Fuhon API to work with local wallet and ledger.

Secondly, Fuhon has different wallet API with Lotus, necessary to resolve it.

Lotus Wallet API:

  • WalletBalance
  • WalletSign
  • WalletSignMessage
  • WalletVerify
  • WalletDefaultAddress
  • WalletSetDefault
  • WalletValidateAddress

Fuhon Wallet API:

  • WalletBalance
  • WalletDefaultAddress
  • WalletHas
  • WalletImport
  • WalletNew
  • WalletSetDefault
  • WalletSign
  • WalletVerify

Read and AddPiece via Remote worker

Add the ability to Read and Add piece

The problem is that we need to transmit large amounts of data via jsonrpc

Possible solution: do that via HTTP when PieceData encodeFrom other side, decode from other side via server handler

Remove worker task TTReadUnsealed

The task was present in sector-storage repo. But it was removed when sector-storage became the part of Lotus. Fuhon code contains the task TTReadUnsealed.
The task is to check that the task type is not used anymore and remove task type TTReadUnsealed and all dependend code.

build without Hunter

I'm trying to package this for NixOS. I can provide every dependency manually. Can I have some quick tips on how to excise Hunter and let cmake find all the dependencies?

Incidentally, Nix or NixOS probably solves the exact set of problems Hunter is being used for, as far as I can tell. You may want to look into using it; it can similarly install and use a version-pinned build environment on any Linux distro.

Write README

Write README file describing main ideas of project and add references to sample projects

Sync messages after blocks

Currently sync process (sync_job.cpp) does not start interpreting until it downloads all blocks and messages between head and target. It fetches blocks and related messages in single blocksync request, iteratively from target (newer) to head (older). Sync has index of branches which are already fetched, it assumes that they have all blocks and messages.

Lotus starts by fetching blocks only (without messages it gives more blocks faster) to find chain of parent hashes. Then it lazily fetches messages which are required to interpret next tipset, iteratively from head to target.

Refactoring sync process would allow to start interpreting earlier, and reduce catchup time. Sync index will then contain branches with fetched blocks, but may have no messages. Messages will be requested when required by interpretation (may prefetch more messages if there is time or make parallel requests for multiple heights). That also requires to keep track of connected peers and head they have, because we need peer to request messages from.

Test Scenario: Reconnect to down worker

Miner and Remote worker are working. Than the worker is down and up.

The worker register in miner second time.

The Miner should reconnect this worker without creating second one.

soralog

Can't compile success, Because the "filesystem" of soralog。
图片
I use version of gcc/g++ where is 7.5.
system: Ubuntu 18 server lts。

CMake error: Configuring incomplete, errors occurred!

OS: WSL(Ubuntu 16.04)
GCC: 7.4
CMake: 3.16.1

Operations:
1, Clone cpp-filecoin Repo and update submodule
2, Set GITHUB_HUNTER_TOKEN and GITHUB_HUNTER_USERNAME in cmake/Hunter/init.cmake file
3, mkdir build && cd build && cmake -DCLANG_TIDY=ON ..

Expected: Configuring successfully.
Result: Configuring incomplete, errors occurred!

Log:
CMakeOutput.log

CMakeError.log

Ledger wallet wrapper

Necessary to implement ledger wallet class in Fuhon to work with Filecoin adaptor of cpp-ledger lib.

Probably we want to create wallet abstraction allows us to work with local data store or ledger the same way.

Note: look at LedgerWallet struct in Lotus

JSON-RPC HTTP Transport

For now only websocket is supported for Fuhon node. The task is to add POST handler support.

Lotus metrics

Add more metrics like lotus/metrics/metrics.go which can be collected by our node.

Our node already has prometheus metrics for counter and gauge. Maybe we need to implement buckets for histogram and summary. Integrating external library may be an option too.

UPD. Didn't find summary metrics usage in lotus yet.

UPD. Switching to prometheus-cpp.

not implemented:

  • info{commit version}
  • peer/count
    • does cpp-libp2p provide disconnect event?
  • *block/failure, *block/success
    • *blocks are not validated, currently measuring tipset vm.
    • #539.
  • block/validation_ms
    • blocks are not explicitly validated (tipset vm time is other metric vm/applyblocks_total_ms).
    • #539.
  • message/failure, message/success, message/validation_ms
    • messages are not explicitly validated.
  • mpool/getnonce_ms, mpool/getbalance_ms, mpool/addts_ms, mpool/add_ms
    • no corresponding functions or they are trivial.
  • mpool/push_ms
    • api/request_duration_ms {endpoint=MpoolPushMessage} is used instead.
  • block/delay
    • is it for SyncSubmitBlock and/or pubsub?
    • does it have timeout?
  • pubsub/published, pubsub/delivered, pubsub/rejected, pubsub/duplicate, pubsub/recv_rpc, pubsub/send_rpc, pubsub/drop_rpc
    • does cpp-libp2p provide these metrics?

Task reassignment after worker fall

The situation when worker fallen with assigned task is now producing problem. The point is that Task was already removed from task queue and scheduler thinks that task is in production. However, even reconnected worker can't know anything about given task. So the task won't be completed after any time.

How is the progress of Fuhon project? I can't find the executable file for mining so called miner_main after build

Hello guys, I'm so exciting that Filecoin protocol has a C++17 implementation. And I have pay close attention to Fuhon project for a while, but a few things confuse me:
1, How is the progress of Fuhon project? Is there a schedule for this project?
2, I can't find the executable file for mining so called miner_main after build, what shoud I do if I want debug the behavior of the miner(like interactive with Venus node)?
3, Is there any documents or guidelines for contribute code to ​this project?
4, It will trigger error if I use -CLANG_TIDY=ON option on master branch, and what' the plan for this problem?

Docker-compose examples

We should provide some examples on how the one can deploy their own node via docker / docker-compose in a maintainable way

  • proper mounting points
  • public images support
  • configured and opened ports for networking / metrics

Test Scenario: Reconnect to down Node

Node and Miner are up. The Miner mine blocks. Then the Node is down and up imedeatly.

Miner should try reconnect to Node and after succesfull reconect the miner continue usual work.

When I build as the instructions tell, it keep saying error

I have generated a token and set environment variables correctly as below:

GITHUB_HUNTER_USERNAME=<github account name>
GITHUB_HUNTER_TOKEN=<github token>

But on executing cmake -DCLANG_TIDY=ON .., it failed and print error like this:

Processing simple request
GitHub Limit: 4987
Simple request pass
Get release-id by tag `cache-73c7906`
Error:
  Repository not found 'https://github.com/soramitsu/hunter-binary-cache' or user '[email protected]' has no access to it
Traceback (most recent call last):
  File "/Users/winkee01/.hunter/_Base/Download/Hunter/0.23.257-soramitsu1/130ab68/Unpacked/scripts/upload-cache-to-github.py", line 594, in <module>
    cache.upload_raw(github)

therefore I can't build this project.

is there any thing I miss?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.