Attempting to start a discussion of what is the best way of integrating/bridging/mounting/etc... libp2p and devp2p

So far, I've seen two ways being suggested to go about this:

Have both lib2p and devp2p running side by side

• Pros:

  • Simpler to implement
  • Clients are bridge nodes that decide when to resolve things over devp2p or request them from the IPFS layer

• Cons:

  • Less reusable across future implementations?
    • arguably, this can be implemented in a way that is still reusable

Deeper integration of devp2p protocols/transports with libp2p

• Pros:

  • possibly easier to gain traction with future projects that want to integrate with both devp2p and libp2p networks

• Cons:

  • complex to implement

summoning @whyrusleeping @diasdavid @kumavis @hermanjunge for feedback

What is this

The Hot Importer, in contrast to the "Cold" Importer, which gets data from a disconnected (hence cold) LevelDB, is designed to be able to bring current Ethereum data to be consumed by the IPFS network.

Vision

Initially thought as a hack to an Ethereum Client. This importer is a client on its own. The vision it represents is to become the Modular Blockchain Client for Storage and Networking, making possible to multiple blockchains, or any other application that relies heavily on content-addressed data, to just plug into the IPFS network to fulfill backend roles, allowing the clients of these blockchains to focus on their core competencies, namely state change computations and consensus. An ambitious goal that is implied as a possible logical step to the solution of several scalability problems.

As of 2017.10.17, the devp2p network consists on 24k nodes (Ethernodes see below), the bitcoin network has 9k nodes (Bitnodes see below), while it is reported that libp2p is made of 9k nodes. Once the way is made for multiple blockchains to enter the field and interoperation becomes common place, the concept of "pluggable networking" will reach relevance, and it will be positive to be prepared in advance by offering an easy to plug library or API for such effect.

Features

The Hot Importer aims to provide the following core features

• Connect to devp2p as another node of the mesh. Identifying itself as mustekala with its version.
• Send messages to its connected neighbors inquiring for
  • State Trie Data
  • Storage Trie Data
  • Block Headers
  • Transactions
  • Transaction Receipts.
• Upon received data, this will be imported into IPFS as IPLD blocks, for further consumption of the libp2p network.

Additional features (nice to have)

• Feed a floodsub subscription to inform its peer on new minted blocks.
• Update tables such as the CHT (Canonical Hash Table), Transaction to Block and future needed tables that can improve the lookup process.
• Provide an API for clients to be used either as a library, or as an independent program.
• Other?

Objects Importing

Has

N/A

Wants

• state trie nodes
• storage trie nodes
• block headers
• transactions
• transaction receipts

Sources

Overview

The Cold importer for each kind of object has two tiers:

• Traverse throughout a disconnected geth levelDB and dump found contents into file system, to be processed afterwards. i.e. It can be compressed, shared, divided, etc.
• Import from file directory into a disconnected IPFS repository.

The idea of having two tiers is that this approach improves the scaling of this process.

Has

• eth-evm-code
  • LevelDB to File System
  • File System to IPFS

Roadmap

Ethereum Objects

• eth-state-trie
  • LevelDB to File System
  • File System to IPFS
• eth-block
• eth-tx-trie + eth-tx
• eth-tx-receipt-trie + eth-tx-receipt
• eth-storage-trie
• eth-block-list (a.k.a. Ommers list)

Improvements

• Provide tools to check whether we have a complete state in our local node.

IPFS + ETH: Backend Roadmap Proposal

go-ipld-eth

• eth-account-snapshot
  • codeHash: Make the right CID reference (0x55)
  • root (storage root): Make the right CID reference (0x98)
• codeHash (0x55)
  • Implement its input
  • As it does not resolve to anything more than its raw data, do not make it an ipld Node.
• eth-storage-trie (0x98)
  • Implement its input from the importers.
• eth-tx-receipt (0x95) - eth-tx-receipt-trie (0x96)
  • Implement just like eth-tx and eth-tx-trie
  • Implement its input:
    • Should receive a json array of all block receipts, and its root must match its block's
• eth-block-list (0x91)
  • Is just a list of block-headers. Find the better implementation for its input.
• Improve the non-linux install (see Issue #8 of go-ipld-eth)

go-ipld-eth-import

• COLD IMPORTER
  • Complete ipfs shell component
  • Import from block 0 to 4,315,000 in crane
  • Import block headers
  • Import state trie nodes
  • Import code (codeHash 0x55 as index)
  • Import storage trie nodes
  • Import transactions
  • Import transaction receipts.
  • Import ommer headers
  • Import block lists (ommer lists)
• Replication daemon:
  • Traverses from crane and add/pin into other servers
• HOT IMPORTER
  • Take geth, and ipfs dag put just before it Db.Put().
• Metrics on Importing
  • Weight in bytes of a block state. (i.e. full traversal of a block)
  • Weight of a "delta" (difference from block N state to block N+1 state)
  • How many trie nodes per block state.
  • Delta of trie nodes #.
  • Time per importing iteration
  • Breakdown of time per importing iteration (DB connection, IPFS client gets and puts).

DevOps

• Make sure the data is available in the following musteka.la servers:
  • fox, bat, tiger, monkey, panda, mantis, crane
  • Make sure there is a geth node (with hot importer) and an IPFS+eth node (SSL gateway for web clients)

Last Block subscription

• Informs the CHT, that is block number of CHT -> Block hash.
• Using IPNS? PubSub?
• Subscription for IPFS clients?

MetaMask/eth-ipfs-browser-client: Lookups? How?

• filters middleware
• getLogs support (ugh)
• CHT (block number -> block hash)
• block tracker / block head syncer
• tx block references (tx -> block)
• block expansion (block -> txs)
• Token data made easy?

MUSTEKALA: The IPFS ETHEREUM BLOCK EXPLORER

• Known Importers Statuses
• Nodes delivering Eth IPLD Blocks
• Network transfer / latency stats.
• Merkle Trie Heat Map?
• Availability of IPLD Links?
• Block Exploring itself: Latests blocks, txs, addresses and balances...