metacraft-labs / dendreth Goto Github PK

We should maintain a documentation resource listing all existing smart contract light client implementations for other blockchains that can be used for creating a two-way bridge.

For Polkadot, the Snowbridge project has stated goals, very similar to ours.

For Solana, a similar bridge have been discussed here:
https://blog.nil.foundation/2021/11/01/solana-ethereum-bridge-design.html

For Cosmos, a light client implementation is being developed here:
https://github.com/ChorusOne/tendermint-sol

Objective: https://hackmd.io/@metacraft-labs/DendrETH-for-liquid-staking-protocols

Subtasks:

• #160

• We already have a naive version of validator_balances.circom circuit: #161
  Changes to Circom Circuit: We will adapt the circom circuit for validator balances to operate with Poseidon hashes, which are computationally cheaper than SHA256. It would require an alteration to the circuit to operate recursively, and we will need to assess the time requirement for the calculation of the validator balances. Considering recursion costs, it might be more efficient to compute some calculations in plonky2 and then validate plonky2 within the final circom circuit along with a few other elements.

• We will also consider using Plonky2's Starky, which offers fast performance for SHA256 (SHA256 Prover Comparison). This sounds like a very promising route as similar scheme is explained here: https://youtu.be/-pucWUDn5Hw?list=PLj80z0cJm8QErn3akRcqvxUsyXWC81OGq

• We have also examined PIL-STARK, developed by J. Baylina, the creator of circom, and used in Polygon's ZK EVM. It may be useful to also try to test the sha256 proofs performance with it.
  Some resources for it:

  • https://youtu.be/0IZ8-tYaNJM
  • https://youtu.be/ZwG3UI_iDAs
  • https://youtu.be/639DZpIC0Fc
  • https://www.notion.so/zkEVM-Audit-Education-Session-11-15-11-22-86d60daceadb438f85908817f7082611
  • https://polygon.technology/blog/pil-zk-tooling-is-now-open-source

The supported languages will be Rust and Nim (through C/C++).

Right now the our uploading and initialising script does not support it. We need to find a workaround for this.

In previous testing we found that this change in the cargo file will solve the problem

cosmwasm-schema = { version = "1.0.0-beta", default-features = false }
cosmwasm-std = { version = "1.0.0-beta", default-features = false, features = ["iterator"]  }
cosmwasm-storage = { version = "1.0.0-beta", default-features = false, features = ["iterator"] }

https://github.com/0xPARC/zk-bug-tracker#common-vulnerabilities-1

I believe we may have problems with Arithmetic Over/Under Flows
Or with Mismatching Bit Lengths
As the assumption was that the LessThan circuit would fail if the number is with incorrect bit length but it was a wrong assumption and turns out that LessThan does not constrain that and it is a common vulnerability for circom circuits

The main issue we are encountering is that the SHA-256 hash function, which we currently use for our Merkle proofs in the Ethereum consensus specification trees, is not zk-friendly.

We propose to create a commitment mapping from the validators' root (a SHA-256 Merkle tree comprising all validators) to a Poseidon root of validators. This process would involve generating a proof that a given SHA-256 Merkle tree of validators matches a corresponding Poseidon Merkle tree of validators. Given the sheer number of validators, this tree will be formed using recursive proofs. Since only a small fraction of validators changes per epoch, we can economically update and reuse the proofs. The Poseidon hash function, being more zk-friendly, could then be used to validate that a specific validator belongs to the tree in a more cost-effective manner.

Steps:

• We will begin by creating the leaves level with a hashtree root of a single validator.
• Then, the second level will aggregate their hashes and the proofs of the leaves, producing one SHA256 hashtree root and one Poseidon hashtree root.
• Eventually, we will provide proof that the SHA256 root corresponds to the Poseidon root.

Plonky2 seems like the suitable proving system for this as recursion there is very cheap and we have verifiers for circom implemented.

Create a circom circuit to validate a merkle multiproof

https://github.com/Veridise/Picus

NLVM is an alternative build of the Nim compiler that compiles the Nim code directly to LLVM IR and could potentially produce smaller and more efficient WASM binaries.

Once we have built-it, you should be able to act as a drop-in replacement for the Nim compiler. Since the build process is likely to be slow, we should cache the result in our Cachix org.

Once #10 is developed, we'll have a file system archive of light client updates for the Prater network. We can expand our wasm-based unit tests to feed the chain of updates to the DendrETH wasm implementation running in NodeJS. This script should be easy to re-target to actual smart contract implementations in the future.

Please create a docker image which will run the discord bot that tracks the dendreth relayer -

Discord Bot Branch for reference

New dependency:
"discord.js": "^14.12.1"

We can obtain the light client updates from the beacon node REST API (ethereum/beacon-APIs#181) which is already supported in Nimbus.

An initial version of the relay will just focus on gathering all updates and storing them in a directory, so we can use them in WASM-based testing scenarios.

• Add a commitment to the public inputs using poseidon/pedersen hash currently the public inputs are

  signal input originator[2];
  signal input nextHeaderHashNum[2];

• Add the vk to the commitment
• Introduce participation rate variables they can be an array of 1000 variables that will tell what was the participation for the sync committee and the commitment of public inputs

#155

When running the job - yarn hardhat start-publishing with argument --slotsjump 1 the relayer is worker is unable to publish proofs

https://github.com/DoHoonKim8/stark-verifier

Since we upgraded to circom 2.1.0 we can use the newly added features for

• Anonymous Components
• The use of <== for signal arrays
  And others described in circom docs: https://docs.circom.io/circom-language/anonymous-components-and-tuples/#the-use-of-for-signal-arrays

Which will make the code much more readable and well structured

Our repository currently features a Nim and Rust implementation of the BN curve primitives required for Groth16 verification.

The Nim implementation is lifted from the Nimbus codebase and the Rust one is taken from arkworks. There are some additional implementations that we can potentially evaluate such as the original BN curve crate from ZCash and the WASM-native work in the wasmcurves project.

Due to various technical details, these implementations are not portable to all of our targeted blockchains, but we need to understand better how they stack against each other performance-wise.

We should create a basic test harness based on the standard WebAssembly run-time in node.js (and potentially other run-times in the future). The benchmark can measure either the wall clock time as a good proxy for the consumed gas or things like executed instructions count if some of the WASM run-times can provide such information.

This will allow us to compare the gas consumption of our Rust and Nim verifiers.

We'll target EOS with our WebAssembly builds:
https://developers.eos.io/welcome/v2.2/getting-started-guide/hello-world

1. @monyarm will create a branch in nix-blockchain-development that upgrades the Circom Nix package to the latest version (currently v2.1.5) and will open a PR.
2. Once the CI for the PR is green, @monyarm will prepare a branch in DendrETH where the nix-blockchain-development flake input points to the branch
3. @Dimo99 will test the new version of the compiler to ensure it works correctly
4. Assuming all goes well, we'll merge the PR in nix-blockchain-development and merge the branch in DendrETH with the upgrade of Circom

From DevOps point of view, the main reason for upgrading is the -l option which was introduced in 2.0.8.

We need a basic example demonstrating how a simple Circom circuit can be compiled, executed and later verified in an Ethereum smart contract.

We should consider using rxjs for the relayer. As we have async code there with a lot of async cases
https://rxjs.dev/

Hi, I'm trying to run the EVM Simulation, but I'm getting the following error when I'm running the nix develop command. I'm able to circumvent the issue by running direnv allow instead, but I'm not sure if that's the right approach.

Submodule 'vendor/build-artifacts' ([email protected]:metacraft-labs/DendrETH-build-artifacts.git) registered for path 'vendor/build-artifacts'
Submodule 'vendor/circom-pairing' (https://github.com/metacraft-labs/circom-pairing.git) registered for path 'vendor/circom-pairing'
Submodule 'vendor/eth2-light-client-updates' ([email protected]:metacraft-labs/eth2-light-client-updates.git) registered for path 'vendor/eth2-light-client-updates'
Submodule 'vendor/nim' ([email protected]:metacraft-labs/nim.git) registered for path 'vendor/nim'
Submodule 'vendor/nim-blscurve' ([email protected]:metacraft-labs/nim-blscurve.git) registered for path 'vendor/nim-blscurve'
Submodule 'vendor/nim-bncurve' (https://github.com/status-im/nim-bncurve) registered for path 'vendor/nim-bncurve'
Submodule 'vendor/nim-confutils' ([email protected]:status-im/nim-confutils.git) registered for path 'vendor/nim-confutils'
Submodule 'vendor/nim-faststreams' ([email protected]:status-im/nim-faststreams.git) registered for path 'vendor/nim-faststreams'
Submodule 'vendor/nim-serialization' ([email protected]:status-im/nim-serialization.git) registered for path 'vendor/nim-serialization'
Submodule 'vendor/nim-ssz-serialization' ([email protected]:metacraft-labs/nim-ssz-serialization.git) registered for path 'vendor/nim-ssz-serialization'
Submodule 'vendor/nim-stew' ([email protected]:status-im/nim-stew.git) registered for path 'vendor/nim-stew'
Submodule 'vendor/nim-stint' ([email protected]:status-im/nim-stint.git) registered for path 'vendor/nim-stint'
Submodule 'vendor/nim-terminaltables' ([email protected]:xmonader/nim-terminaltables.git) registered for path 'vendor/nim-terminaltables'
Submodule 'vendor/nimcrypto' ([email protected]:cheatfate/nimcrypto.git) registered for path 'vendor/nimcrypto'
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/build-artifacts'...
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/circom-pairing'...
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/eth2-light-client-updates'...
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/nim'...
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/nim-blscurve'...
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/nim-bncurve'...
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/nim-confutils'...
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/nim-faststreams'...
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/nim-serialization'...
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/nim-ssz-serialization'...
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/nim-stew'...
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/nim-stint'...
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/nim-terminaltables'...
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/nimcrypto'...
git-lfs filter-process: git-lfs: command not found
fatal: the remote end hung up unexpectedly
Submodule path 'vendor/circom-pairing': checked out '82bac4a27499fc11cf3ebbb64c525b526f13b552'
Submodule path 'vendor/eth2-light-client-updates': checked out 'c9d5a907b55db0e503661bcfa3963d6fd47eb462'
Submodule path 'vendor/nim': checked out '77edc2bacac1418aeee934fec871eb75b1b18b77'
Submodule path 'vendor/nim-blscurve': checked out 'd93da7af30a9a2160f68d84c5210a12c4d16df00'
Submodule 'vendor/blst' (https://github.com/supranational/blst) registered for path 'vendor/nim-blscurve/vendor/blst'
Submodule 'miracl-core' (https://github.com/miracl/core) registered for path 'vendor/nim-blscurve/vendor/miracl-core'
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/nim-blscurve/vendor/blst'...
Cloning into '/Users/jspark/Documents/GitHub/hackathon/2023ZKHackathon/second-try/DendrETH/vendor/nim-blscurve/vendor/miracl-core'...
Submodule path 'vendor/nim-blscurve/vendor/blst': checked out 'b38a52c86e4f5ae69ba6eca41d050707b6fcc72b'
Submodule path 'vendor/nim-blscurve/vendor/miracl-core': checked out 'd799a3f7f4edfdad8c8b709e04f8a0a7b5a7d251'
remote: Enumerating objects: 34, done.
remote: Counting objects: 100% (34/34), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 19 (delta 14), reused 19 (delta 14), pack-reused 0
Unpacking objects: 100% (19/19), 2.51 KiB | 20.00 KiB/s, done.
From https://github.com/status-im/nim-bncurve
 * branch            8136ebdc515df4a3f269eba5f05c625ae742df83 -> FETCH_HEAD
Submodule path 'vendor/nim-bncurve': checked out '8136ebdc515df4a3f269eba5f05c625ae742df83'
Submodule path 'vendor/nim-confutils': checked out 'a26bfab7e5fb2f9fc018e5d778c169bc05772ee6'
Submodule path 'vendor/nim-faststreams': checked out 'c80701f7d23815fab0b6362569f3195957e57856'
Submodule path 'vendor/nim-serialization': checked out '493d18b8292fc03aa4f835fd825dea1183f97466'
Submodule path 'vendor/nim-ssz-serialization': checked out '66097b911158d459e5114fabd0998f0b2870f853'
Submodule path 'vendor/nim-stew': checked out '06621a2fcddf01b0231aaa6d18531b0a746b3140'
Submodule path 'vendor/nim-stint': checked out '27a7608f33a485fb837c9759ddb4a7c874eb7ad2'
Submodule path 'vendor/nim-terminaltables': checked out '37981f5d403fb55688e00d24ab9b1d56e252f52b'
Submodule path 'vendor/nimcrypto': checked out '24e006df85927f64916e60511620583b11403178'
Unable to checkout '24a97aae330fc12007fe51d4918293f5fdc2e62b' in submodule path 'vendor/build-artifacts'
➜  DendrETH git:(main) git submodule update --init --recursive
➜  DendrETH git:(main) nix develop
warning: ignoring untrusted substituter 'https://nix-blockchain-development.cachix.org', you are not a trusted user.
Run `man nix.conf` for more information on the `substituters` configuration option.
error:
       … while evaluating the attribute 'optionalValue.value'

         at /nix/store/g738q3jxmym83jx24d422nwc0nas4zyv-source/lib/modules.nix:799:5:

          798|
          799|     optionalValue =
             |     ^
          800|       if isDefined then { value = mergedValue; }

       … while evaluating a branch condition

         at /nix/store/g738q3jxmym83jx24d422nwc0nas4zyv-source/lib/modules.nix:800:7:

          799|     optionalValue =
          800|       if isDefined then { value = mergedValue; }
             |       ^
          801|       else {};

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: NAR hash mismatch in input 'github:numtide/flake-utils/93a2b84fc4b70d9e089d029deacc3583435c2ed6' (/nix/store/dc2fivkjahwx47zhjs9jx19ybd9wwcdr-source), expected 'sha256-nuEHfE/LcWyuSWnS8t12N1wc105Qtau+/OdUAjtQ0rA=', got 'sha256-U02riOqrKKzwjsxc/400XnElV+UtPUQWpANPlyazjH0='

The existing deployments will continue to be maintained by the Relayer

The public inputs to the circuit are:

• validatorsAccumulator: A Merkle tree accumulator of validator public keys and Eth1 deposit indexes. Eth1 deposit index is need to validate if the validator is already part of the beacon chain.
• stateRoot: The current beacon state root against which the proof is made.
• balanceSum: The sum of all active validator balances.

The workflow of the circuit is as follows:

1. Epoch Calculation: The currentEpoch is calculated from the provided slot. This calculation is then constrained. A Merkle proof verifies that the slot aligns with the provided stateRoot.

2. Index and Root Validations: The circuit checks if the currentEth1DepositIndex aligns with the stateRoot. It also validates whether the provided validatorsRoot and balancesRoot are part of the stateRoot.

3. Validator Verification: The circuit validates all validators within validatorsRoot. It uses the RangeCheck circuit to confirm that the currentEpoch falls between the validator's activation and exit epochs. If a validator is inactive or their Eth1 deposit index precedes the validator's Eth1 deposit index, the bitmask for that validator is set to 0. Validators whose Eth1 deposit index is larger than the current Eth1 deposit index aren't constrained to have valid Merkle proof because they aren't yet part of the tree.

4. Balance Index Check: The circuit checks that provided balancesProofIndexes correspond to the validatorIndex divided by 4 plus balancesProofIndexesRemainders. This is because balances are stored in a 256-bit array, with each balance occupying a separate 64 bits of the 256-bit segment.

5. Balance Validations: It validates that all the passed balances are valid with respect to the balancesRoot.

6. Validator Accumulator Validation: The circuit verifies that the hash tree root of all passed validators and their Eth1 deposit indexes matches the passed validatorAccumulator.

7. Balance Sum Calculation: The circuit sums all the balances according to the bitmask calculated for the validators.

8. Hashing and Output Generation: The public values are hashed. The circuit outputs the first 253 bits of the SHA-256 hash of the public inputs to make circuit verification more cost-effective.

The circuit is implemented in #153

Currently the ports on which the Prometheus service is actively listening are a predetermined set.

Extend the run-relay.sh script so that it counts the number of launched services and dynamically creates the Prometheus targets in its config file.

We have a lot erroneous code that does things like String(process.env['SOME_VAR']), which simply masks possible issues when the variable is not defined, or defined but not following the expected schema.

The solution is three-fold:

• Introduce a helper module for accessing env variables and validating that they match the expected format
• Add ESLint rule that forbids access to process.env
• Run ESLint in the CI pipeline

We should be able to target NEAR with our WebAssembly builds:
https://docs.near.org/develop/quickstart-guide

FAIL  tests/helpers/helpers.ts
 ● Test suite failed to run

    Your test suite must contain at least one test.

      at onResult (node_modules/@jest/core/build/TestScheduler.js:172:18)
      at node_modules/@jest/core/build/TestScheduler.js:300:17
          at Array.map (<anonymous>)

Do we have any unadded tests @kkirkov ? Please fix, since this is the only failing test.

Constantine is a highly-optimized Nim library implementing the BN254 curve used for Groth16 verification:
https://github.com/mratsim/constantine

Its performance is largely based on the use of carefully tuned inline assembly (which we cannot use when targeting WASM), but according to the author of the library (Mamy Ratsimbazafy), even without these tricks it should be more performant than nim-bncurve.

To disable the use of inline assembly, we need to compile the library with -d:CttASM=false.

1. Write unit tests for individual functions
2. Use mock tests for API calls
3. Think of ways to write integration tests for an asynchronous system

Following the steps as described in the documentation work until yarn install.
The step with make evm-simulation fails with output:

https://gist.github.com/RafaelAPB/c789f91817486f2591fa807188f27924

The project will generate and maintain a public archive of downloadable light client updates, encoded in SSZ form and covering the history of the beacon chain networks such as Mainnet and Prater since the Altair fork.

The archive will be generated by our relay node.

The light client implementation will be extracted from the Nimbus codebase. We will compare the size and the efficiency of the resulting code against the Rust implementation created by the Snowbridge project on a real-world client updates obtained from the beacon chain mainnet. The results of the comparison will be reproducible and tracked over time in our CI.

The tested WebAssembly compiler toolchains will be Rust, Nim+Clang, NLVM.

We should be able to target Substrate, Cosmos, the Internet Computer, Elrond and Solana from the same codebase as well as well as stand-alone WebAssembly rumtimes such as Wasmer for a simplified form of unit testing.

The supported languages will be Rust and Nim (through C/C++).

1. Update the relayer polling task to work with {from, to} tasks that will generate proof from a given slot to a given slot.

2. Introduce an orchestration module that will be triggered every SLOTS_JUMP*12s. This module will create a new task from a given slot to the best multiple of SLOTS_JUMP.

The orchestration module will review all the smart contracts and their respective slots. If any contracts are not aligned to the same slot, the orchestration will generate an optimized set of updates to synchronize them.

In Circom 2.0.8 the parallel keyword was added which allows for some part of the witness generation to be computed in parallel. Which can boost the performance of our witness generation
https://github.com/iden3/circom/blob/master/RELEASES.md#august-26-2022-circom-208

We may be able to compile parts of our WebAssembly client as a circuit using the C++ toolchain available here:

https://github.com/risc0/risc0

Remove the following code snippets and text from -

In How does the relayer work?

INITIAL_SLOT=5355991
SLOTS_JUMP=64

...

We have set up a recurring job that repeats itself after a specified time interval (slotsjump) and starts from an initial slot. The job follows a specific network, currently supporting Pratter and Mainnet. To run this job, execute the following command in the beacon-light-client/solidity folder:
yarn hardhat run-update --initialslot $INITIAL_SLOT --slotsjump $SLOTS_JUMP --follownetwork pratter

@Dimo99, is this step completely redundant or should it be replaced by something else?

In Running DendrETH relay node

This passage -

You can also build a custom image yourself by executing the make dendreth-relay-node command within the development environment provided by this repository.

implies this step is optional while it is the default way to set up the docker environment in which we can deploy the relayer.

In Running the test suites

The WebAssembly tests of the Nim light client:
yarn test-emcc seems to be deprecated and should be replaced with yarn test

Additionally

Solidity tests fail if ETHEREUM_MAINNET_RPC is not set in .env

wasm nim-lc tests fail if

CUDOS_MNEMONIC
CUDOS_PUBLIC_KEY
CUDOS_CONTRACT_ADDRESS
CUDOS_RPC_ENDPOINT
COSMOS_LOCAL_MNEMONIC
COSMOS_LOCAL_RPC_ENDPOINT
MALAGA_ADDRESS
MALAGA_RPC_ENDPOINT
MALAGA_MNEMONIC

are missing from .env
These should be included in the README.md or better yet - provide the user with a preset .env

The docker image requires more than the default partition size - steps to increase the docker partition size should be included -

sudo systemctl stop docker.service docker.socket
sudo zfs umount -f zfs_root/nixos/var/lib/docker
sudo zfs destroy -fr zfs_root/nixos/var/lib/docker
sudo zfs create zfs_root/nixos/var/lib/docker -o canmount=on -o quota=100G
systemctl start docker.service docker.socket

sudo zfs set "zfs_root/nixos/var/lib/docker" -o canmount=on -o quota=300G
sudo zfs quota=300G "zfs_root/nixos/var/lib/docker"

• The supervisor is central to the execution and common commands to interact with it should be provided.
  This is how you should shutdown the supervisor - sudo unlink /tmp/supervisor.sock

• Add expected gas fee under deployed smart contracts table

Once the archive of light client updates in place, develop a script based on HardHat, Ethers.js and TypeScript that will simulate a full sync of a Solidity-based light client implementation such as the one from Darwinia (MIT licenesed). The main objective is to gather some data on the required gas under such realistic conditions. We shall maintain our own copy of all Solidity contracts in this repo in order to be able to experiment with gas usage optimizations that will be contributed back to the Darwinia project.