This leads to proving failures for 34 out of the 700 blocks sequence on the hard test chain.

@BGluth assigning it to you as I think you had already dealt with it somewhere?

Just updated my nightly, and it looks like paladin fails to compile on the latest version. It works on 2024-02-29 however. Haven't got a chance to dig into this, but can if no one else has any spare cycles.

src/runtime/mod.rs:209:5
    |
209 |     #[instrument(skip_all, level = "debug")]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...so that the type `Op` will meet its required lifetime bounds
210 |     async fn get_task_sender<'a, Op: Operation, Metadata: Serializable + 'a>(
    |                              -- the parameter type `Op` must be valid for the lifetime `'a` as defined here...

If block depends of previous block proof, proof generation needs to be serialized. We should introduce new BlockProofFuture type that would resolve when the proof generation for the previous block is finished. In the meantime, we could parallelize other tasks (TX proof generation), and wait for the previous block proof when it is actually needed.

The current github actions workflow is not ideal, as clippy / rustfmt / tests are being run on the same job, hiding useful information in case of failure. It would be beneficial to split them into distinct steps as to easily highlight which job, if any, has failed.

(as per slack discussion)

zero-bin currently supports pre-processed circuits loading from disk (though this could be greatly improved, see 0xPolygonZero/plonky2#1394).

It however doesn't support plonky2 Kernel loading from disk, which impedes worker bootstrapping by adding an extra overhead of possibly around ~4sec to ~5sec for it to generate. The Kernel data itself takes about ~240kB when stored in a JSON file, hence loading time should be close to negligible.
Given that small txns should be in the ballpark of ~12sec to ~18sec give or take (in addition to Kernel generation), removing this extra 4sec to 5sec of bootstrapping would have a non-negligible impact on overall proving throughput.

#96 introduced multi-block processing with the Jerigon mode. While this is the mode to use in a real-life setting, testing & benchmarking are often performed in stdio mode (the former for easy reproducibility, the latter to not be impacted by networking conditions).

As such, it would be beneficial to also support multi block proving from this mode. We would probably need to enforce some rules on the witness files naming to ensure proper ordering when parsing.

I notice that zero-bin uses Paladin as its distributed processing backend. I was wondering why this choice was made as opposed to using Kafka. I think Kafka would be well suited to the task and there's a few clients written in Rust:

• rust-rdkafka
• kafka-rust

Right now circuits are stored as prover_state_foo_N where foo refers to the circuit name (arithmetic, CPU, memory, ...) and N the initial STARK table size associated.
As such, if one bumps evm_arithmetization version because of circuits changes, but forgets to flush the old prover state, they'll get some discrepancies leading to proving failure which may be hard to track. We should strengthen this process by detecting discrepancies between versions, and have some automatic flush&regenerate mechanism, or store circuit data differently as to refer to a specific version they were generated with.

zero-bin currently only supports proving a chain from the actual genesis. The requirement for starting from checkpoint heights requires considering the checkpoint block as the new "local" genesis. That is:

• returning the checkpoint block state trie root instead of the genesis root
• returning the 256 previous blockhashes, possibly going past the checkpoint block (we currently stop at genesis, for obvious reasons)
• changing the block numbers passed to the prover. As genesis has index 0, every checkpoint height is being reinitialized to 0 after the final checkpoint proof is submitted.

Otherwise, if no change here is preferable, we'd need to alter the logic inside plonky2 zkEVM block_circuit so that we do not refer to genesis when passing None as previous proof argument, but target the previously implied one.

Since #96, we have the ability to prove a sequence of blocks in one go. This however writes all generated block proofs to disk, which is not needed.
More specifically, say we have checkpoint height $N$ and we generate blocks $N+1$ to $M$, then every subsequent proof $\pi_t$ will be attesting validity of all previous block proofs $pi_{N < i < t}$. As such, only the latter is really useful and needs to be persisted.

Using the BlockInterval input from #87 and BlockProofFuture from #88 if blocks are dependent, parallelize block proof generation for standard zero-bin scenarios.

Currently zero-bin allows fetching block trace data from a Erigon instance with the proper witness APIs as well as loading this trace data from a file and kicking off proofs.

To enable easier integration testing, as well as to give users a nice way to target chains which do not yet have the proper trace APIs, we would like to integrate the code from eth-tx-proof into zero-bin such that we can fetch information from an existing RPC node and kick off the proving process for a block or transaction.

There is a new tx hash field in the latest Erigon json output from the debug_traceBlockByNumber

At the moment, the verifier binary loads the entire ProverState to be able to verify block proofs. This is a bit wasteful, as the verifier only needs a few kBs of data as opposed to the prover who needs several GBs.

plonky2, as of commit 6dd2e313c4a1d67fb5de8efbe4068554119f7f3b (November 28th), allows to call final_verifier_data() from AllRecursiveCircuits.

It would be nice to have zero-bin store a verifier version of the preprocessed prover_state so that the verifier binary does not need to load all the useless data.

eth-tx-proof and #51 are relying on ethers-core, which is on its way to be deprecated, with no plan to support EIP-4844 type 3 transactions, making it incompatible with Cancun HF.

As such, we'd need to consider moving away from it and rely on an alternative. A natural replacement would be alloy-rs, although it is partly WIP and would need us to rely on github revisions for a while instead of crates.io versions.

From its maintainer, the current status of the overall alloy project is as follows:

• Consensus types are good
• Transport stack and RPC client are good
• Some niggling bugs with proper application of eip-155 to signatures for legacy txns only
• Provider interfaces are almost settled. just working on improving eth_call right now
• Network abstraction is still expanding, but changes will be mostly invisible
• Provider type naming is still hard for beginner rust users, and we may change generics/bounds a lot to try to fix this
• Contract bindings are experimental

System

OS: macOS 14.0
Rust: rustc 1.76.0-nightly (3a85a5cfe 2023-11-20)

Behavior

% RUST_LOG=debug cargo r --release --bin leader -- --mode http --runtime in-memory --output-dir ./output

# Skip tons of stuff

     Running `target/release/leader --mode http --runtime in-memory --output-dir ./output`
Error: No such file or directory (os error 2)

Expected Behavior

Create the output directory if it doesn't exist or give a nice error.

We want easier sdtio block proving testing. So there could be one or multiple blocks as json prover input.
We need a tool that would allow us to easily create these test files form the current version of Jerigon.
Related to #105

We currently only have access in the logs to the proving trace upon failure.

It would be nice to also log

• the GenerationInputs payload passed to generate_txn_proof() (to be able to regenerate locally easily when debugging)
• the PublicValues associated to the AggregatableProof LHS and RHS arguments of generate_agg_proof(), and the ones for previous block proof and current one in generate_block_proof()

Once we have a proper versioning process for plonky2 and zk_evm monorepos, we should replace our git dependencies here to versions.

Context

Background 0xPolygonZero/zk_evm#297 0xPolygonZero/zk_evm#296 #62

Originally, the intent with zero-bin was to implement a very simple CLI tool that could prove blocks with plonky2. So the CLI interface is very much designed with this intent — provide a block number as input, and generate a plonky2 proof as output.
As we move towards making this repository production ready it's a good time expand the capabilities of zero-bin in its ability to understand what I am going to call block intervals.

Block Intervals

Block invervals encapsulate the semantics regarding how zero-bin manages a set of proving jobs relative to some higher order intent, like following a chain tip, for example. I will use interval notation to describe them.

Note for all intervals we assume ∈ ℤ

[x,x]

Prove a single block. This is the same (only) block mode that zero-bin currently supports, generalized to interval notation.

In Rust

x..=x

[x,y]

Prove a range of blocks. For example [1,100] entails proving blocks 1 through 100.

In Rust

x..=y

[x,∞)

Follow chain tip, starting from x. This entails making zero-bin aware of the tip of the target chain such that it contiguously proves blocks towards it.

In Rust

x..

Block Connection

A bit of background: zk_evm's prove_block method accepts an optional parent block proof.

With the introduction of block intervals, we will need an additional boolean flag which specifies whether contiguous blocks must be connected when generating proofs. This flag will determine whether zero-bin maintains a record of parent proof outputs to plug into these prove_block calls.

The implications on the prover module

The API of the prover module is at odds with the ability to facilitate block connection in an efficient manner. In particular, the prove method accepts the parent block proof as an argument with the type Option<PlonkyProofIntern>. In other words, this prove method is currently set up to be atomic, in the sense that it executes all proof steps in a single logical step. This of course makes sense in context of the original intent of zero-bin (proving one block at a time), but to support block intervals, this will need to be updated.

zero-bin will need to be able to kick off multiple proving jobs in parallel -- it should not have to wait for block n to complete its block proof before kicking off a proof for block n + 1. This entails heavier orchestration logic in the leader or some clever usage of Future. For example, rather than accepting an Option<PlonkyProofIntern>, it could accept some Option<impl Future<Output = PlonkyProofIntern>> which resolves when its parent proof has completed. This could likely be facilitated with some BlockProofFuture type that uses channels to asynchronously signal block proof completion. My sense is that this will be a more ergonomic solution than a more top-down imperative orchestration system -- but I leave that decision to the implementor.

Summary

Block Intervals are a key capability for wider applicability and thus adoption of zero-bin. The features and details specified herein should be broken into smaller sub issues before implementing, while this issue can serve as the higher level discussion environment and reference definition for the capability as a whole.

When debugging failing transactions / blocks, we may want to have a simpler setup to make sure the regression is addressed without having to bother with all the heaviness of computing actual proofs / aggregating txns together.

As most of these proving failures occur during witness generation, we could just have the leader send the individual txn IR, and have workers proceed to a single Operation that would call generate_traces from the plonky2_evm API instead of one of the prove functions.

We need to unify zero-bin program input to handle one, multiple blocks, or continuous stream of new blocks, as discussed in #86. Introduce new type BlockInterval as a CLI argument, and update API of the main use-cases to use it.

Also, introduce new parameter that would indicate if the blocks in the range are dependent from previous block proof. If they are not dependent, their proof generation could be fully parallelized in some future PR.

Right now we store all circuit data at the root level, which can become a bit messy as we're now splitting prover state and storing each individual circuit size separately.
It would probably be cleaner to store them in a dedicated circuits folder, or equivalently named.

Currently, if no circuit ranges are specified when calling the leader, this results in using a default set of ranges. This can be problematic / cumbersome whenever we:

• forget to manually specify the set of ranges we're interested in that differ from the default ones (which are purposely really large, to account for all scenarios in real-life settings)
• use the debug_block.sh script, that ignores the notion of prover state as it only focuses on witness generation

When #27 is merged, being in one of these two scenarios will flush the previous circuits and rewrite new ones, which is time consuming and fairly wasteful. Especially if in a debugging session, and alternating say between debug_block.sh and prove_block.sh scripts for instance.

Instead, I'd suggest to tweak default circuit loading, i.e. when no specific ranges are specified, by first loading from disk the existing ones, if existing, and defaulting to an arbitrary set of default hardcoded values if non-existing. This would alleviate the circuit wiping issue, but would still incur possibly heavy burden on disk IO / memory consumption for the debugging script which does not need any of the circuitry. To this end, we could just load a BaseProverState under the test-only feature flag, i.e. containing only the higher level circuits which remain fairly small, and completely ignore the recursive table circuits. This would fail in a regular proof generation context but this suffices for witness generation and doesn't require any API change in the dependencies.