Description

Write up contributor documentation in the developers/contributing.md section of the docs mdbook.

Description

There are many areas of the codebase that need documentation, and two primary categories of docs to tackle.

Inline Documentation

This is a list of documentation issues for inline docs. These are doc comments that are inside source code files prefixed with a triple slash (///) that get compiled into documentation by rustdoc.

• #22
• #23
• #24
• #25
• #26

Markdown Docs

This is a list of issues for documenting items in markdown as part of the docs/ mdbook.

• #30
• #31
• #32
• #33

Description

Inside the stages crate, there is a stage/cannon.rs stage for the cannon prestate generation, a stage/allocs.rs Allocs stage for devnet allocs, and a cloning stage to clone the optimism monorepo.

This is a naive, sequential pipeline that starts with the cloning of the monorepo and then proceeds with generating the prestate and allocs.

This ticket is to make the monorepo cloning configurable through the stack.toml, such that under a [stages] table, a user can define something like isolate=true to force each stage to use isolate environments. In an optimism-monorepo exclusive stack, this would force each stage to clone the monorepo and execute it's behavior.

For non-isolated stages, the prestate stage would come first and check that the optimism monorepo is cloned, cloning it if not and leaving it available for all subsequent stages.

Description

Minimal documentation should be done inline with extensive markdown book docs to avoid inconsistencies and duplication.

For this issue, the primitives crate readme.md file should be removed and minimal inline docs should be added to crates/primitives/src/lib.rs.

In the future, more extensive documentation will be added to documents in the docs/ mdbook.

Description

Following the precedent set in reth, minimal binary documentation should be done inline with extensive markdown book docs to avoid duplicate, and out-of-sync documentation.

For this issue, the opup binary readme.md file should be removed and minimal inline docs should be added to opup/src/lib.rs.

In the future, more extensive documentation will be added to documents in the docs/ mdbook.

Description

The op-up binary should have a --dry-run flag to allow for "dry runs" for a given stack.

In the L1 Executor stage, we should add support for the Erigon execution client.

This will require setting up the Dockerfile (and an entrypoint file if necessry) and checking that:

1. the image is built successfully
2. the container is created successfully
3. the container is started successfully
4. the ports are bound correctly, and the assigned RPC API port is responding

Description

Currently, the opup cli binary directly runs the op stack "devnet". It should be extended to accept Subcommands inside the Args struct, and then dispatch to the subcommand logic based on the selection. When no subcommand is provided, an inquire prompt should confirm that the user wants to run a stack based on the local stack.toml config file or environment variables + defaults if none is present. This prompt should not be shown when the up subcommand is passed into the cli.

For a subcommand reference implementation, see the reth binary.

Task List

• #18
• #19
• #20
• #21
• #55
• #62

Description

Warning

Blocked by implementing component TOML configs in #14

Remove hardcoded docker artifacts - the docker directory - and replace with example op stack toml configuration files.

Adding a HEALTHCHECK to dockerfiles is useful both as a documentation of the best way to check the health of the container. Since docker containers are being spun up programmatically they also help with docker inspect being a good way to check the health of a container

Description

Inside the mdbook docs directory, the breakdown of supported op stack components needs to be documented within the components/ subdir.

This should break down the various components and their features.

We should add a Docker network for services to communicate with each other without the need to bind ports on the host.
Moreover, this will allow for communication via container names, e.g. ws://opup-l1:8586.

The Composer should create a network tagged with the same label as other opup components: com.docker.compose.project=op-up. By default each new container should be part of the same opup-net network but this should be configurable via the create_container function arguments.

Description

Minimal documentation should be done inline with extensive markdown book docs to avoid inconsistencies and duplication.

For this issue, the stages crate readme.md file should be removed and minimal inline docs should be added to crates/stages/src/lib.rs.

In the future, more extensive documentation will be added to documents in the docs/ mdbook.

cloned the op-up repo and when I run cargo run I get the error:
failed to run custom build command for jemalloc-sys v0.5.4+5.3.0-patched

Description

Introduce a new book.yml github ci workflow based on reth's book.yaml.

This will require the specified https://stack.anton.systems url to point to a deployed github page.

Alternatively, there may be a way for a deployment service to automatically build and deploy the site on a trunk update. (ie when a pr is merged into main, some service automatically builds and deploys the mdbook to the specified https://stack.anton.systems site).

When executing the up command, the allocs stage incurs in a critical failure.

error:

Exception: Exception occurred in child process: 
Command '['forge', 'script', 'scripts/Deploy.s.sol:Deploy', '--sender', 
'0x0dbf9c4872d17003bd9a9c3cd5b71bfff61cac4a', '--rpc-url', 'http://127.0.0.1:8545', 
'--broadcast', '--unlocked']' returned non-zero exit status 1

this error originates in the bedrock devnet python script here:

https://github.com/ethereum-optimism/optimism/blob/6bc21cb17702284a80e1a3e50fa70a996a721e97/bedrock-devnet/devnet/__init__.py#L129

Description

Inside the stages crate, there is a stage/cannon.rs stage for the cannon prestate generation.

The cannon stage should be renamed such that it is more closely tied to it's use as a fault proof system "hook" to generate the prestate instead of tying it down to "cannon".

The output of this issue should be to migration the cannon.rs module to something like prestate.rs.

Description

As part of #17, this task is to introduce a convenience nuke subcommand that nukes all running op-up stack components (including cleaning/pruning any docker containers, images, and volumes). Since this is a destructive command, it should use inquire to prompt the user to confirm that they want to perform the given actions (described in detail before the confirmation prompt).

Nuking logic should be entirely handled by the StageManager system orchestrator.

The extent of how to nuke stacks should be managed through cli flags and passed into the StageManager when executing the nuke() call. Since nuking also involves the local stack, the opup cli binary should build the StageManager from the config, and then call nuke() -> eyre::Result<()>. How the nuke cli flags are configured in the StageManager when building from the config or passing them directly in the nuke() call as arguments is an implementation detail, that is up to the implementor.

For example

StageManager::from(config).nuke()?;

Since the cli binary logic is minimal, it should be placed inside the cli.rs file where subcommand dispatching is handled.

In the stages that don't use Docker (e.g. L1 genesis, L2 genesis, Prestate, Allocs) we should double check which dependencies are used. In particular, the make commands could hide dependencies by launching scripts with multiple steps inside.

• Here we should add make to the list of binaries

• for go, we need to make a separate function go() similar to foundry() in the same file. The function should perform the brew install go command on MacOS, and use the package_manager() helper to install the package called golang-go on Linux. We can skip Windows at the moment.

Description

The current opup list subcommand fetches a simple list of docker container information using the op-composer.

This subcommand should be migrated to a more generic opup inspect subcommand that inspects op stack info.
It should build a config from the stack.toml present in the executing directory or the global directory and use this config (along with its various component configs) to check if those components are running (docker or otherwise). This could probably be done by checking ports or alternatively, wrapping components with an op-up specific endpoint for querying the status of the component.

Other information like persistent volumes/data dirs could be queried using flags to inspect (e.g. opup inspect --volumes)

inspect could also provide a method of interacting directly with a component or changing its configuration on the fly. For example, opup inspect --mod <component_name>, which would open up a subshell with specific commands into that component. It might be easier to do this as a separate subcommand to the op-up binary though in another ticket like opup edge <component_name>.

Description

Inside the op-composer crate, there is a list_containers method to list docker containers that have the op-up label.

A nice to have subcommand would be a simple cli subcommand like containers that would pretty print a nice list of op-up docker containers to stdout as a table

For example:

$ cargo run -- list

op-up docker containers
----------------------------------------------------------
| container            | status      | id       | size   |
----------------------------------------------------------
| l1-execution-client  | up: 1:24:00 | abcdefgh | 156gb  |
| l2-execution-client  | up: 1:24:00 | abcdefgi | 32gb   |
----------------------------------------------------------

Met this issue when exec make devnet ethereum-optimism/optimism#3087

Description

Document resources to help troubleshoot op-up usage inside docs/developers/troubleshooting.

Description

Minimal documentation should be done inline with extensive markdown book docs to avoid inconsistencies and duplication.

For this issue, the contracts crate readme.md file should be removed and minimal inline docs should be added to crates/contracts/src/lib.rs.

In the future, more extensive documentation will be added to documents in the docs/ mdbook.

Description

As part of #17, this task is to introduce a new down subcommand to op-up that winds down the op-stack defined by the local stack.toml configuration file.

Since this is a destructive action, it should use inquire to prompt the user to confirm running the command.

The logic of winding down a stack based on the config should be abstracted into the Stage Manager orchestrator (doesn't exist yet). This allows the opup cli binary to build the stage manager from the config, and then be able to wind down the stack via a simple down() -> eyre::Result<()> call.

For example

StageManager::from(config).down()?;

The logic behind building a stage manager orchestrator from the stack config is so that it can handle finding existing running stack components based on the configuration. This separates concerns whereby the stack configuration solely handles component and stack configuration while the stage manager can handle orchestrating the stack stages/components.

Since the stage manager .down() call is very minimal, the down subcommand logic can be placed inside cli.rs alongside subcommand dispatching.

In the L1 Executor stage, we should add support for the Reth execution client.

This will require setting up the Dockerfile (and an entrypoint file if necessry) and checking that:

1. the image is built successfully
2. the container is created successfully
3. the container is started successfully
4. the ports are bound correctly, and the assigned RPC API port is responding

Description

When executing the directories stage, the monorepo clone fails if the directory already exists. It should only clone the repository if the directory does not exist and gracefully log a warning if an error would be thrown.

Seeing the following output:

Error: Failed to clone [email protected]:ethereum-optimism/optimism.git in "...": fatal: destination path 'optimism' already exists and is not an empty directory.

Description

The op_config crate offers a Config TOML that allows you to configure the OP Stack.

While this is useful for stack-wide configuration, op-up needs component-level TOML configs.

TOML is the config of choice to enable for configuration parsing into native rust so configs can be used as a library. This is a backwards-compatible alternative to defining custom dockerfiles for each component, and will allow for #3.

To make the TOMLs backwards-compatible, a field should allow for component TOMLs to reference base docker images or local Dockerfiles.

The TOML could look like the following

[metadata]
name = "challenger-go"
version = "1.1.2"

[command]
git = "[email protected]:ethereum-optimism/optimism.git"
# Default test hardhat deploy key - not safe for production environments
deployer = "ac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80"

[docker]
# The remote docker url will be tried first to fetch the container.
remote = "<remote dockerhub container url>"
# If the remote field is not set, or the fetch fails, the local dockerfile will be referenced.
local = "./Dockerfile.challenger-go"

# Below are inlined docker-file specific fields that are only used if the above remote or local fields are not specified.

[docker.build]
base = "golang:1.19.0-alpine3.15"
dependencies = [
    "make",
    "gcc",
    "musl-dev",
    "linux-headers",
    "git",
    "jq",
    "bash"
]
run = "make op-challenger GOOS=$TARGETOS GOARCH=$TARGETARCH"

[docker.exec]
base = "alpine:3.15"
command = ["/bin/sh", "/op-challenger-go-entrypoint.sh"]

[flags]
l1-eth-rpc = "http://l1:8545"
rollup-rpc = "http://rollup-client:8545"
l2oo-address = "$L2OO_ADDRESS"
dgf-address = "$DGF_ADDRESS"

Description

Creating this issue as a placeholder - it may be useful/necessary to put all docker-dependent code behind a docker feature flag.

Thinking: the op-composer internally has a Docker bollard reference that we may not want to "leak" to the stages crate unless the docker flag is enabled (could be by default unless necessary).

Effectively, a good start would be to add a docker feature flag to the stages crate and hide all docker-dependent code in the crate behind this flag.

Description

In the top-level readme, op-up has a section that is intended to detail how to use the project as a library.

This ticket is to flesh out the Using op-up as a Library documentation section in the top-level readme.

Description

Path localization happens in an unsafe way inside individual stage handlers, many of which are redundant. This was an initial port to make space for this nice refactor as a followup.

This ticket is to pull all monorepo directory management and path localization into the Monorepo primitive in a safe way and have the Monorepo be propagated down to individual stages.

Description

As part of #17, this task is to introduce a new clean subcommand to op-up that cleans stack artifacts and any additional generated files.

The logic of cleaning a stack based on the config should be abstracted into the Stage Manager orchestrator (doesn't exist yet). This allows the opup cli binary to build the stage manager from the config, and then be able to clean the stack via a simple clean() -> eyre::Result<()> call.

For example

StageManager::from(config).clean()?;

The logic behind building a stage manager orchestrator from the stack config is so that it can handle finding existing running stack components based on the configuration. This separates concerns whereby the stack configuration solely handles component and stack configuration while the stage manager can handle orchestrating the stack stages/components.

Since this logic should be minimal, it can be placed inside cli.rs alongside subcommand dispatching.

Description

op-up keeps a default stack.toml available at the repository's top level. This issue is to introduce a ci check that verifies the contents of stack.toml matches the default config. The default config should be generated as a toml and then the files can be diffed.

• After #14

Once we have our stack components laid out as TOML files, we will need to parse them into concrete Docker containers in order to start running, orchestrating and monitoring them.

With Bollard, we can use the Docker::create_container() method to prepare a container for a start operation. This requires parsing all image related data along with the rest of the Config.

Description

This is a fairly involved undertaking to document the full CLI usage. It should reference how Reth wrote their mdbook cli docs.

Description

Minimal documentation should be done inline with extensive markdown book docs to avoid duplicate, and out-of-sync documentation.

For this issue, the config crate readme.md file should be removed and minimal inline docs should be added to crates/config/src/lib.rs.

In the future, more extensive documentation will be added to documents in the docs/ mdbook.

Description

As part of #17, this task is to move the Stack logic in stack.rs to an up subcommand. Most of the time, op-up will be run without specifying this subcommand since it is redundant to execute op-up up.

The up subcommand itself should have a --devnet flag that can be used to override the local stack.toml and default config for a hard-coded devnet stack configuration.

Since this subcommand will have flags and some additional logic, it should be moved to a new up.rs file. So, just moving the contents of stack.rs to this new file and then dispatching the top-level cli.rs to this handler by default and when the up subcommand is provided.

Currently the main README includes this paragraph:

First, make sure you have a few things installed.

- [Rust](https://www.rust-lang.org/tools/install)
- [Docker](https://www.docker.com/)
- [Docker Compose](https://docs.docker.com/compose/)
- [Make](https://www.gnu.org/software/make/)
- [jq](https://jqlang.github.io/jq/)

Ideally op-up should be able to be run on by everyone on a new machine.

A potential solution includes the use of a custom Nix environment.
Never used this tool personally, so I'm also curious to learn and explore it for this use case.

A simpler solution is to just check if pre-requisites are installed (e.g. by using which x or x --version commands) and prompt the user to finish installing the ones that aren't yet on $PATH.