mds1 / convex-shutdown-simulation Goto Github PK

Simulates a call to Convex Finance's system shutdown method, which uses about 16M gas

Makefile 2.14% Solidity 3.72% TypeScript 61.66% Shell 16.20% Dockerfile 16.28%

convex-shutdown-simulation's Introduction

Convex System Shutdown Benchmarks

This repository benchmarks performance of various Ethereum development frameworks by simulating a call to Convex's systemShutdown method. This method uses about 16M gas and performs a number of token transfers

Benchmarks

Benchmarks were run at mainnet block 14,445,961. Blocknative only simulates against the latest block (support for simulation at historical blocks is planned), so the Blocknative simulation was run first and other benchmarks were run against the same block.

Notes on benchmarks:

"Remote RPC" benchmarks used Alchemy as the RPC provider, connected over HTTP.
"Local RPC" benchmarks used a local Erigon node.
"Cached" benchmarks use RPC responses that the framework caches locally instead of making HTTP requests. We assume that since Blocknative and Tenderly are infrastructure providers, they aggressively cache and load hot data in-memory to improve performance, which is why their results are also in the "Cached" column.
Benchmarks were performed on macOS 11.6.2 with a 2.3 GHz 8-Core Intel Core i9 and 32 GB 2667 MHz DDR4.

Framework	Version
Blocknative	HTTP request on 2021-03-23
Dapptools	dapp 0.35.0, hevm 0.49.0
Ganache	7.5.0
Hardhat	2.12.3
Foundry	forge 0.2.0 (250cc85 2022-12-02T00:18:26.200251Z)
Tenderly	HTTP request on 2021-03-23

Framework	Remote RPC	Local RPC	Cached
Blocknative	N/A	N/A	0m3.529s
Dapptools	52m17.447s	17m34.869s	3m25.896s
Ganache	7m24.070s	0m29.631s	0m15.665s
Hardhat	6m17.896s	0m25.642s	0m9.110s
Foundry	5m55.141s	0m11.606s	0m0.999s
Tenderly	N/A	N/A	0m1.9315s

Notes on gas usage:

Ganache, Hardhat, and Tenderly all agree on gas usage after refunds and are therefore likely the truth value.
Foundry usage matches those values: The debugger shows the actual call costs 22,558,945 gas. Adding 21,000 intrinsic gas and 64 gas for calldata (both are excluded from the reported gas number by default) gives the same amount reported by Hardhat and others. The remaining 2995 gas is overhead from the contract based setup and Solidity-generated call checks.
Blocknative's gas usage does not account for refunds as this is not yet supported by their platform.
Dapptools' gas usage also does not account for refunds.

Framework	Gas Used
Blocknative	26,668,845
Dapptools	24,066,128
Ganache	22,580,009
Hardhat	22,580,009
Foundry	22,561,940
Tenderly	22,580,009

Usage

Run cp .env.example .env, and in the resulting .env file enter a URL to an Ethereum archive node in the ETH_RPC_URL environment variable. (Alchemy provides free archive node data). Also fill out the TENDERLY_* variables to benchmark the Tenderly API
Run yarn to install dependencies for Ganache and Hardhat
Install Foundry's forge and Dapptools using the installation instructionshere and here respectively
Run dapp update to install dependencies for Dapptools and Foundry
Run any command in the Makefile to benchmark that tool. For example, use make benchmark-hardhat to run the simulation against Hardhat. Alternatively, run make benchmark-all to run all tools

Tips

Set export CLEAR_CACHE=1 in your .env file to clear the Ganache and Hardhat caches
Consider running the benchmarks via Docker. See the comment header in the Dockerfile for details.

convex-shutdown-simulation's People

Contributors

Stargazers

Watchers

Forkers

gnidan nomicfoundation feugenea alexcampbelling blocknative micaiahreid alcuadrado malcador jeffsmale90 bingshuangs

convex-shutdown-simulation's Issues

Out of gas exception when running simulation on Ganache

The transaction runs out of gas (hits block limit of 30M) during simulation with Ganache (I am using Alchemy as a node provider).
script output:

$  ./scripts/convex.ganache.ts
yarn run v1.22.15
Setting up Ganache...
  eth_chainId
  eth_chainId
  eth_call
  eth_accounts
  eth_sendTransaction

    Transaction: 0x22ba0b17256c047bf139d53b98a6cdbb8a1bee5ec10e1573883588c805e5c820
    Contract created: 0x324c828fbc98e722afd73ec91fbf4263eeccb0dc
    Gas usage: 64323
    Block number: 14121829
    Block time: Thu May 19 2022 18:26:28 GMT-0400 (Eastern Daylight Saving Time)

  eth_chainId
  eth_getTransactionReceipt
  eth_blockNumber
  eth_chainId
  eth_sendTransaction

    Transaction: 0xf9f9c5a0f511df85a01319364f0cfd157130c290eaf06fd89b263e43777a0245
    Contract created: 0xd62f3dfaf07eb471fbd4ab88bfe2c6c8a7dc8b3c
    Gas usage: 64323
    Block number: 14121830
    Block time: Thu May 19 2022 18:26:29 GMT-0400 (Eastern Daylight Saving Time)

  eth_chainId
  eth_getTransactionReceipt
  evm_addAccount
  personal_unlockAccount
setup-ganache: 3.101s

Simulating shutdown...
  eth_sendTransaction
  
    Transaction: 0x8617941537d4f6ddd9934752b93cbf7b335969f1c87956ccbfb110a018aeaaf9
    Gas usage: 30000000
    Block number: 14121831
    Block time: Thu May 19 2022 18:26:29 GMT-0400 (Eastern Daylight Saving Time)
    Runtime error: out of gas

I replicated the logic using Brownie with the same network parameters, but increased the block gas limit to 100M, and found it used around 60M gas:

Running 'scripts/sim.py::main'...
Executing system shutdown...
Transaction sent: 0x3007e01e625dd6e40d30890496ae5defc3206145de1636a7ee079f96b988d536
  Gas price: 0.0 gwei   Gas limit: 100000000   Nonce: 1
  Booster.shutdownSystem confirmed   Block: 14445963   Gas used: 60707716 (60.71%)

It would be great to know why my results differ from what's given in the readme.

fix: Hardhat is estimating gas

Since logging was enabled in 2bb428a, you can now see in the run of the benchmark that Hardhat is running an eth_estimateGas call.

fix: hardcode `owner` address

Some simulations query the node for this, some hardcode the address, and some just overwrite the owner with a local account. For consistency, let's just hardcode the owner address (the from address used) in each simulation for now

ci: setup github actions

Ideally the CI will run the benchmarks and save results somewhere. However, @alcuadrado has mentioned the performance of GH runners may fluctuate, though this may only occur between jobs and not within the same job. Before relying on CI to report the benchmarks, let's perform a series of benchmarks locally and within a single CI job and ensure the relative performance between tools is similar in the two

docs: specify that the RPC endpoint is HTTP/S

When foundry-rs/foundry#1003 is merged Forge will support WebSocket and IPC endpoints which should (in theory, at least) significantly speed up forking, since instead of closing and re-opening the RPC at every request, we would keep the connection alive.

As a sidenote, would it make sense to try and compare with a WebSocket endpoint as well? IPC might be a bit too much.

feat: report benchmarks of each tool

For each tool, benchmark scripts should report

full execution time, i.e. including tool startup time
execution time of just the transaction simulation (not sure if this is possible for each tool)
gas used
some simple state verification, e.g. check logs or traces, or perhaps we just check this one manually for now?

feat: look into websocket support and add websocket benchmarks

This sim relies on a lot of network requests, which are done over HTTPS. Foundry is adding websocket support, and if Hardhat and Ganache both support websockets as well we can add a separate table with websocket benchmarks

fix: ganache simulation fails with out of gas

Discovered by @feuGeneA in #4, moving tracking of that issue here.

Ganache simulation fails with out of gas error, which @gnidan and @davidmurdoch are planning to look into. For reference, here's how much gas used is reported when simulating this transaction with various tools:

Gas usage at block 13,724,056:

Foundry: 15,515,282 (using the fix in foundry-rs/foundry#524)
Hardhat: 15,344,483
Dapptools: 16,779,553 (I don't think this accounts for refunds)
Tenderly: 15,537,677 (not in this repo, but will check it manually using their simulator)