Cross-team EVM Benchmark Suite about evmjit HOT 11 CLOSED

Oh, this is awesome. It ties in to something that I've been trying to tackle; ability to fuzz VM implementations. Right now, it's very difficult to compare the inner workings of the VM:s. The straight-forward way to do it, is to run three full nodes on a private network (Go/Cpp/Py) and then submit blocks to them "normally". However, small VM differences may not be detected, since not all diffs lead to consensus issues (e.g. memory differences not written to state). And it's a PITA to setup and keep patched/updated.

The stand-alone binaries are a better alternative; as identical EVM code can be fed directly into them. I've been thinking to add per-step hash of the VM state (e.g hash of all VM internals; pc, memory, etc) and output as a stream of hashes, and aborting whenver a VM-state diff occurs.

What do you think, could these two ideas somehow be combined?

from evmjit.

This is a great idea. We should definitely do it, IMHO.

from evmjit.

LGTM. @chfast can you lead this effort? If you don't have time to do it, who will do the work? (writing the script, etc.)

from evmjit.

@holiman Standalone evm binaries exist right now for go-ethereum (evm) and webthree (ethvm), so in theory you can just do it now.

from evmjit.

yes, I know, it's not undoable. No python yet, though. But sure, I'll need to either reuse existing struct-logs (but I think they are not equal) or implement my hash-stream, and I'm not sure if they both support some way to set pre-existing state, e.g. existing contracts and/or environment variables.

from evmjit.

I just run 'time ethvm ... whatever.bin' for timings so I can factor out just the user time - no attempt at automated runs or reports. I have just a few small tests so far, intended to stress the particular things I'm working on. Seems we would need a larger collection of tests covering a wide and reasonably representative range of contracts. The tests need to check for correct output as well. And it has to be easy to pull single tests out of the suite to debug and profile them. So maybe the first thing is to agree on what a test should look like, so they can all be plugged into this framework. My few bad examples are here: https://github.com/gcolvin/libethereum/tree/develop/times

from evmjit.

I think this is a wonderful idea! A few notes, however.

I don't think that the ethvm or the evm binaries do an adequate job for doing benchmarks. If we want to do benchmarking properly we'd have to come up with a set of rules that specify exactly what is to be benchmarked (e.g. I don't think that setting up a VM should be taken in to account when doing the benchmarks), how long the benchmarks should run, how many passes, etc.

A few things that'd need to decide on is:

• Will the VM benchmarks have full state access (what is state?)
• Does it make use of the trie or some other mechanism
  • if other, what, and: should it be benchmarked
  • if trie, does it ever commit, how often, and: should it be benchmarked
• If capable of optimisations, should it run

Another thing that comes to mind is the Go "JIT" EVM. It requires compilation of byte code to instructions. Should that be taken in to the benchmark or should it only test the execution (I guess this also goes for the C++ JITVM).

I guess the question is: what should be benchmarked?

from evmjit.

What is the status of the Go "JIT" EVM, @obscuren? Is there some existing integration of http://github.com/ethereum/evmjit, or is that a different codebase?

Yeah - benchmarking of a JIT is slightly complicated, isn't it, given that that it will almost certainly be slower for single or small number of runs, but should be better over a higher number of runs.

Perhaps the benchmarking needs dividing into two groups - Interpreters and JITs, with the JITs having their setup and run times recorded separately? Because we are not comparing apples to apples otherwise.

from evmjit.

@obscuren, the idea is that tools will measure timings, not the Suite. We don't want to measure time of executable startup and test preparation, right? Imagine the following set of "VMs" to benchmark (example).

from evmjit.

The work is in progress, initial version is in https://github.com/ethereum/test-tools.

You can compare Go's VM with and without JIT by using evm tool.

from evmjit.

Anyone has any comments? Any ideas how to synchronize with https://github.com/karalabe/hive?

from evmjit.