cd tools
nuget restore tools.sln
xbuild
Each Mono configuration requires a .conf
file. The files in the configs
directory are examples. The JSON structure is as follow :
- Name : name of the config (must be unique across all configs and benchmarks)
- Count : number of time to run the benchmark (optional, default : 5)
- Mono : path to the mono executable (optional, default to system one)
- MonoOptions : command line parameters to pass to the mono runtime (optional)
- MonoEnvironmentVariables : environment variables to set to run the benchmark (optional)
- ResultsDirectory : path to the results directory, relative to the benchmarker repository root directory (optional, default to results/)
Each benchmark requires a .benchmark
file. The files in the benchmarks
directory are examples. The JSON structure is as follow :
- Name : name of the benchmark (must be unique across all configs and benchmarks)
- TestDirectory : path to the working directory to run the benchmark, relative to the benchmarker repository root directory
- CommandLine : command line to run the benchnark, does not contain the runtime (mono) executable
- Timeout: benchmark specific timeout, override the command line one
To compare two or more revisions and/or configurations directly, use tools/compare.exe
:
./compare.exe [parameters] [--] <tests-dir> <results-dir> <benchmarks-dir> <config-file> [<config-file>+]
Where :
- tests-dir : path to tests/ directory
- results-dir : path to results/ directory
- benchmarks-dir : path to benchmarks/ directory
- config-file : path to a configuration file to run, there is one or more
Store the graph to "graph.svg" in current directory by default.
The new JSON results format is as follow :
- DateTime : date and time at which this benchmark was run
- Benchmark : copy data of the
benchmarks/*.benchmark
corresponding file- Name : name of the benchmark
- TestDirectory : working directory to use to run the benchmark, relative to tests/
- CommandLine : command line parameters to pass to the benchmark
- Timeout : timeout specific to this benchmark, in seconds
- Config : copy data of the
configs/*.conf
corresponding file- Name : name of the config
- Count : number of time to run the benchmark
- Mono : path to the mono executable
- MonoOptions : command line parameters to pass to the mono runtime
- MonoEnvironmentVariables : environment variables to set to run the benchmark
- ResultsDirectory : path to the results directory, relative to the benchmarker repository root directory
- Version : standard output when run with
--version
runtime command line parameter - Timedout : true if any of the run of the benchmark has timed out
- Runs : collections of the runs for the benchnark, size is equal to Config.Count
- WallClockTime : wall clock time taken to run the benchmark
- Output : standard output of the benchmark
- Error : standard error of the benchmark