A standardized benchmark suite for auto-tuners

BAT is a standardized benchmark suite for auto-tuners that is based on benchmarks from SHOC and contains benchmarks for CUDA programs. The benchmarks are for both whole programs and kernel-code. BAT will save all your JSON and CSV results to an own results directory after auto-tuning is completed. Then it will parse specified files and print out the best parameters found by the auto-tuner. The parameters and other benchmarking information will be printed out prettified in the terminal.

This benchmark suite will be useful for you if you're making your own auto-tuner and want to use the benchmarks for testing or would like to compare your auto-tuner to other known auto-tuners. BAT can also be used to check how a parameter's value changes for different architectures.

Parameters and search space for the algorithms can be seen in the src directory here.

• Python 3 (Or Docker, see section Within a Docker container)

Without using Docker, the following steps are required to download and install the auto-tuners:

• OpenTuner
  • Can be downloaded along other needed dependencies by calling pip3 install -r requirements.txt from the tuning_examples/opentuner directory.
• Kernel Tuner
  • Can be downloaded along other needed dependencies by calling pip3 install -r requirements.txt from the tuning_examples/kernel_tuner directory.
• CLTune
  • Need to set the environment variable KTT_PATH=/path/to/KTT for using the benchmarks.
• KTT
  • Need to set the environment variable CLTUNE_PATH=/path/to/CLTune for using the benchmarks.

# Run all benchmark for all auto-tuners
python3 main.py

# Run the `sort` benchmark for all auto-tuners
python3 main.py -b sort

# Run all benchmarks for auto-tuner `OpenTuner`
python3 main.py -a opentuner

# Run benchmark `scan` for auto-tuner `CLTune`
python3 main.py -b scan -a cltune

Default: none

Benchmark to run. Example: sort. If no benchmark is selected, all benchmarks are ran for selected auto-tuner(s).

Default: none

Auto-tuner to run benchmarks for. Example: ktt. If no auto-tuner is selected, all auto-tuners are selected for benchmarking.

Default: false

If all stdout and stderr should be printed out during building of the benchmark(s). By default it does not print out the information during the building.

Default: 1

Problem size for the data in the benchmarks. By default it uses a problem size of 1. This is up to the specific auto-tuner to handle.

Default: brute_force

Tuning technique to use for benchmarking. If no technique is specified, the brute force technique is selected. This is up to the specific auto-tuner to handle.

It is easy to add new auto-tuner implementations for the benchmarks, just follow these steps:

1. Implement the benchmark(s) you want with your auto-tuner. If your auto-tuner tunes a whole program, the benchmarks can be found in src/programs. However if you have an auto-tuner that tunes kernels, the benchmarks can be found in src/kernels, and you have to generate the input data. Generating of input data can be done like in the KTT examples found here.
2. Store your auto-tuner implementation of a benchmark inside a auto-tuner subdirectory in tuning_examples. The path to the benchmark implementation should look similar to ./tuning_examples/kernel_tuner/sort/.
3. Create a config.json file in the same directory as the auto-tuner with content similar to this:

{
    "build": [
        "make clean",
        "make"
    ],
    "run": "./sort",
    "results": [
        "best-sort-results.json"
    ]
}

• build: A list of commands that will be ran before the run command. Note, it does not work correctly with && between commands. This is because of a limitation in the package subprocess to run the commands in Python. A solution is therefore to split them in a list.
• run: The command to run the auto-tuning benchmark.
• results: A list of result files that contains the best parameters found in the auto-tuner benchmark. These will be printed out by BAT after the auto-tuning is completed.

Here are some examples of how to build the different auto-tuner Docker images:

# Build OpenTuner Dockerfile
$ docker build -t bat-opentuner -f docker/opentuner.Dockerfile .

# Build Kernel Tuner Dockerfile
$ docker build -t bat-kernel_tuner -f docker/kernel_tuner.Dockerfile .

# Build CLTune Dockerfile
$ docker build -t bat-cltune -f docker/cltune.Dockerfile .

# Build KTT Dockerfile
$ docker build -t bat-ktt -f docker/ktt.Dockerfile .

Here are some examples of how to run the different auto-tuner Docker containers:

# Run the KTT container
$ docker run -ti --gpus all bat-ktt

# Example of running container detatched
$ docker run -d -ti --gpus all bat-ktt

# Open a shell into a detatched container
$ docker exec -it <container-id> sh

# After this the commands shown in the `Running benchmarks` section can be used
# Example:
$ main.py -b sort -a ktt -t mcmc -s 4

Data from running BAT with different auto-tuners can be found here.

Use the following citation when publishing work using BAT.

@article{sund_bat_2021,
	title = {{BAT}: A Benchmark suite for {AutoTuners}},
	rights = {Copyright (c) 2021},
	issn = {1892-0721},
	url = {https://ojs.bibsys.no/index.php/NIK/article/view/915},
	pages = {44--57},
	number = {1},
	journaltitle = {Norsk {IKT}-konferanse for forskning og utdanning},
	author = {Sund, Ingunn and Kirkhorn, Knut A. and Tørring, Jacob O. and Elster, Anne. C},
	urldate = {2021-12-10},
	date = {2021-11-14},
	langid = {english},
	note = {Number: 1},
}