This repository contains the rfuzz research platform which was
created at UC Berkeley's ADEPT Lab
to investigate the use of coverage-directed fuzzing for RTL
pre-silicon testing.
The source code is release under a BSD-3-Clause license in order to allow for reproduction of experimental results as well as a basis for further research.
git clone https://github.com/ekiwi/rfuzz.git
cd rfuzz
# switch submodules to use HTTPS instead of SSH
sed -i 's/[email protected]:/https:\/\/github.com\//' .gitmodules
# initialize and update submodules
git submodule update --initNote: instead of setting up your system manually you can try the
Vagrantfile provided with this repository. For more information see
vagrantup.com.
On a fresh installation of Ubuntu 18.04 (Bionic Beaver) the following
installation steps were necessary:
- Install dependencies from the default repositories (requires root privileges):
apt update && apt upgrade
apt install build-essential meson pkg-config openjdk-8-jdk verilator cargo
apt install python3-toml python3-numpy python3-matplotlib graphviz- Install
sbtby following the official docs:
echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
sudo apt-get update
sudo apt-get install sbt- On a headless server: add the following to your
~/.config/matplotlib/matplotlibrc:
backend: agg
Note: rfuzz is developed on Fedora Linux Workstation
which thus also offers good support
In order to fuzz test a particular RTL design, we need to take the FIRRTL source code, instrument it and compile it into a fast RTL simulation using the verilator tool.
The exact build steps are encoded in the toplevel Makefile provided with
this repository.
In order to build the final binary you can use the pseudo target bin.
If you also want to start the binary you can use the pseudo target run
(WARNING: this will create the /tmp/fpga directory and will delete any
existing directory of the same name).
Thus to start the RTL simulation (also called the fuzz server)
for the default Sodor3Stage benchmark you just need to execute make run.
This should provide you with the following output:
rm -rf /tmp/fpga
mkdir /tmp/fpga
/home/ubuntu/rfuzz/build/ICache_server
Fuzz Server for ICache
Allocated Bytes per Input: 40
Allocated Bytes per Coverage: 30
created tx fifo
created rx fifo
This signals that the fuzz server is ready to receive test input from the fuzzer as well as provide coverage feedback.
The fuzzer is implemented in software and connects to the software simulation
fuzz server or the FPGA driver through shared memory.
As opposed to the fuzz server, the fuzzer itself is design agnostic and
thus only needs to be compiled once. The design specific information
is propagated through a TOML file generated alongside the fuzz server
when executing make bin (or make run).
To build the fuzzer change to the fuzzer directory and execute:
cargo build --releaseNote: the performance of release builds can be around 10x faster.
Do NOT run benchmarks with debug builds (the default configuration).
To see a list of options that the fuzzer supports run:
cargo run --release -- -hThis should provide you with an output similar to this:
kfuzz 0.1.0
Kevin Laeufer <[email protected]>
AFL-style fuzzer specialized for fuzzing RTL circuits.
USAGE:
kfuzz [FLAGS] [OPTIONS] <TOML> --output-directory <DIR>
FLAGS:
-h, --help Prints help information
-q, --print-queue Prints queue content at the end of a fuzzing run.
-c, --print-total-cov Prints the union coverage at the end of a fuzzing run.
-r, --random Generate independent random inputs instead of using the fuzzing algorithm.
-d, --skip-deterministic Skip all deterministic mutation strategies.
-n, --skip-non-deterministic Skip all non-deterministic mutation strategies.
-t, --test-mode Test the fuzz server with known input/coverage pairs.
-v, --version Prints version information
OPTIONS:
-s, --server-id <fuzz_server_id> The id of the fuzz server isntance to connect to. [default: 0]
-i, --input-directory <DIR> The output directory of a previous run from which to resume.
-j, --jqf-level <jqf_level> Select which level of JQF to apply. [default: 2] [possible values: 0, 1, 2]
-o, --output-directory <DIR> Used to log this session. Must be empty!
--seed-cycles <seed_cycles> The starting seed consits of all zeros for N cycles. [default: 5]
ARGS:
<TOML> TOML file describing the circuit being fuzzed
To quickly fuzz the default configuration, make sure that the fuzz server is running (see previous section) and then launch the fuzzer like this:
cargo run --release -- -c -o out ../build/Sodor3Stage.tomlTo terminate fuzzing use Ctrl+C and wait for the fuzzer to shut down
(this can take some time depending on how fast the design under test executes).
The FPGA acceleration is not working on the current main branch anymore.
Please have a look at the iccad18 branch instead.
Also, please feel free to contact the authors since the FPGA fuzzing
is not very well documented.
The scripts that were used to analyze the fuzzing results and generate
graphs for our ICCAD'18 paper can be found in the analysis/ directory.
Our raw results from running the fuzz server and fuzzer on the AWS cloud
can be found in the rfuzz-results
repository in the
jack
folder.
In order to ignore any functional changes that were made to rfuzz
after the ICCAD'18 submission, please make sure that you use the
iccad18 branch in the rfuzz repository.
In order to regenerate the graphs, make sure that your installation has a
graphical desktop environment (or add the appropriate code to dump graphs to disk
to the end of analysis.py).
Then make sure that the appropriate binaries are available:
make FIR=FFTSmall.fir DUT=FFTSmall bin
make FIR=Sodor1Stage.fir DUT=Sodor1Stage bin
make FIR=Sodor3Stage.fir DUT=Sodor3Stage bin
make FIR=Sodor5Stage.fir DUT=Sodor5Stage bin
make FIR=TLI2C.fir DUT=TLI2C bin
make FIR=TLSPI.fir DUT=TLSPI binNow you can run the analysis script like this:
./analysis.py ../../rfuzz-results/jack/Sodor3Stage.jqf1.seed5.random.out ../../rfuzz-results/jack/Sodor3Stage.jqf2.seed5.outThis will display the resulting graph which should be an exact copy of the one
printed in our paper (assuming you used the iccad18 branch).
It will also generate a mutation history graph for each fuzzing run,
e.g. 0.Sodor3Stage.out_mutations.png.
Note: the analysis.py script uses a version of the design with minimal
instrumentation and restarts the RTL simulator for each test input in order
to increase our confidence in the analysis results.
Invalid inputs as indicated by failing assumptions in the design
are automatically discarded and excluded from the coverage results.
Have a look at the scripts in the analysis/ directory to learn the details.
A collection of benchmarks in the form of
RTL circuits in the FIRRTL format
can be found in the benchmarks/ directory.
Please consult the local Readme to learn more about
how each individual benchmark was created.
Our Makefile takes the name of a FIRRTL (*.fir) file and the name
of the corresponding RTL toplevel module as parameters, e.g.:
make FIR=TLI2C.fir DUT=TLI2C runThe following benchmarks are available, benchmarks used in our ICCAD'18 paper
are listed in bold:
| Name | FIR |
DUT |
Description | Source |
|---|---|---|---|---|
| FFT Small | FFTSmall.fir |
FFTSmall |
FFT DSP circuit | fft |
| ICache | ICache.fir |
ICache |
instruction cache from Rocket Chip | rocket-chip |
| ICache w/ Coverage | ICacheCover.fir |
ICache |
ICache w/ user defined coverage annotations | rocket-chip |
| Non-Blocking Data Cache | NonBlockingDCache.fir |
NonBlockingDCache |
data cache from Rocket Chip | rocket-chip |
| Rocket Chip Tile | RocketTile.fir |
RocketTile |
64-bit RISCV CPU Tile | rocket-chip |
| Sodor 1-Stage | Sodor1Stage.fir |
Sodor1Stage |
educational RISCV core with 1 pipeline stage | riscv-sodor |
| Sodor 3-Stage | Sodor3Stage.fir |
Sodor3Stage |
educational RISCV core with 3 pipeline stages | riscv-sodor |
| Sodor 5-Stage | Sodor5Stage.fir |
Sodor5Stage |
educational RISCV core with 5 pipeline stages | riscv-sodor |
| I2C Peripheral | TLI2C.fir |
TLI2C |
I2C controller connected to TileLink bus | sifive-blocks |
| PWM Peripheral | TLPWM.fir |
TLPWM |
PWM timer connected to TileLink bus | sifive-blocks |
| SPI Peripheral | TLSPI.fir |
TLSPI |
SPI controller connected to TileLink bus | sifive-blocks |
| UART Peripheral | TLUART.fir |
TLUART |
UART controller connected to TileLink bus | sifive-blocks |
Note: the benchmarks that were not used in our paper have not been thoroughly evaluated and thus may not work reliably or may produce spurious results.
You can learn more about rfuzz in our ICCAD'18 paper which explains
how we adapted the coverage-directed fuzzing ideas from software testing
to RTL verification while building the basic research platform.
In order to reproduce our results, please make sure to use the iccad18
branch of this repository.
Kevin Laeufer and Jack Koenig and Donggyu Kim and Jonathan Bachrach and Koushik Sen. RFUZZ: Coverage-Directed Fuzz Testing of RTL on FPGAs. In International Conference On Computer Aided Design, 2018 (ICCAD'18), San Diego, CA, November 2018.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent