This project aims to report incorrect/phantom statistics generated by the llvm static compiler. We wrote a LLVM IR pass that reorders basic blocks randomly in the hope to see if it will fire varied statistics.

The experiment is as follows

This is in stats_seen_so_far.txt

This script reads the statistics files, diffs them and checks if there are any different stats from the global register.

Take a source bitcode file, call is source.bc and compile it two times, with/without our reordering pass and store the statistics in both cases. If we see a diff in the statistics (using diff_and_check.py), then the script returns status interesting (exit 1) for that source file otherwise (exit 0).

The parent.py script invokes llvm-reduce with script.sh to check if the input is interesting. If so, we save it. We also update the global register with new statistics that were seen and continue for all source files. In our experiment, the source files were named with integer values, for instance, 0.bc, 1.bc ... There is a dataset in samples

Each numbered folder in INTERESTING_FILES is a unique source file. The folder contains the reduced part of the original bitcode file.

For instance, 1/59489066aba24d4321ea37e2bf9882233ce9374c83ffc490b09bbf23eda4b16f.bc

The llvm IR has the extension .ll For instance, 1/59489066aba24d4321ea37e2bf9882233ce9374c83ffc490b09bbf23eda4b16f.ll

original.bc -- The bitcode file compiled with opt using -O3

reordered.bc -- The bitcode file compiled with opt using -O3 and the random basic block permutation pass

original.ll -- The corresponding llvm IR

reordered.ll -- The corresponding llvm IR

original.stats -- The statistics by obtained compiling original.bc with llc using -O3

reordered.stats -- The statistics obtained by compiling reordered.bc with llc using -O3

set_difference.stats -- Stores the diff of the statistics

original.s -- assembly of orignal file

reordered.s -- assembly of reordered file

For example, the source file 1 has the following diff in set_difference.stats

pass - description: <before> vs <after>

branch-folder       - Number of branches optimized      :  1   vs Nil  
dagcombine          - Number of dag nodes combined      :  Nil vs 4

• LLVM version with basic block reordering: https://github.com/vidsinghal/llvm-project/tree/shuffle_bb
• The LLVM version is added as a submodule.
• Make sure you checkout to shuffle_bb branch.
• Build LLVM with -stats enabled.
• A dataset of source files to test, we have around 1000 files in samples.

• Python3 parent.py runs the experiment.

Vidush Singhal

Johannes Doerfert