The code demo of DistXplore

We have tested DistXplore based on Python 3.7 on Ubuntu 20.04, theoretically it should also work on other operating systems. To get the dependencies of DistXplore, it is sufficient to run the following command.

pip install -r requirements.txt

The version of the library 'protobuf' maybe unsuitable, you can run the following command to fix it.

pip install --upgrade protobuf==3.20

This directory contains the core implementation of DistXplore, the subdirectory profile and seeds provides the a test subject model and initial seeds. (You can download the other pretrained model files from https://drive.google.com/drive/folders/1rgZA2xuMLhcYE40u4llWMxqEsew4rbzb?usp=sharing)

The training data and test data of MNIST, Fashion-MNIST and CIFAR can be directly loaded from keras, and the data of SVHN can be downloaded from the google drive above and loaded by the function load_svhn() in the code, the directory should be set.

This directory contains the implementation of three adversarial defense techniques: DISSECTOR, Attack-as-Defense and Data transformation.

This directory contains the code of model retrain and retrain model evaluation.

We provide a script to generate distribution-aware test samples for LeNet4 model trained on MNIST dataset. You can download other models from the google drive mentioned above. test

test
cd DistXplore/dist-guided
sh generate_demo.sh

cd DistXplore/defence/dissector
python merge_output_tech.py
       -mode mnist
       -tech bim
       -truth 0
       -target 1

The meaning of the options are:

1. -mode: the type of the dataset

2. -tech: the technique to generate the test samples

3. -truth: for the targeted techniques, set the truth label

4. -target: for the targeted techniques, set the target label

Change the directory of data in merge_output_tech.py to change test cases generated by different tools.

The dependencies of A2D is different, see the requirements in defence/attack as defence.

Firstly, to get the attack cost, run

cd DistXplore/defence/attack as defence/scripts
python get_attack_cost_mnist.py

Then, use the cost recorded to detect the adversarial samples

python attack_as_defense_detector.py
       --dataset mnist
       -d knn
       -a JSMA

The meaning of the options are:

1. --dataset: the type of the dataset

2. -d: detetor type

3. -a: attack to use; recommanded to use JSMA, BIM or BIM2.

4. --init: for the first run, add this argument to train the detector

Change the directory of data in get_attack_cost_mnist.py and attack_as_defense_detector to change test cases generated by different tools.

The data dir and model dir should be set first.

python mnist_finetune_diversity.py
       -ft_epoch 20

-ft_epoch: the num of the retrain epoch

python evaluae.py

To run adversarial attack baselinse, the required dependencies are listed in baseline/adv/requirements.txt. The seeds are in the single_cluster_seeds.zip. Please set the model dir and seed dir in each python files.

cd baseline/adv
sh adv_attack_mnist.sh

The shell file contains :

python adv_attack_test_mnist.py -truth 0 -target 1

-truth：the truth label of the seeds

-target: the attack target label

To run deephunter baselinse, the required dependencies are listed in baseline/deephunter/requirements.txt. The seeds are in the seeds.zip. The MNIST LeNet-5 model is contained. You can download the other models from https://drive.google.com/drive/folders/1rgZA2xuMLhcYE40u4llWMxqEsew4rbzb?usp=sharing.

cd baseline/deephunter/deephunter
sh deephunter.sh

The shell file contains :

python image_fuzzer.py -i ../test_seeds/mmd_ga_seed_svhn 
-o ./deephunter_outputs/svhn_resnet_ga_kmnc_iter_5000_efficient/outputs_50 
-model svhn_resnet 
-criteria kmnc 
-max_iteration 5000 
-random 0 
-select prob 
-gpu_index 0 
--save_path ./tmp/svhn_vgg

-i：seeds dir

-o: output dir

-model: model dir

-criteria: coverage criteria (KMNC, NBC or SA)

--save_path: the dir to save the temp data of SA

After each deephunter generation, to get the prepared test cases, run:

python prepare_crash_dataset.py

The dir of model and data should be set first in the python file.

Download the model and dataset from https://drive.google.com/drive/folders/1rgZA2xuMLhcYE40u4llWMxqEsew4rbzb?