Core implementation of Paper: "Order-Disorder: Imitation Adversarial Attacks for Black-box Neural Ranking Models".

• Python 3.8
• Pytorch==1.10.0
• transformers==4.6.1
• sentence-transformers==2.1.0
• apex==0.1
• tqdm
• nltk
• pytrec_eval
• trectools

• Tesla V100 32GB GPU x 8
• CUDA 11.2
• Memory 256GB

• MSMARCO Passage Ranking
• TREC Deep Learning Track 2019
• TREC Microblog Track
• Natural Question

• Data Processing Build Dev data into pickle file to speedup the evaluation.

  1. MSMARCO Passage Ranking Download MSMARCO Passage Ranking dataset. Download sub small dev set of MSMARCO, used for accerlating evaluation during training process

  python ./bert_ranker/dataloader/preprocess_pr.py

  2. TREC DL2019

  python ./bert_ranker/dataloader/preprocess_trec_dl.py

  3. TREC MB2014

  python ./bert_ranker/dataloader/preprocess_mb.py

  4. Natural Question

  python ./bert_ranker/dataloader.preprocess_nq.py

• Train Pairwise-BERT Ranker from scratch

  python ./bert_ranker/run_pairwise_ranker.py

• Get runs file (TREC Format) from the publicly available ranking model.

  python ./bert_ranker/dev_public_bert_ranker.py

  --mode determines which dataset to evaluate on --transformer_model determines which open source model to verify on. The default setting is "bert-large-uncased". There is an option, and we will also use the verification result later, which sets as "ms-marco-MiniLM-L-12-v2".

• Sample training data from runs file of public model to train imitation model.

  python ./bert_ranker/dataloader/sample_runs.py

• Train imitation model using sampled data.

  python ./bert_ranker/run_imitation_pairwise.py

• Evaluate the similarity between imitation model and victim model using runs file.

  python imitation_agreement.py

• Evaluate ranking performance using runs file Note that the evaluation metrics during training and development are not consistent with the official evaluation method. We get the standard ranking performance by official trec tools, which are implemented in trec_eval_tools.py

• The data preprocessing is implemented in ./adv_ir/data_utils.py. We need extract the query, query id, scores (imitation model), and target candidate passages from runs file.

• The Pairwise Anchor-based Trigger generation is implemented in ./adv_ir/attack_methods.py function name: pairwise_anchor_trigger()

• For generating adversarial triggers for ranking attack.

  python pat_attack.py --target=mini --imitation_model=imitate.v2  --nsp --lamba_1=0.6 --lambda_2=0.1 --num_beams=10 --topk=128 --mode=train

Note that we adopted the fine-tuned BERT LM from Song et al.(2020)

• Test the transferability of triggers

  python pat_attack.py --target=mini --imitation_model=imitate.v2  --nsp --lamba_1=0.6 --lambda_2=0.1 --num_beams=10 --topk=128 --mode=test

@inproceedings{liu2022order,
  title={Order-Disorder: Imitation Adversarial Attacks for Black-box Neural Ranking Models},
  author={Liu, Jiawei and Kang, Yangyang and Tang, Di and Song, Kaisong and Sun, Changlong and Wang, Xiaofeng and Lu, Wei and Liu, Xiaozhong},
  booktitle={Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security},
  pages={2025--2039},
  year={2022}
}