ADEPT

Source code and data for ADEPT: A DEbiasing PrompT Framework (AAAI-23).

An illustration of how debiasing works using ADEPT and for downstream tasks:

(a) While debiasing, ADEPT only trains the prompt parameters and keeps the base model frozen.

(b) When performing downstream tasks, ADEPT conditions the base model or both the prompt and the base model.

Replication

We conduct experiments on the bert-large-uncased pre-trained model from HuggingFace. By using ADEPT, we need only train 1.97M parameters when prompt-tuning with 40 prompt tokens, orders of magnitude smaller than the 335M parameters required for finetuning.

We provide bash scripts and codes to replicate our findings. Our environment is:

Ubuntu servers with NVIDIA GeForce RTX 3090 (24G) GPUs
cuda 11.1
packages with certain versions

Environment Setup

Create environment:

conda create -n debias python=3.8.5
conda activate debias

Install pytorch and python packages:

conda install -n pt2 pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch
cd ADEPT
pip install -r requirements.txt

Data

We've already included word lists for attributes in the ./data folder, so there is no need to acquire them from other resources. As for larger corpora, you can download News-Commentary v15 here and Hugging Face's BookCorpus replica here. New-Commentary alone can support gender debiasing. You may need to create a new text file by combining the two corpora mentioned above so that there are sufficient sentences for religion debiasing.

Experiments

Collect sentences:

cd script
bash ./collect_sentences.sh bert [corpus_path] gender final

Debias:

bash ./debias.sh bert 0 ADEPT gender # for ADEPT
bash ./debias.sh bert 0 DPCE gender # for DPCE

Preprocess corpus for plotting words' correlation (for better visualizing pairwise words' correlation, we highly suggest that you choose a large corpus, like a subset of BookCorpus sampled with function sample_sentences_from_bookcorpus in utils.py, because we have set the minimum threshold for $len(S_m^{a(i)})$ if word $w_m^{a(i)}$ is to be plotted):

bash ./preprocess_plot_word_correlation.sh bert 0 gender [corpus_path]

Plot words' correlation:

bash ./plot_word_correlation.sh bert 0 ADEPT gender [model_name_or_path] # for ADEPT
bash ./plot_word_correlation.sh bert 0 DPCE gender [model_name_or_path] # for DPCE