This repository contains the code for the paper in EMNLP 2021: "HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression".
git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
Download the vocabulary file of BERT-base (uncased) from HERE, and put it into ./pretrained_ckpt/
.
Download the pre-trained checkpoint of BERT-base (uncased) from HERE, and put it into ./pretrained_ckpt/
.
Download the 2nd general distillation checkpoint of TinyBERT from HERE, and extract them into ./pretrained_ckpt/
.
Download the GLUE dataset (containing MNLI) using the script in HERE, and put the files into ./dataset/glue/
.
Download the Amazon Reviews dataset from HERE, and extract it into ./dataset/amazon_review/
bash train_domain.sh
bash finetune_domain.sh
bash train_multi_domain.sh
And then put the checkpoints to the specified directories (see the beginning of finetune_multi_domain.py
for more details).
bash finetune_multi_domain.sh
If you find this code helpful for your research, please cite the following paper.
@inproceedings{dong2021hrkd,
title = {{HRKD}: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression},
author = {Chenhe Dong and Yaliang Li and Ying Shen and Minghui Qiu},
booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
year = {2021}
}