This is the official code for paper Text Representation Distillation via Information Bottleneck Principle.

This code is based on the SimCSE code

transformers==4.2.1
torch==1.7.1
scipy
datasets
pandas
scikit-learn
prettytable
gradio
setuptools

Before training, you need to prepare the training data for both the distillation stage and the fine-tuning stage.

Toy data for each stage can be found in the data folder:

• unsup_toy_sents.txt: Training sentences for the distillation stage. Each line represents a sentence.

• toy_embs.pt: Embedding of the teacher model. This is a N * D matrix, where N is the number of training sentences and D is the dimension of the teacher model.

• sup_toy_data.csv: Training data for the fine-tuning stage. Each line contains three sentences: the anchor text, the positive sentence, and the negative sentence. The sentences are separated by commas.

bash run_unsup.sh

bash run_sup.sh

bash eval.sh

We have published our model checkpont on modelscope: