This is the official GitHub repository for the following paper:
Learning with Different Amounts of Annotation: From Zero to Many Labels.
Shujian Zhang, Chengyue Gong, and Eunsol Choi.
EMNLP, 2021.
If using, please cite:
@article{zhang2021learning,
title={Learning with different amounts of annotation: From zero to many labels},
author={Zhang, Shujian and Gong, Chengyue and Choi, Eunsol},
journal={arXiv preprint arXiv:2109.04408},
year={2021}
}
This repository is based on the HuggingFace Transformers library.
Install packages:
pip install -r requirements.txt
To train a GLUE-style model using this repository:
TASK=snli
MODEL_OUTPUT_DIR=./model_output_snli_xxx/
Setting=xxx
Tra=xxx
Ft=xxx
python -m cartography.classification.run_glue \
-c configs/$TASK.jsonnet \
--do_train \
--do_eval \
--do_finetune \
--num_train_epochs $Tra \
--ft_num_train_epochs $Ft \
--label_propagation \
--setting $Setting \
--overwrite_output_dir \
--overwrite_cache \
-o $MODEL_OUTPUT_DIR
The best configurations for our experiments for each of the $TASKs (SNLI and MNLI) are provided under configs.
To select (different amounts of) data based on various metrics from training dynamics:
python -m cartography.selection.train_dy_filtering \
--filter \
--task_name $TASK \
--model_dir $PATH_TO_MODEL_OUTPUT_DIR_WITH_TRAINING_DYNAMICS \
--metric $METRIC \
--data_dir $PATH_TO_GLUE_DIR_WITH_ORIGINAL_DATA_IN_TSV_FORMAT
Supported $TASKs include SNLI, QNLI, MNLI and WINOGRANDE, and $METRICs include confidence, variability, correctness, forgetfulness and threshold_closeness; see paper for more details.
To select hard-to-learn instances, set $METRIC as "confidence" and for ambiguous, set $METRIC as "variability". For easy-to-learn instances: set $METRIC as "confidence" and use the flag --worst.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent