l-zhe / btmpg Goto Github PK

Code for paper Pushing Paraphrase Away from Original Sentence: A Multi-Round Paraphrase Generation Approach by Zhe Lin, Xiaojun Wan. This paper is accepted by Findings of ACL'21.

License: MIT License

Python 100.00%

acl acl2021 paraphrase paraphrase-generation natural-language-generation natural-language-processing deep-learning paper

btmpg's Introduction

BTmPG

Code for paper Pushing Paraphrase Away from Original Sentence: A Multi-Round Paraphrase Generation Approach by Zhe Lin, Xiaojun Wan. This paper is accepted by Findings of ACL'21. Please contact me at [email protected] for any question.

Dependencies

PyTorch 1.4
NLTK 3.5

Model

Create Vocabulary

You should first create a vocabulary from your corpora. You can use the following command.

python createVocab.py --file ~/context/train.tgt ~/context/train.src  \
                      --save_path ~/context/vocab.pkl \
                      --vocab_num 50000

Train

You can train your model leveraged the following command:

python train.py --cuda --cuda_num 5 \
                --train_source ~/context/train.src \
                --train_target ~/context/train.tgt \
                --test_source  ~/context/test.src \
                --test_target  ~/context/test.tgt \
                --vocab_path ~/context/vocab.pkl \
                --batch_size 32\
                --epoch 100 \
                --num_rounds 2 \
                --max_length 110 \
                --clip_length 100 \
                --model_save_path ~/context/output/model.pth \
                --generation_save_path ~/context/output

Inference

After training, you can leverage the following command to generate multi-round paraphrase.

python generator.py --cuda      --cuda_num 3 \
                    --source ~/context/test.src \
                    --target ~/context/test.tgt \
                    --vocab_path ~/context/vocab.pkl \
                    --batch_size 64 \
                    --num_rounds 10 \
                    --max_length 60 \
                    --model_path ~/context/model.pth \
                    --save_path ~/context/output/

We also provide the pretrain-model file in releases page.

Result

Case Study

Reference

If you use any content of this repo for your work, please cite the following bib entry:

@inproceedings{lin-wan-2021-pushing,
    title = "Pushing Paraphrase Away from Original Sentence: A Multi-Round Paraphrase Generation Approach",
    author = "Lin, Zhe  and
      Wan, Xiaojun",
    booktitle = "Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021",
    month = aug,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.findings-acl.135",
    doi = "10.18653/v1/2021.findings-acl.135",
    pages = "1548--1557",
}

btmpg's People

Contributors

Stargazers

Watchers

Forkers

xennygrimmato c00renut huiyangzhou gpt2l timothhhyf

btmpg's Issues

Data Split

For Quora, there are actually 149,263 samples in total, rather than the data split reported in the paper (129,263\3k\3k). Is there a reason why not to use the full dataset? Thanks.

Lower performance with retrained model

When I use a checkpoint that I've trained from scratch instead of the checkpoint downloaded from here, performance is ~2 iBLEU lower. The command used to train the model was:

python train.py --cuda \
                --train_source ./data/qqp_train.src \
                --train_target ./data/qqp_train.tgt \
                --test_source  ./data/qqp_dev.src \
                --test_target  ./data/qqp_dev.tgt \
                --vocab_path ./checkpoints/qqp.vocab \
                --batch_size 8 \
                --epoch 100 \
                --num_rounds 2 \
                --max_length 50 \
                --clip_length 50 \
                --model_save_path ./checkpoints/qqp.model \
                --generation_save_path ./outputs/qqp/

Are there additional hyperparameters that I need to set?

Traceback (most recent call last):
  File "train.py", line 182, in <module>
    generation_save_path=args.generation_save_path)
  File "/disk/nfs/ostrom/s1717552/btmpg/utils/run.py", line 133, in __call__
    self.run()
  File "/disk/nfs/ostrom/s1717552/btmpg/utils/run.py", line 100, in run
    max_length=self.max_length)
  File "/disk/nfs/ostrom/s1717552/btmpg/model/VAE.py", line 206, in round
    out_embed = self.embed(self.GS(sentence[:, -1:, :]))
  File "/disk/nfs/ostrom/s1717552/btmpg/btmpgenv/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/disk/nfs/ostrom/s1717552/btmpg/model/gumbleSoftmax.py", line 17, in forward
    sigma = min(self.tau_max, (self.tau_max ** (self.n / self.N)))
OverflowError: (34, 'Numerical result out of range')

This happens after a few days of training, around epoch 39 for MSCOCO and epoch 77 for Quora.

The command used was:

python train.py --cuda \
                --train_source ./data/qqp_train.src \
                --train_target ./data/qqp_train.tgt \
                --test_source  ./data/qqp_dev.src \
                --test_target  ./data/qqp_dev.tgt \
                --vocab_path ./checkpoints/qqp.vocab \
                --batch_size 8 \
                --epoch 100 \
                --num_rounds 2 \
                --max_length 50 \
                --clip_length 50 \
                --model_save_path ./checkpoints/qqp.model \
                --generation_save_path ./outputs/qqp/