This repository contains the code for DiscoEval Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations (EMNLP 2019).

The structure of this repo:

• train: the training code
• discoeval: the Discourse Evaluation framework
• data: the DiscoEval evaluation datasets

The pretrained models with different training signals can be downloaded from https://drive.google.com/file/d/1I0wFkNb2fmoC7kcj-FPxyVkHKyfX-MoX/view?usp=sharing

The training data (generated from Wikipedia) can be downloaded from

https://drive.google.com/open?id=1WPRJylC7PzLtYcg8-PMNX_ZUNNRkO3Bp

Evaluation example code

train/discoeval_example.py

The code is tested under the following environment/versions:

• Python 3.6.2
• PyTorch 1.0.0
• numpy 1.16.0

Some code in this repo is adopted from SentEval.

You may notice some difference from the above table with our camera-ready version appeared on EMNLP 2019. The differences are: we removed the hidden states in SSP (previously 2000 by mistake), we regenerated the SP dataset (previously the sentence orders were shuffled, now the sentences are in the original order except the first sentence).

@inproceedings{mchen-discoeval-19,
  author    = {Mingda Chen and Zewei Chu and Kevin Gimpel},
  title     = {Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations},
  booktitle = {Proc. of {EMNLP}},
  year      = {2019}
}