This repo is for our ACL 2023 paper "Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation". We provide the scripts for our RoSE benchmark and meta-evaluation.

Please visit here for a demo page of this project.

• RoSE 🌹 Benchmark
  • ACU Annotations
  • Human Annotations with Different Evaluation Protocols
• Meta-Evaluation
  • correlation.py
  • stat_test.py
  • power_analysis.py
  • demo.py
• Citation

RoSE can be downloaded with Hugging Face Datasets under Salesforce/rose. We provide a notebook, demo.ipynb, for basic usage of our dataset.

RoSE benchmark contains system outputs annotated with our ACU protocol. It contains four parts:

• CNNDM, test set annotations
• CNNDM, validation set annotations
• XSum, test set annotations
• SamSum, test set annotations

We summarize the statistics below.

We have system outputs annotated with four different human evaluation protocols in total. We summarize them below.

We annotated two sets of system summaries.

1. Summaries of 12 fine-tuned systems. The huggingface data split name is cnndm_protocol.
2. Zero-shot summaries from large language models (GPT3, T0), together with summaries from BRIO and BART. The huggingface data split name is cnndm_protocol_gpt3.

We provide scripts for statistical analysis of the meta-evaluation results in our paper.

functions for computing correlation coefficients

functions for conducting statistical tests, including bootstrap, permutation test and computing confidence interval

functions for computing power analysis, please note that computing power analysis can be time-consuming, and please maximize the number of processes to speed up the computation

demo script for utilizing the functions in the above files

Please cite our paper if you use RoSE in your work:

@inproceedings{Liu2022RevisitingTG,
  title={Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation},
  author={Yixin Liu and Alexander R. Fabbri and Pengfei Liu and Yilun Zhao and Linyong Nan and Ruilin Han and Simeng Han and Shafiq R. Joty and Chien-Sheng Wu and Caiming Xiong and Dragomir R. Radev},
  booktitle={Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics},
  year={2023},
}