This is the Repo for the paper: Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluation

Human evaluation serves as the gold standard for assessing the quality of Natural Language Generation (NLG) systems. Nevertheless, the evaluation guideline, as a pivotal element ensuring reliable and reproducible human assessment, has received limited attention. Our investigation revealed that only 29.84% of recent papers involving human evaluation at top conferences release their evaluation guidelines, with vulnerabilities identified in 77.09% of these guidelines. Unreliable evaluation guidelines can yield inaccurate assessment outcomes, potentially impeding the advancement of NLG in the right direction. To address these challenges, we take an initial step towards reliable evaluation guidelines and propose the first human evaluation guideline dataset by collecting annotations of guidelines extracted from existing papers as well as generated via Large Language Models (LLMs). We then introduce a taxonomy of eight vulnerabilities and formulate a principle for composing evaluation guidelines. Furthermore, a method for detecting guideline vulnerabilities has been explored using LLMs, and we offer a set of recommendations to enhance reliability in human evaluation. The annotated human evaluation guideline dataset and code for the vulnerability detection method are publicly available online.

The main contribution of this paper is as follows:

1. We are the first to study vulnerabilities in human evaluation guidelines and release the first human evaluation guideline dataset with annotated vulnerabilities for advancing reliable human evaluation;

2. We analyze the existing human evaluation guidelines and introduce a taxonomy of eight vulnerabilities for evaluation guidelines; Furthermore, we establish a principle for writing a reliable human evaluation guideline;

3. We explore an LLM-based method for detecting guideline vulnerabilities. We recommend employing this method to assess the reliability of the guidelines before conducting human evaluations;

4. We present a set of recommendations designed to elevate the reliability of human evaluation by offering guidance on writing robust guidelines and identifying potential vulnerabilities.

Please cite our work if you find it useful.

@inproceedings{ruan2024defining,
  title={Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluation},
  author={Ruan, Jie and WangWenqing, WangWenqing and Wan, Xiaojun},
  booktitle={Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  pages={7958--7982},
  year={2024}
}