asjqmasjqm / convabuse Goto Github PK

Data from the paper ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI by Amanda Cercas Curry, Gavin Abercrombie, and Verena Rieser.

Link to paper

Please cite as:

@inproceedings{cercas-curry-etal-2021-convabuse,
title = "{C}onv{A}buse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational {AI}",
author = "Cercas Curry, Amanda and
Abercrombie, Gavin and
Rieser, Verena",
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2021",
address = "Online and Punta Cana, Dominican Republic",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.emnlp-main.587",
doi = "10.18653/v1/2021.emnlp-main.587",
pages = "7388--7403"
}

We provide two versions of the dataset:

1. ConvAbuseEMNLPfull.csv: The full dataset, including repeat annotations used to calculate intra-annotator agreement. Each row represents the response of an individual annotator
2. ConvAbuseEMNLPtrain.csv, ConvAbuseEMNLPvalid.csv, ConvAbuseEMNLPtest.csv: The train, validation, and test splits used for the experiments in the paper. Each row includes the annotations of all the annotators who labelled that example.

All annotations are presented in binary format (1/0). Annotators only labelled secondary tasks (e.g. abuse type, target etc.) in cases they had considered to be abusive (i.e. -1/-2/-3).

The columns are:

In (2), the dataset is divided into train, vailidation, and test splits. (2) contains the same fields as (1), but each row includes all the annotators responses, with the annotator response column headers preceded by the annotator ID, e.g. Annotator1.is_abuse.-1. It also includes an extra column is_abuse_majority, representing the majority vote of the annotators on the binary abuse/not abuse task.

Note that for privacy reasons, we provide data from CarbonBot and E.L.I.Z.A only. We are unable to release the examples from Alana used in the paper.