NetEaseCrowd: A Dataset for Long-term and Online Crowdsourcing Truth Inference

We introduce NetEaseCrowd, a large-scale crowdsourcing annotation dataset based on a mature Chinese data crowdsourcing platform of NetEase Inc.. NetEaseCrowd dataset contains about 2,400 workers, 1,000,000 tasks, and 6,000,000 annotations between them, where the annotations are collected in about 6 months. In this dataset, we provide ground truths for all the tasks and record timestamps for all the annotations.

NetEaseCrowd dataset is built based on a gesture comparison task. Each task contains three choices, where two are similar gestures and the other one is not. Annotators are required to pick out the different one. An example is shown below:

Compared with the existing crowdsourcing datasets, our NetEaseCrowd dataset has the following characteristics:

The basic statistics of NetEaseCrowd dataset and other previous datasets are as follows:

Two ways to access the dataset:

• Directly download overall NetEaseCrowd in Hugging Face [Recommended]

• Under the data/ folder, the NetEaseCrowd dataset is provided in partitions in the csv file format. Each partition is named as NetEaseCrowd_part_x.csv. Concat them to get the entire NetEaseCrowd dataset.

In the dataset, each line of record represents an interaction between a worker and a task, with the following columns:

• taskId: The unique id of the annotated task.
• tasksetId: The unique id of the task set. Each task set contains unspecified number of tasks. Each task belongs to exactly one task set.
• workerId: The unique id of the worker.
• answer: The annotation given by the worker, which is an enumeric number starting from 0.
• completeTime: The integer timestamp recording the completion time of the annotation.
• truth: The groundtruth of the annotated task, which, in consistency with answer, is also an enumeric number starting from 0.
• capability: The unique id of the capability required by the annotated taskset. Each taskset belongs to exactly one capability.

For the privacy concerns, all sensitive content like as -Ids, has been anonymized.

In the example above, there are three annotations, all from the same taskset 6980 and the same task 1012658482844795232. Three annotators, with ids 64, 150, and 263, provide annotations of 2, 1, and 0, respectively. They do the task at different time. The truth label for this task is 1, and the capability id of the task is 69.

We test several existing truth inference methods in our dataset, and detailed analysis with more experimental setups can be found in our paper.

The NetEaseCrowd dataset has been integrated into the crowd-kit (with pull request here), you can use it directly in your code with the following code(with crowd-kit version > 1.2.1):

from crowdkit.aggregation import DawidSkene
from crowdkit.datasets import load_dataset

df, gt = load_dataset('netease_crowd')

ds = DawidSkene(10)
result = ds.fit_predict(df)

print(len(result))
# 999799

We provide a curated list for other public datasets towards truth inference task.

If you use this project in your research or work, please cite it using the following BibTeX entry:

@misc{wang2024dataset,
      title={A Dataset for the Validation of Truth Inference Algorithms Suitable for Online Deployment}, 
      author={Fei Wang and Haoyu Liu and Haoyang Bi and Xiangzhuang Shen and Renyu Zhu and Runze Wu and Minmin Lin and Tangjie Lv and Changjie Fan and Qi Liu and Zhenya Huang and Enhong Chen},
      year={2024},
      eprint={2403.08826},
      archivePrefix={arXiv},
      primaryClass={cs.HC}
}

The NetEaseCrowd dataset is licensed under CC-BY-SA-4.0.