Dataset for the paper Incorporating Peer Reviews and Rebuttal Counter-Arguments for Meta-Review Generation.
├── ICLR_Submission (separated by years)
│ ├── ICLR_2017.json
│ ├── ICLR_2018.json
│ ├── ICLR_2019.json
│ ├── ICLR_2020.json
│ ├── ICLR_2021.json
│ └── ICLR_2022.json
│
├── MetaReview_Generation_Corpus (access by each submission year and forum id)
│ ├── 2020_H1gBhkBFDH.json
│ ├── 2019_rket4i0qtX.json
│ ├── ...
│ ├── ...
│ ├── ...
│ └── 2021_ASAJvUPWaDI.json
Do not upload ICLR_Submission since it exceeds the maximum file size limit. Can download it via the google drive link
This folder contains all the submission related data we crawl from OpenReview platform.
The raw data is available as
jsonfiles separated by its submission year.
Each submission can be accessed by its forum id
example of item access by the json dictionary
├── forum (Sy0GnUxCb - Unique id from Openreview)
│
├── submission_title (paper title)
│
├── reviews (subdict access with review id)
│ │
│ ├──(key) Sy0GnUxCb - 0
│ │ ├── review_id (Sy0GnUxCb - 0)
│ │ ├── review_title
│ │ ├── review (review content)
│ │ ├── rating (review score from 0 to 9)
│ │ │
│ │ ├── first_reply (rebuttal content)
│ │ │ ├── title
│ │ │ ├── tcdate (create time)
│ │ │ ├── tmdate (last modified time)
│ │ │ ├── number (thread order sorted by tcdate)
│ │ │ ├── id (thread id)
│ │ │ ├── replyto (reply content id)
│ │ │ ├── writer
│ │ │ ├── content
│ │ │ └── aspect_labels
│ │ │
│ │ ├── tcdate (review create time)
│ │ ├── tmdate (review last modified time)
│ │ │
│ │ ├── discussion_thread (list of discussion of the reviews)
│ │ │ └──list of discussion that same as first_reply structure
│ │ │
│ │ ├── conformity (review quality) (list of conformity score range from 1 to 4)
│ │ │ ├── WorkerId
│ │ │ └── rating (1 to 4)
│ │ │
│ │ ├── aspect_labels (list of aspect polarity)
│ │ │ ├── start position (character index)
│ │ │ ├── end position (character index)
│ │ │ └── polarity (motivation_positive)
│ │ │
│ │ ├── has_RR_pair (True, False) (Whether have RR alignment pair)
│ │ │
│ │ ├── Review_ADU (List of Review's ADUs with label)
│ │ │ ├── start (start index of ADU)
│ │ │ ├── end (end index of ADU)
│ │ │ ├── label (ADU label align with Reply label)
│ │ │ └── sent (ADU span)
│ │ │
│ │ └── Reply_ADU (List of Reply's ADUs with label similar to Review_ADU)
│ ├──
│
├── Decision (one of four)
│ ├──Accept (Poster)
│ ├──Accept (Spotlight)
│ ├──Accept (Oral)
│ └──Reject
│
└── MetaReview
The data we used to generated MetaReview
For each submission, we collect the review, rebuttal content, reviewers ratings, and the final decision.
The raw data is available as
jsonfiles separated by each submission with its ++submission year and forum id++.
| key | value | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| year | 2020 (Submission year) |
||||||||||
| forum | HkxlcnVFwB (Unique id from Openreview) |
||||||||||
| title | GenDICE: Generalized Offline Estimation of Stationary Values (Submission Paper Title) |
||||||||||
| decision | Accept (Oral) |
||||||||||
| meta_review | The authors develop a framework for off-policy value estimation for infinite horizon RL tasks, for estimating the stationary distribution of a Markov chain. Reviewers were uniformly impressed by the work, and satisfied by the author response. Congratulations! |
||||||||||
| reviews |
|
| Year | # Submissions | Avg Rating | Acceptance | Avg Meta-review Len |
|---|---|---|---|---|
| 2017 | 293 | 5.94 | 45.39% | 114.83 |
| 2018 | 677 | 5.70 | 43.72% | 104.56 |
| 2019 | 1153 | 5.69 | 41.63% | 147.11 |
| 2020 | 1807 | 4.68 | 34.86% | 128.92 |
| 2021 | 2208 | 5.62 | 35.73% | 182.96 |
| Total | 6138 | 5.40 | 37.93% | 148.42 |
- Aspect Typology Label
- Paper - Can We Automate Scientific Reviewing?
- Link - Dataset (Aspect Tagger)
- Argumentative Structure
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent