BERT for Coreference Resolution: Baselines and Analysis

Contribution summary

• Joshi et al. proposed BERT-based CR method
• to employ BERT's ability of passage-level understanding.
• The model achieved SOTA on the GAP and OntoNotes benchmarks. The qualitative analysis showed that (1) handling pronouns in conversations and (2) mention paraphrasing are still difficult for the model.

Authors

Mandar Joshi, Omer Levy, Daniel S. Weld, and Luke Zettlemoyer
(University of Washington, AI2, FAIR)

Motivation

• BERT's major improvement is passage-level training, which allows it to better model longer sequences
• Can we apply it to CR task?

Method

• Proposed BERT-based CR method.
• Two ways of extending the c2f-coref, ELMo-based CR model:
  • The independent variant uses non-overlapping segments each of which acts as an independent instance for BERT
  • The overlap variant splits the document into overlapping segments so as to provide the model with context beyond 512 tokens

Results / Insight

Dataset

• GAP: human-labeled dataset of pronoun-name pairs from Wikipedia snippets
• OntoNotes 5.0: document-level dataset from the CoNLL-2012

Results

• Achieved SOTA on the GAP and OntoNotes benchmarks
  • with +6.2 F1 (baseline: BERT+RR) and +0.3 F1 (baseline: EE)
• The overlap variant offers no improvement over independent

Insight

• Unable to handle conversations: Modeling pronouns especially in the context of conversations (Table 3), continues to be difficult for all models, perhaps partly because c2f-coref does very little to model dialog structure of the document.
• Importance of entity information: The models are unable to resolve cases requiring mention paraphrasing.
  • E.g., Bridging the Royals with Prince Charles and his wife Camilla likely requires pretraining models to encode relations between entities

Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answering

Contribution summary

• Kim et al. proposed the CQA method which jointly encodes the raw and rewritten question
• in order to infuse explicit guidance on resolving conversational dependencies,
• and achieved +1.2 F1 on both QuAC and CANARD.

Authors

Gangwoo Kim, Hyunjae Kim, Jungsoo Park, and Jaewoo Kang (Korea University)

Motivation

Two types of existing methods: (1) end-to-end and (2) QR-based pipelined model.

• In (1), the end-to-end model jointly encodes the evidence document, the current question, and the whole conversation history. However, the model has limitations to do so without explicit guidance on how to resolve these dependencies.
• In (2), QA models are dependent on QR models; hence QA models suffer from the error propagation from QR models

Method

• Proposed "unified framework": method jointly encode them to train QA models with consistency regularization.
• Specifically, when original questions are given, QA models are encouraged to yield similar answers to those when self-contained questions are given

Results / Insight

• Main results:
  • Improves the QA performance +1.2 F1 on QuAC (RoBERTa, end-to-end) and CANARD (RoBERTa, pipelined)
  • QR did not work well on QuAC, where human rewritten query is not available for train QR
    • What hideaki-j learned: if human rewritten query is not available, then go for end-to-end approach.
• Transferability:
  • Tested on CoQA, where question distributions are significantly different from QuAC and CANARD.
  • Results: improved QA performance (+0.6 for RoBERTa, CoQA)

Asking Clarification Questions in Knowledge-Based Question Answering

Contribution summary

[Author] did [Method] to solve [Motivation] and found [Insight]

Authors

Motivation

Existing method is/has/uses ...

Method

Proposed method is/has/uses ...

we formulate three tasks in our dataset, including clarification identification, clarification question generation, and clarificationbased question answering

• New clarification dataset, CLAQUA, with nearly 40K open-domain examples.
• The dataset supports three serial tasks:
  1. given a question, identify whether clarification is needed;
  2. if yes, generate a clarification question;
  3. then predict answers base on external user feedback.

Results / Insight

Coreferential Reasoning Learning for Language Representation

Contribution summary

[Author] did [Method] to solve [Motivation] and found [Insight]

Authors

Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, Zhiyuan Liu
(Tsinghua University, Beijing National Research Center for Information Science and Technology, Beijing Academy of Artificial Intelligence, Tencent)

Motivation

Existing method is/has/uses ...

Method

Proposed method is/has/uses ...

Mention reference prediction (MRP): this is a novel training task which is proposed to enhance coreferential reasoning ability. MRP utilizes the mention reference masking strategy to mask one of the repeated mentions and then employs a copybased training objective to predict the masked tokens by copying from other tokens in the sequence.

Results / Insight

BERT with History Answer Embedding for Conversational Question Answering

Contribution summary

[Author] did [Method] to solve [Motivation] and found [Insight]

Authors

Chen Qu¹, Liu Yang¹, Minghui Qiu², W. Bruce Croft¹, Yongfeng Zhang³, Mohit Iyyer¹
(¹University of Massachusetts Amherst, ²Alibaba Group, ³Rutgers University)

Motivation

Existing method is/has/uses ...

Method

Proposed method is/has/uses ...

• Add the history answer embedding (HAE) to BERT’s word embeddings.
• HAE encodes the information of the answer spans from the previous questions.
  (This explanation is from "Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answering")

Results / Insight

End-to-end Neural Coreference Resolution (EMNLP17)

Contribution summary

[Author] did [Method] to solve [Motivation] and found [Insight]

Authors

Kenton Lee† , Luheng He† , Mike Lewis‡ , and Luke Zettlemoyer†∗
(†Univ. of Washington, ∗AI2, and ‡FAIR)

Motivation

Existing method is/has/uses ...

• All recent coreference models, including neural approaches that achieved impressive performance gains (Wiseman et al., 2016; Clark and Manning, 2016b,a), rely on syntactic parsers, both for head-word features and as the input to carefully hand-engineered mention proposal algorithms.

Method

Proposed method is/has/uses ...

• The key idea is to directly consider all spans in a document as potential mentions and learn distributions over possible antecedents for each.
• Scoring all span pairs in our end-to-end model is impractical.
  • Therefore we factor the model over unary mention scores and pairwise antecedent scores, both of which are simple functions of the learned span embedding.

Results / Insight

Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph

Contribution summary

[Author] did [Method] to solve [Motivation] and found [Insight]

Authors

Motivation

Existing method is/has/uses ...

Method

Proposed method is/has/uses ...

constructs clarification questions base on predicate-independent templates

Results / Insight

CorefQA: Coreference Resolution as Query-based Span Prediction

Contribution summary

[Author] did [Method] to solve [Motivation] and found [Insight]

Authors

Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, Jiwei Li
(Zhejiang University, Stanford, ShannonAI)

Motivation

Existing method is/has/uses ...

Method

Proposed method is/has/uses ...

Results / Insight

Personal Knowledge Base Construction from Text-based Lifelogs

Contribution summary

• Yen et al. proposed automatic personalized knowledge graph (PKG) creation method using Chinese Twitter data,
• targeting both explicitly and implicitly mentioned general life events.
• Each module was confirmed to work for extracting lifelog events, however, the pipelined system achieved the F1 score of 15.63%, suggesting the difficulty of this task.

Authors

• Anzi Yen (National Taiwan University), Hen Hsen Huang, and Hsinhsi Chen (National Chengchi University)

Motivation

• Previous work focuses on the detection of major life events such as marriage and graduation.
• However, general life events such as dining and visiting a local place remain to be solved.

Proposed method

Proposed system has two stages.

• Stage 1: Decide if the tweet contains life events and detect predicates
• Stage 2: Extract subject, object, and time.

Results / Insight

• Evaluation data: Chinese Twitter data with 18 users (8K tweets).
• F1 score is 15.63% for pipelined system.

Coreference Resolution without Span Representations (ACL21)

Contribution summary

• Kirstain et al. proposed s2e (start-to-end) coref model, a simple and efficient transformer-based coref model,
• to mitigate large memory footprint
• and achieved three times better memory efficiency.

Authors

Yuval Kirstain, Ori Ram, and Omer Levy (Tel Aviv University)

Motivation

• The previous method uses span representation to perform coref, which requires O(n²d) space and much GPU memory.

Method

• The proposed method uses only span boundaries (ie, its start and end tokens); here there is no representation for each span (but just for boundaries), leading smaller space.

Results / Insight

• Three times more (GPU) memory efficient, without deteriorating the performance.

Cross-document Coreference Resolution over Predicted Mentions

Contribution summary

• Cattan et al. proposed the first end-to-end model for CD coref and set a baseline for it,
• to create CD coref model which can be applied to realistic settings, and
• achieved high efficiency without relying on external resources, however, performance has room for improvement

Authors

Arie Cattan¹, Alon Eirew^1,2, Gabriel Stanovsky³, Mandar Joshi⁴, Ido Dagan¹
(1) Bar Ilan University, (2) Intel Labs, Israel, (3) The Hebrew University of Jerusalem, (4) University of Washington

Motivation

Cross-document (CD) coreference resolution remained relatively under-explored

Issues

• Existing CD coref models rely on gold mentions or external resources, such as SRL (semantic role labeling) and paraphrase dataset, preventing them from being applied in realistic settings.
• Limited size of the CD dataset, here ECB+

Method

Proposed the first end-to-end model for CD coref and set the baseline for it
(note that they set END-TO-END CD coref baseline, not CD coref in general).

• Addressed data size limitation of ECB+: Pre-train the mention scorer sm() on the gold mention spans (see also the insight below)
• Addressed memory constraints: Freezed output representations from RoBERTa instead of fine-tuning all parameters

Results / Insight

Proposed model is simpler and substantially more efficient than existing models, however,
the performance has room for improvement (existing 85.6 vs. proposed 54.4 for CoNLL F1)

Insight

• If a dataset size is not sufficient, then train the modules separately (should avoid end-to-end optimization)
  • "Skipping the pre-training of the mention scorer results in a 3.2 F1 points drop in performance. Indeed, the relatively small training data in the ECB+ dataset might be not sufficient when using only end-to-end optimization, and pretraining of the mention scorer helps generate good candidate spans from the first epoch."

Higher-order Coreference Resolution with Coarse-to-fine Inference

Contribution summary

[Author] did [Method] to solve [Motivation] and found [Insight]

Authors

Motivation

Existing method is/has/uses ...

Method

Proposed method is/has/uses ...

Results / Insight

Joint Detection and Coreference Resolution of Entities and Events with Document-level Context Aggregation

Contribution summary

• Kriman et al. proposed IE (including entity CR) model which handles full-document as a context,
• to understand cross-sentence dependencies, and
• achieved higher performance than baseline model, OneIE.

Authors

Samuel Kriman and Heng Ji (University of Illinois at Urbana-Champaign)

Motivation

• Existing methods focus on extracting information from sentences or paragraphs, meaning cannot consider full-document context
• To improve performance, Incorporating cross-sentence dependencies is needed

Method

• Proposed IE model that can be used for IE subtasks, including entity CR (coreference resolution), at the document level
• Input is a document and output are mention-entity and/or mention-events pairs
• Steps
  • Generate representation for the mentions, including pronouns
  • ?

Results / Insight

• Dataset: ACE05-E+, OIE dataset including pronoun mention annotations
• Entity and event trigger extraction results
  • Higher than baseline model of OneIE
• Coref results
  • F1 B³_sys: 84.83
  • No baseline is set

Questions

• How do the representations are used to identify corresponding entities and events?

Toward Conversational Query Reformulation

Contribution summary

• Kiesel et al. analyzed the query reformulation in conversational search by casting the reformulation as CRUD (create, read, update, delete) operation
• to understand the principles of "editing conversations", and
• found that ambiguities in the reformulations will be a major challenge for conversational search.

Authors

Johannes Kiesel1 , Xiaoni Cai1 , Roxanne El Baff2 , Benno Stein1 and Matthias Hagen3

1. Bauhaus-Universität Weimar, Bauhausstraße 11, 99423 Weimar, Germany
2. German Aerospace Center (DLR), Germany
3. Martin-Luther-Universität Halle-Wittenberg, Von-Seckendorff-Platz 1, 06120 Halle (Saale), Germany

Motivation

• In conversational search, users "describe" their changes to a query; however, the principles of such “editing conversations” have not been analyzed

Method

• Cast reformulations (i.e., "edit") as meta-queries that imply operations on the original query and categorize the operations following the standard CRUD terminology (create, read, update, delete)
• Crowdsourced a dataset with 2694 human reformulations across four search domains

Results / Insight

• Ambiguities in the reformulations will likely be a major challenge for conversational search