Giter Club home page Giter Club logo

Comments (1)

ns-moosavi avatar ns-moosavi commented on August 29, 2024

(1) Yes, that is correct.
Self-links, i.e. singleton links, are not exactly the same as single pair links though.
Self-links are only considered for mentions which do not have any links to other mentions.
If a singleton is connected to another mention in the system output, the resolution score is zero.

(2) The issue with your second point is that there are a lot of disagreements with regard to handling singletons in coreference resolution.
There are different annotation schemes for various coreference corpora, e.g. MUC, ACE, CoNLL, SEMEVAl-2010, in which singletons are handled differently.
The existence of various distinct annotation schemes for coreference resolution
is an indicator that there is a disagreement in defining coreference relations and the role of singletons in coreference resolution.

Our point of view in assigning self-links to singletons is that detecting all mentions that refer to an entity, i.e. either singletons or coreferent mentions, is much easier than recognizing coreferent mentions.
For instance, all singletons in the SEMEVAl-2010 corpus are annotated automatically and by heuristics.

We design LEA in a way that it does not reward the recognition of referring expressions regardless of their coreference decisions.
The recognition of a singleton is rewarded only if it is also recognized as a singleton in the system output, not when it is assigned to some random output entity.
Similarly, the recognition of a coreferent mention is rewarded only if it is linked to at least one of its correct coreferring mentions, not when it is somewhere in the system output either as a singleton or a coreferent mention.
If you equally consider self-links for all entities, the existence of any referring expression, singleton or coreferent, in the system output will be rewarded no matter how it is classified.

However, one may disagree with our point of view and wants to treat all entities, singletons or coreferent entities, equally.
This way the resolution score would be resolution_score = (n(n-1)/2) +1, and there would be no special self-links for singletons.
As long as one uses links instead of mentions, LEA is different from B3.

(3) LEA is designed for intra-document coreference evaluation.
For each entity, it considers the "importance" of the entity in the document and the "resolution score" of the entity within the document, i.e. intra_resolution_score.
I believe it makes sense to also incorporate an inter_resolution_score, i.e. whether the entity is resolved correctly to a knowledge base ID, If you want to adapt LEA for coreference resolution in the context of entity linking.
There could be various ways of combining inter_resolution_score and intra_resolution_score.
For instance
LEA = \sum_{entities} (importance * intra_resolution_score * (W_1 + inter_resolution_score)) / \sum_{entities} ((1+W1) * importance)

or

LEA = \sum_{entities} importance * (W_2 * inter_resolution_score + W_3 * intra_resolution_score) / \sum_{entities} importance
where W_2+W_3 = 1

from neleval.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.