Comments (1)
(1) Yes, that is correct.
Self-links, i.e. singleton links, are not exactly the same as single pair links though.
Self-links are only considered for mentions which do not have any links to other mentions.
If a singleton is connected to another mention in the system output, the resolution score is zero.
(2) The issue with your second point is that there are a lot of disagreements with regard to handling singletons in coreference resolution.
There are different annotation schemes for various coreference corpora, e.g. MUC, ACE, CoNLL, SEMEVAl-2010, in which singletons are handled differently.
The existence of various distinct annotation schemes for coreference resolution
is an indicator that there is a disagreement in defining coreference relations and the role of singletons in coreference resolution.
Our point of view in assigning self-links to singletons is that detecting all mentions that refer to an entity, i.e. either singletons or coreferent mentions, is much easier than recognizing coreferent mentions.
For instance, all singletons in the SEMEVAl-2010 corpus are annotated automatically and by heuristics.
We design LEA in a way that it does not reward the recognition of referring expressions regardless of their coreference decisions.
The recognition of a singleton is rewarded only if it is also recognized as a singleton in the system output, not when it is assigned to some random output entity.
Similarly, the recognition of a coreferent mention is rewarded only if it is linked to at least one of its correct coreferring mentions, not when it is somewhere in the system output either as a singleton or a coreferent mention.
If you equally consider self-links for all entities, the existence of any referring expression, singleton or coreferent, in the system output will be rewarded no matter how it is classified.
However, one may disagree with our point of view and wants to treat all entities, singletons or coreferent entities, equally.
This way the resolution score would be resolution_score = (n(n-1)/2) +1, and there would be no special self-links for singletons.
As long as one uses links instead of mentions, LEA is different from B3.
(3) LEA is designed for intra-document coreference evaluation.
For each entity, it considers the "importance" of the entity in the document and the "resolution score" of the entity within the document, i.e. intra_resolution_score.
I believe it makes sense to also incorporate an inter_resolution_score, i.e. whether the entity is resolved correctly to a knowledge base ID, If you want to adapt LEA for coreference resolution in the context of entity linking.
There could be various ways of combining inter_resolution_score and intra_resolution_score.
For instance
LEA = \sum_{entities} (importance * intra_resolution_score * (W_1 + inter_resolution_score)) / \sum_{entities} ((1+W1) * importance)
or
LEA = \sum_{entities} importance * (W_2 * inter_resolution_score + W_3 * intra_resolution_score) / \sum_{entities} importance
where W_2+W_3 = 1
from neleval.
Related Issues (20)
- prepare-tac should merge multiple candidates at same location HOT 4
- Support evaluating with incomplete gold standards
- Within-document evaluation mode for cross-doc coreference evaluation
- Feature Request: Specifying multiple metrics for file HOT 1
- Citation? HOT 1
- I am a KBP
- The results obtained with this procedure are not the same as the official results of TAC HOT 2
- The order of lines in evaluate output can change HOT 1
- Entity CEAF true positives and false positives off by factor of 2. HOT 3
- i can't run the shell scripts δΈθ¦
- i can't run the shell scripts HOT 1
- Rewrite neleval.summary to use dataframes to store previous results
- Duplicate NIL cases result in wrong results
- use streams rather than strings to output from __call__
- improve test coverage
- SPHINXOPTS=-W in travis doesn't appear to be working
- Reimplement coref metrics in terms of contingency matrix and missing/spurious weights
- prepare-conll-coref does not convert AIDA-YAGO2-dataset HOT 5
- Add data validation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from neleval.