Comments (3)
The "UMLS full" model is certainly not the one reference in the paper. It has been trained relatively recently.
I wasn't part of the team back then, but the model they would have used would almost certainly have been a "v0.x" model. I don't think these are still publicly available anywhere. Though I could be wrong.
from medcat.
Thank you for the answer, @mart-r,
Ah ok, yea I understand that those early models are not available anymore. But it was nice to see some performance metrics of the model.
Are the available models validated on any annotated corpora? I would be nice to see their metrics.
I am running the models against ShARe/CLEF and MedMentions and get a lower performance than the original model presented in the publication.
ShareClef:
Model | R | P | F1 |
---|---|---|---|
Publication model | 0.688 | 0.796 | 0.74 |
UMLS small | 0.47 | 0.76 | 0.58 |
UMLS full | 0.24 | 0.67 | 0.47 |
MedMentions:
Model | R | P | F1 |
---|---|---|---|
Publication model | 0.500 | 0.406 | 0.448 |
UMLS small | 0.12 | 0.38 | 0.18 |
UMLS full | 0.17 | 0.67 | 0.23 |
An extraction was correct if the span overlapped and the CUI was correct, as reported in the publication: " For each manual annotation we check whether it was detected and linked to the correct Unified Medical Language System (UMLS) concept.".
What do you think is the reason for this drop in performance?
from medcat.
The model(s) used in the paper were trained in self-supervised as well as supervised capacity.
However, no supervised training was performed for the models publicly available through the link in the README.
This may not account for all the performance drop, but probably most of it.
The MedMentions model that's used within the tutorials (e.g Part 3.2) did receive some supervised training (on the MedMentions dataset). You can try that one as well.
But it also won't be identical to the models within the paper.
As with most machine learning models, a model is best at the task it's produced to perform. These public models weren't produced to break any performance records. They are there for use case demonstration.
Though in general, for discussion, I recommend the discourse:
https://discourse.cogstack.org/
Not many people monitor the GitHub repo issues.
There's also some discussion about public models and their training in there already.
from medcat.
Related Issues (20)
- How to use CUI filtering
- Version 1.6.1 is not on pypi HOT 1
- new install attempt with SNOMED model throws spacy/thinc config validation error HOT 4
- remove elasticsearch dependency HOT 5
- Show nested ents results in error HOT 2
- Modifing the weights of words in the models HOT 3
- i am running NER-L tutorial but while running this cell, model and dataset are not downloading. HOT 1
- Model resources no longer available. HOT 4
- TypeError: Snomed._refset_df2dict() takes 1 positional argument but 2 were given HOT 1
- Concept not found if token order is slightly changed contrary to mentioned note in paper HOT 3
- Install MedCAT with pytorch cpu HOT 2
- Error: [E050] when trying to load sample annotate entity project
- Stopwords do not load properly HOT 4
- How do I download the models? HOT 3
- Service unavailable HOT 4
- How to use SNOMED trained model HOT 4
- Writing SNOMED dataframe - cannot access local variable in preprocess_snomed file HOT 4
- ```cat.cdb.print_stats()``` returns empty HOT 10
- Hard coded SNOMED release file bug. HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from medcat.