Comments (4)
Hi, thanks for your interest.
evaluate.py does evaluation in the 'traditional' sense - it calculates log probabilities of all entity sequences (5M) using the Transformer model and then ranks these entities. It does not generate predictions, hence the long time.
Please use the code in eval_accelerate.py for the sampling generation technique (which is described in the paper as well). evaluate.py is not the code used for the numbers reported in the paper, eval_accelerate.py is used. Please see the Readme for more details. I would be happy to clarify any other doubts as well.
from kgt5.
Thanks for your reply! But it seems that eval_accelerate.py just gives the unfiltered Hit@1, could you provide the official implementation for calculating MRR and filtered H@N? Appreciate your help!
from kgt5.
For that I would suggest you try the following:
- Switch to branch
apoorv-dump
instead ofmain
. This contains the latest code but is not cleaned, hence the large number of files. - In this branch, use
eval_accelerate.py
to evaluate (use sample size > 200). This saves all the generated predictions and scores to a scores file. - Use notebook
filtered_eval_from_scores.ipynb
which uses these saved scores to evaluate MRR etc. I would suggest you go through the notebook to see the exact way these are calculated (if you see any issues please do contact me)
I will be cleaning the code/adding documentation etc. for this soon. Meanwhile I suggest you try the scores + notebook approach. Let me know if this works.
from kgt5.
It seems to work great! Thank you so much for your patience!
from kgt5.
Related Issues (20)
- Implementing the KGT5 pipeline for my own constructed KG HOT 9
- What is the usage of --save_prefix wd5m-1gpu ?
- KGQA data split HOT 12
- Question about KGC datasets HOT 5
- Question about training in link prediction? HOT 1
- Question about QA-fine-tuning HOT 7
- KGC data split HOT 5
- 1to1 mapping split issues HOT 4
- Question about training step HOT 3
- Questions about Link Prediction Task HOT 16
- MRR calculation HOT 4
- Verbalization procedure question
- Answering questions HOT 3
- Pretrained checkpoints for QA task
- How to get a one-to-one mapping between an entity/relationship and its text representation with mateQA HOT 1
- Tokenizer questions HOT 1
- questions of link-prediction
- Is there a checkpoint that I can use directly? HOT 1
- kgqa
- Questions about dataset HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kgt5.