Comments (4)
Hi, thanks for your interest.
As mentioned in #8, please try the following:
- Switch to branch
apoorv-dump
instead ofmain
. This contains the latest code but is not cleaned, hence the large number of files. - In this branch, use
eval_accelerate.py
to evaluate (use sample size > 200). This saves all the generated predictions and scores to a scores file. - Use notebook
filtered_eval_from_scores.ipynb
which uses these saved scores to evaluate MRR etc. I would suggest you go through the notebook to see the exact way these are calculated (if you see any issues please do contact me)
The notebook contains the logic for MRR calculation from raw scores. Predictions are sorted by score, then 'seen' entities are removed from the prediction list (filtering), and then the rank of the correct entity is used to calculate MRR.
from kgt5.
I meant num_predictions=200 with beam_size=1 . Rest were all left default
from kgt5.
In this branch, use eval_accelerate.py to evaluate (use sample size > 200). This saves all the generated predictions and scores to a scores file.
@apoorvumang, I am sorry, did you mean beam_size or num_predictions instead of sample size? I didn't manage to find such parameter in your code and would like to make sure I understood you correctly. Also, what were other parameters equal to in your settings (like length_penalty, max_output_length, length_normalization, and of course # predictions and beam size)?
from kgt5.
hi, @apoorvumang
As you said, I set (num_reduction) to 10 in (eval_accelerate. py), which has caused my CUDA memory to exceed (30G). How do I set 200?
Thank you for your support!
from kgt5.
Related Issues (20)
- Implementing the KGT5 pipeline for my own constructed KG HOT 9
- What is the usage of --save_prefix wd5m-1gpu ?
- KGQA data split HOT 12
- Question about KGC datasets HOT 5
- Question about training in link prediction? HOT 1
- Question about QA-fine-tuning HOT 7
- KGC data split HOT 5
- 1to1 mapping split issues HOT 4
- Question about training step HOT 3
- Questions about Link Prediction Task HOT 16
- Verbalization procedure question
- Answering questions HOT 3
- Pretrained checkpoints for QA task
- How to get a one-to-one mapping between an entity/relationship and its text representation with mateQA HOT 1
- Tokenizer questions HOT 1
- questions of link-prediction
- Is there a checkpoint that I can use directly? HOT 1
- kgqa
- Questions about dataset HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kgt5.