Comments (6)
It is hard to estimate training time for a dataset since each requires different number of epochs to reach reasonable performance. For FB15k-237, we did not achieve good performance even if we kept training indefinitely - best we achieved was around 0.27 MRR. So for comparison I'll give you the numbers for WN18RR:
For WN18RR, if we use definitions as part of entity, we had median tokenized entity length of around 20 tokens. We trained t5-small arch. model using 4 1080Ti GPUs (11GB memory each) with 64 batch size (so total 64*4 training examples per batch), max input and output seq length=60. In this setting, each training epoch takes roughly 4 minutes (about 173k examples in train set if you count both head and tail prediction). Also, each epoch consists of 679 training steps here.
The evaluation was done at 200k training steps = ~294 epochs, which is about 20 hours of training. However you do not need to train for this long to get reasonable results. Also, you can converge faster and achieve better results if you initialize with the pretrained LM weights (intfloat/SimKGC#1 (comment)) but then you can't be sure whether the model has seen the test data during pretraining or not.
By comparison, our results for Wikidata5M were with training for 50 epochs, while for WikiKG90Mv2 were for 1.5 epochs - however these training runs went on for approx 10 days since the datasets are quite large.
I hope this gives you some idea about the training time.
from kgt5.
Thank you for the detailed response! On a related note (with SimKGC), did you also try KGT5 with pretrained weights?
from kgt5.
Yes we did. For WikiKG90Mv2 it didn't make much difference except faster convergence (for T5-small. for larger models I don't know what will happen)
from kgt5.
Okay, just out of curiosity - was there any specific reason of not using pretrained KGT5 for all the experiments?
from kgt5.
2 reasons
- For smaller KGs, if we use default tokenizer then a lot of tokens are never seen in the train triples. And if you want to use a custom tokenizer like we do with Wikidata5M so that whole vocab gets finetuned, then you can't use pretrained weights (as far as I know there's no easy solution to this).
- People can have an argument (and rightly so) that model is predicting test triples correctly because it read Wikipedia during pretraining, and WikiData is based off of Wikipedia. So to prevent any potential 'test data leakage' we tried (and reported results) without pretrained weights
from kgt5.
Thank you very much for providing the insights and clarifications! Closing the issue.
from kgt5.
Related Issues (20)
- Implementing the KGT5 pipeline for my own constructed KG HOT 9
- What is the usage of --save_prefix wd5m-1gpu ?
- KGQA data split HOT 12
- Question about KGC datasets HOT 5
- Question about training in link prediction? HOT 1
- Question about QA-fine-tuning HOT 7
- KGC data split HOT 5
- 1to1 mapping split issues HOT 4
- Question about training step HOT 3
- Questions about Link Prediction Task HOT 16
- MRR calculation HOT 4
- Verbalization procedure question
- Answering questions HOT 3
- Pretrained checkpoints for QA task
- How to get a one-to-one mapping between an entity/relationship and its text representation with mateQA HOT 1
- Tokenizer questions HOT 1
- questions of link-prediction
- Is there a checkpoint that I can use directly? HOT 1
- kgqa
- Questions about dataset HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kgt5.