Comments (3)
This post does the same as in your line of code. I also manually checked that the parameters are different in the two ways: AutoModel.from_config
and AutoModel.from_pretrained
. Could you please confirm this?
from kgt5.
Hi Shubham
Pretrained weights are not loaded when loading with config (see https://huggingface.co/docs/transformers/v4.18.0/en/model_doc/t5#transformers.T5ForConditionalGeneration). In the current code, as you pointed out, model is loaded as follows
config = T5Config().from_pretrained('t5-small')
model = T5ForConditionalGeneration(config)
This is not the same as model = T5ForConditionalGeneration.from_pretrained('t5-small')
. In our case, from_pretrained method of T5ForConditionalGeneration is not called, hence weights aren't loaded.
I confirmed this in a colab notebook here: https://colab.research.google.com/drive/1Gh5gbp6tl9P-DBHQ_ottV2rBVMB_gSRU?usp=sharing. You can first try generating from the model loaded using the config route (as is done in KGT5 code) and it generates garbage. If you switch to model = T5ForConditionalGeneration.from_pretrained('t5-small')
, you can see that it outputs meaningful generations.
Please let me know if you still have doubts on whether it is initialized with pretrained weights or not. Also, I would suggest you look into the 'apoorv-dump' branch instead of main if you want to see a more updated codebase (but its not clean). In that branch, there is a specific flag for enabling the use of pretrained LM weights. I will try and update to using AutoModel.from_config
to make it clear that weights aren't loaded, thanks for the suggestion.
from kgt5.
Thanks for the confirmation! Closing the issue.
from kgt5.
Related Issues (20)
- Implementing the KGT5 pipeline for my own constructed KG HOT 9
- What is the usage of --save_prefix wd5m-1gpu ?
- KGQA data split HOT 12
- Question about KGC datasets HOT 5
- Question about training in link prediction? HOT 1
- Question about QA-fine-tuning HOT 7
- KGC data split HOT 5
- 1to1 mapping split issues HOT 4
- Question about training step HOT 3
- Questions about Link Prediction Task HOT 16
- MRR calculation HOT 4
- Verbalization procedure question
- Answering questions HOT 3
- Pretrained checkpoints for QA task
- How to get a one-to-one mapping between an entity/relationship and its text representation with mateQA HOT 1
- Tokenizer questions HOT 1
- questions of link-prediction
- Is there a checkpoint that I can use directly? HOT 1
- kgqa
- Questions about dataset HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kgt5.