How is T5 randomly initialised and NOT using pretrained LM weights? about kgt5 HOT 3 CLOSED

This post does the same as in your line of code. I also manually checked that the parameters are different in the two ways: AutoModel.from_config and AutoModel.from_pretrained. Could you please confirm this?

from kgt5.

Hi Shubham

Pretrained weights are not loaded when loading with config (see https://huggingface.co/docs/transformers/v4.18.0/en/model_doc/t5#transformers.T5ForConditionalGeneration). In the current code, as you pointed out, model is loaded as follows

config = T5Config().from_pretrained('t5-small')
model = T5ForConditionalGeneration(config)

This is not the same as model = T5ForConditionalGeneration.from_pretrained('t5-small'). In our case, from_pretrained method of T5ForConditionalGeneration is not called, hence weights aren't loaded.

I confirmed this in a colab notebook here: https://colab.research.google.com/drive/1Gh5gbp6tl9P-DBHQ_ottV2rBVMB_gSRU?usp=sharing. You can first try generating from the model loaded using the config route (as is done in KGT5 code) and it generates garbage. If you switch to model = T5ForConditionalGeneration.from_pretrained('t5-small'), you can see that it outputs meaningful generations.

Please let me know if you still have doubts on whether it is initialized with pretrained weights or not. Also, I would suggest you look into the 'apoorv-dump' branch instead of main if you want to see a more updated codebase (but its not clean). In that branch, there is a specific flag for enabling the use of pretrained LM weights. I will try and update to using AutoModel.from_config to make it clear that weights aren't loaded, thanks for the suggestion.

from kgt5.

Thanks for the confirmation! Closing the issue.

from kgt5.