Issue about checkpoint of CT-CLIP about ct-clip HOT 6 CLOSED

Hello @zhi-xuan-chen, thank you very much for your interest in our work. Are you using our zero-shot inference code to load checkpoints? Also, could you please check your torch version, which should be 2.0.1? It's unexpected because it works in my case. Discr blocks are not used in CT-CLIP anyway, as the encoder is from another work of ours that require discr blocks. Therefore, it would be better if I remove them from the checkpoints and the model code (as it takes some unnecessary GPU memory). I will send the PR for edited codes and update the weight files in hugging face today after ensuring they do not break anything.

from ct-clip.

Hello @zhi-xuan-chen, I have updated model weights and the transformer_maskgit code. Discr blocks should be removed now and it should work in your case as well. Please let me know if you can run the model.

from ct-clip.

OK! Only the transformer_maskgit code was changed, so I only need to update the code of this part?

from ct-clip.

I have noticed only ctvit.py was modified. I will try your new code immediately!

from ct-clip.

Yes, I only modified ctvit.py to remove weights that are not used in the forward run of CT-Clip. Let me know if you have any further inquiries on this!

from ct-clip.

The model checkpoint can be successfully loaded. Thank you very much!

from ct-clip.