`self.mask_token` at 553 line in `vision_transformer.py` about vilt HOT 6 OPEN

Hi @hspark92,

mask_it=False is a legacy argument for the MPP (masked patch projection) objective.
I refer you to #18

from vilt.

Hi @dandelin ,

Thanks you for your reply.
It's helpful !

But i wonder why self.mask_token is trainable parameters.
if self.mask_toekn is cnostant, will it affect training ?

Thanks,

from vilt.

@hspark92

I set it as a trainable parameter following BERT's way of treating [MASK] tokens.
I'm not sure whether constant [MASK] tokens' embeddings affect the training or not.

But if an experiment on those fixed [MASK] tokens' embeddings exists in the NLP domain, then I believe ViLT's MPP would follow the trend.

Thanks.

from vilt.

I get it, thank you for answer !

from vilt.

Hi, I find the same problem as yours. The 'self.mask_token' has not been defined. So I can't run the pre-train code. How do you solve it? Thanks!

from vilt.

Hi, i'm really impressed with your work and i'm getting a lot of help !

But, there is an error in vilt/modules/vision_transformer.py when mask_it == True. The error occur because self.mask_token at 553 line in vision_transformer.py is not initilized. So, i wonder what self.mask_token means?

Thanks.

Hi, I find the same problem as yours. The 'self.mask_token' has not been defined. So I can't run the pre-train code. How do you solve it? Thanks!

from vilt.