How to use the maskfeat model to imagenet dataset about videotransformer-pytorch HOT 10 CLOSED

@TinnyFlames @TinnyFlames Not straightforward, we have tried to pre-train the vit-base 100 epochs with hog prediction in ImageNet-1k under the MAE architecture. The hog targets is slightly higher than the pixel norm (82.7% vs 82.4%).

from videotransformer-pytorch.

Hi @mx-mark! Thanks for your reply. Is it possible to share suggestions on implementing the mask code? I tried to implement maskfeat for Imagenet but was confused about the mask part. The image size is (224,224,3) but the hog feature size is (14,14,108). If we randomly mask image patches, how to mask the hog features correctly since the dimension is not matched?
CubeMaskGenerator is a little bit obscure to rewrite for me.

from videotransformer-pytorch.

@mx-mark May I ask you how you calculate your loss function in the imagenet? just as the same as this?
https://github.com/mx-mark/VideoTransformer-pytorch/blob/main/video_transformer.py#L899

from videotransformer-pytorch.

@RechelTeamo right, the loss minimizes the L2 distance between the predicted and original HOG feature.

from videotransformer-pytorch.

Thanks for your answer.
I try this loss function. But I meet the problem that my loss becomes NaN. I find this problem in the blocks(x). I think the problem is lr. May I ask about your lr setting? I use blr=1e-4 and batch size 1024 in 8 V100 GPU.

from videotransformer-pytorch.

@RechelTeamo the setting for what, pre-training or fine-tuning?

from videotransformer-pytorch.

for pre-training

from videotransformer-pytorch.

@RechelTeamo There are some related problems reported in the original MAE repos. You can check if it works facebookresearch/mae#65, facebookresearch/mae#42

from videotransformer-pytorch.

@RechelTeamo For my pretraining settings, the blr is 1.5e-4 and the effective batch size is 4096.

from videotransformer-pytorch.

@mx-mark Oh thanks a lot. I set blr: 5.00e-05 and batch 1024 (actual lr=2e-4)now it's epoch 16. I will report my result when it is finished.

My blr=1e-4 batch=1024 (actual lr=4e-4) will loss NaN.

from videotransformer-pytorch.