yuleiniu / rva Goto Github PK

Under Extracting Features (Optional), you have this step:

Prepare the MSCOCO and Flickr images.

• Where do I find the Flickr images? Could you remind me if that is the valid and test set?
• MSCOCO images: did you download the train images from here? These are raw images, right? You didnt take any keypoints/captioning? So we have to download all the train/valid/test set?

Any preprocessing step required after that?

Thanks for your help :)

Hi, thank you for sharing you code.
In your paper, You wrote that rva model obtain one-hot vector using Eq. 19. Strictly speaking, I think it is not correct.

Rva model obtain one-hot vector one_hot(argmax(Eq. 20)) not Eq. 19.
In other words, softmax operation is always used regardless of the argument (soft / hard).

For more details, please refer to the F.gumbel_softmax code as below:
https://pytorch.org/docs/stable/_modules/torch/nn/functional.html

Follow the operation process of github， the loss will be nan at 5 epoch. I lower the learning rate, but the same situation will occur.

Hi Yulei,

Thanks for your sharing of this copy of code. I am running the code as a baseline. In the Glove embedding initialization step,

python data/init_glove.py

I think the file 'init_glove.py' is missing.

Hello, i want to ask about the image features which are extracted from the detectron. When i extract them, i find their shapes are (36,1024), but the features' shapes which are extracted by batra-mlp-lab are (36,2048). So my question is how to deal with this problem? Thanks.

In rva/visdialch/data/readers.py", line 191 there is no self.loc_feats.

Hi,

Did you also face this parallelization error while training? This seems to be strange since I am using in fact only one gpu-id!

  File " /home/shubham/rva/visdialch/encoders/modules.py", line 233, in forward
    ques_prob_refine = torch.bmm(ques_gs[:, i, :].view(-1, 1, 2), ques_prob).view(-1, 1, 2) # shape: [batch_size, num_rounds, 2]
RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:479

Any suggestions please

Hi,

I find the distribution, i.e. pi in Eq. (19) and Eq. (20) should use log before added to g. However, I have checked your codes, I find you do not use log on pi. Does it matter?