Hi, thanks for sharing the code of your interesting work.

I saw run_pretrain.py in your repo; however, there is a missing config file ./configs/Pretrain.yaml' and script for running the pretraining.
Since the pretraining strategy is one of your novel contributions, would you have planned to release the complete code for reproducing the pretraining?

Hi, thanks for sharing the code of your interesting work.

1. I want to reproduce the phrase groundinig task,
   So when I tried running the following command on the flicker dataset, I encountered the following error.
   The flicker json file does not have keys such as tokens_positive or not_crop_bbox_list. How can I resolve this issue?

python -m torch.distributed.launch --nproc_per_node=8 --master_port=12451 --use_env run_grounding_train.py --train 1 --pretrain 0 --test_dataset flickr --config ./configs/visual_grounding.yaml --output_dir ./output/phrase_grounding --checkpoint grounding.pth --eval_step 500

2. in flicker.json

file_name": "flickr30k_images/flickr30k_images/1000092795.jpg", "text_type": "caption", "height": 500, "width": 333, "pseudo_caption": "Two young guys with shaggy hair look at their hands @@ [pos_242] [pos_188] [pos_302] [pos_229]

while hanging out in the yard .", "normal_caption": "Two young guys with shaggy hair look at their hands while hanging out in the yard .", "bbox": [158.0, 184.0, 40.0, 41.0], "bbox_list": [[158.0, 184.0, 40.0, 41.0]]},

What is the meaning of '@@ [pos_242][pos_188][pos_302][pos_229]? If I want to fine-tune on my custom dataset, I need to create a JSON file that follows the same input format, right?

3. In Refcoco.json, what is the meaning of not_crop_bbox_list, positive token, negative token? If I want to fine-tune on my custom dataset, I need to create a JSON file that follows the same input format, right?

Thank you so much!

Thank author for prviding clear code.I wonder how to get the tokens positive and tokens negative?

Hi, can you share the codebase of finetuning for VCR task with pure ALBEF model? Thanks.

Hi, i cannot find these files in this repo:
train_vcr_file: ['./2022_01_06_vcr_pretrain_prompt_data.json']
train_val_vcr_q2a_file: ['./pevl_vcr_itm_finetune_val_QA_data.json']
train_val_vcr_qa2r_file: ['./pevl_vcr_itm_finetune_val_QAR_data.json']

and could you offer the command to train vcr task

Hi, I have noticed when finetuning for VCR task, both ita and itm are used. The itm loss is computed by the cross entropy between the itm prediction and label. But for the ita loss, I think you directly regard the question and negative answer as a positive pair? Since you initialize sim_targets = torch.zeros(sim_i2t_m.size()).to(image.device)and
sim_targets.fill_diagonal_(1)
Is this reasonable?

Hi,

Thank you for the amazing work and code! Will you plan to provide the code and checkpoints for VQA task?

Best,
Ziyan

It is currently very slow to download all of the checkpoints, the speed is only ~1KB/s (I am based in the US). Can the authors upload the checkpoints to any other sites? Or is there anyone who has downloaded them upload them somewhere else?

Thanks!

Hi, could you share the checkpoint file for second-stage pretrained VCR?

Hi，i find the download link of finetuned VCR id failed when i tried to use it

i see there are different pre-trained model for each downstream tasks, what is the difference?

Hi Authors,

Thank you for making the code for this awesome piece of work available. I wanted to know as hsown in L180 what does this double loop do in this sequence token ?

Thanks in Advance

Hi AUthors,

Thanks for making code available for this awesome work. My question is like Pix2seq did for object detection , can this work can be applied to dense prediction task like : Semantic Segmentation etc ?

hi, i notice that when finetuning ssp model on vcr task, the performance drop a lot at each 5000 steps in the first epoch.
before finuetuning, the result for Q2A and QA2R are both more than 74%
step 5000: 67% and 66%
step 10000: 64.7% and 64.9%
step 15000: 61.3% and 60.9%
step 20000: 60.8% and 57.9%
and the other results have no been gained. still training.

is it true? is the model trained in a right way? have you notice this phenomenon when you finetune it?
of course, i have no V100 so i train the model in 4 2080ti. with the limitation of memory, i set batch size=1, test_batch_size=4, gradient_accumulation_steps=8. the other config is the same as your vcr.yaml.

looking for you help, thanks!

Hello,

Thank you for the wonderful work.

I was trying to reproduce the results shown in paper for VRD.

Currently I am getting around the following scores:
R@20: 0.5417 R@50: 0.6160 R@100: 0.6350
mR@20: 0.1220 mR@50: 0.1606 mR@100: 0.1723

I finetuned the pretrained checkpoint for 10 epochs with 8 v100 GPUs as instructed with a batch size of 8 and 32.

I also separately finetuned the pretrained checkpoint for 10 epochs with 10 100a GPUs with a batch size of 100.

Would you be kind of enough to provide some reasoning behind these results?

Thank you.

when i run run_vcr_train.py on four 2080ti, i got:

 File "run_vcr_train.py", line 253, in main                                                                                                                                                                                                             
    config, args.training_mode, args, vcr_val_q2a_loader, vcr_val_qa2r_loader)                                                                                                                                                                           
  File "run_vcr_train.py", line 109, in train                                                                                                                                                                                                            
    loss_ita, loss_itm = model(images, text, alpha, itm_labels, mode='finetuning')                                                                                                                                                                       
  File "/anaconda3/envs/pevl/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl                                                                                                                           
    result = self.forward(*input, **kwargs)                                                                                                                                                                                                              
  File "/anaconda3/envs/pevl/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 692, in forward                                                                                                                        
    if self.reducer._rebuild_buckets():                                        
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one.  This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing t
he keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).

in my opinion, the parameters of the momentum model are not used in producing loss. how to deal with it?

and the difference between the code of finetune part and pretrain part on vcr task is only deletion of MLM and soft loss. is it true?

looking for your help, thanks!