Many thanks for the good work. I am curious about how the background thresholds for PASCAL VOC and PASCAL context were determined?

Hi,

Can you please provide more details as how you do the non-parametric grouping on CLIP's features (obtained from ViT encoder)?

Hi！Very good work. I have some questions. Why not consider aligning the 8 segment tokens with the generated text? would this be better

Hi! I realized that the commands for downloading GCC 3M and 12M have a couple of typos. The corrected version for the 12M is below:

sed -i '1s/^/url\tcaption\n/' gcc12m.tsv
img2dataset --url_list gcc12m.tsv --input_format "tsv" \
            --url_col "url" --caption_col "caption" --output_format webdataset\
            --output_folder local_data/gcc12m_shards \
            --processes_count 16 --thread_count 64 \
            --image_size 512 --resize_mode keep_ratio --resize_only_if_bigger True \
            --enable_wandb True --save_metadata False --oom_shard_count 6

It would be nice if you can update the README.me with it.
Great work! Thank for sharing it.

Hi,

I was training the network with validation on coco dataset using 8 gpus on a single node. It seems like the network hangs while using reduce_tensor method inside validate_cls(). Is there a solution known for this issue?

python -m yfcc100m.convert_metadata . -o yfcc100m_meta --skip_verification

gives the following error

convert_metadata.py: error: unrecognized arguments: --skip_verification

Thanks for your great works!

when I evaluate your pretrained model on VOC and Pascal Context datasets, I could not get consistent results as you provided in the paper. I am not sure what are the issues preventing me to get your reported in the paper, so I hope you can help me out of this. thanks!

The results I got from your pretrained model group_vit_gcc_yfcc_30e-879422e0.pth on VOC 2012 is:
------------+-------+-------+
| Class | IoU | Acc |
+------------+-------+-------+
| background | 80.53 | 95.76 |
| aeroplane | 40.76 | 51.58 |
| bicycle | 36.07 | 70.07 |
| bird | 54.19 | 56.7 |
| boat | 36.01 | 51.18 |
| bottle | 32.97 | 36.98 |
| bus | 31.66 | 33.96 |
| car | 51.58 | 55.66 |
| cat | 76.48 | 84.09 |
| chair | 14.19 | 17.46 |
| cow | 63.01 | 67.99 |
| table | 21.48 | 26.53 |
| dog | 70.72 | 76.04 |
| horse | 70.28 | 75.15 |
| motorbike | 40.7 | 50.26 |
| person | 18.89 | 19.19 |
| plant | 28.56 | 33.04 |
| sheep | 55.39 | 60.27 |
| sofa | 28.03 | 34.41 |
| train | 23.3 | 28.74 |
| monitor | 24.64 | 38.76 |
+------------+-------+-------+
Summary:

+-------+-------+-------+
| aAcc | mIoU | mAcc |
+-------+-------+-------+
| 82.38 | 42.83 | 50.66 |
+-------+-------+-------+

The results for Pascal Context is:

+------------+-------+-------+
| Class | IoU | Acc |
+------------+-------+-------+
| background | 13.48 | 89.03 |
| airplane | 43.22 | 51.06 |
| bag | 5.67 | 7.07 |
| bed | 11.9 | 13.89 |
| bedclothes | 0.0 | 0.0 |
| bench | 8.74 | 17.63 |
| bicycle | 48.39 | 59.64 |
| bird | 52.0 | 57.12 |
| boat | 34.42 | 42.73 |
| book | 1.3 | 1.36 |
| bottle | 37.57 | 42.64 |
| building | 1.0 | 1.0 |
| bus | 34.33 | 37.25 |
| cabinet | 6.01 | 8.21 |
| car | 42.33 | 45.41 |
| cat | 70.61 | 81.02 |
| ceiling | 3.34 | 3.4 |
| chair | 14.0 | 15.47 |
| cloth | 2.8 | 3.2 |
| computer | 5.81 | 18.75 |
| cow | 58.19 | 67.05 |
| cup | 2.35 | 2.87 |
| curtain | 1.63 | 1.7 |
| dog | 63.04 | 69.54 |
| door | 0.0 | 0.0 |
| fence | 1.35 | 1.39 |
| floor | 0.86 | 0.87 |
| flower | 3.96 | 4.92 |
| food | 13.98 | 18.65 |
| grass | 0.34 | 0.34 |
| ground | 0.21 | 0.21 |
| horse | 61.52 | 69.12 |
| keyboard | 4.97 | 5.65 |
| light | 1.24 | 1.5 |
| motorbike | 40.99 | 48.74 |
| mountain | 9.55 | 10.09 |
| mouse | 0.0 | 0.0 |
| person | 16.52 | 16.85 |
| plate | 7.14 | 12.29 |
| platform | 2.16 | 3.59 |
| plant | 16.85 | 17.64 |
| road | 1.09 | 1.12 |
| rock | 6.24 | 6.38 |
| sheep | 48.16 | 55.33 |
| shelves | 2.7 | 2.85 |
| sidewalk | 0.65 | 0.69 |
| sign | 2.84 | 2.97 |
| sky | 1.15 | 1.15 |
| snow | 6.41 | 7.25 |
| sofa | 20.15 | 23.2 |
| table | 13.88 | 18.24 |
| track | 7.75 | 8.52 |
| train | 26.09 | 30.97 |
| tree | 1.36 | 1.36 |
| truck | 1.54 | 2.65 |
| monitor | 14.22 | 15.41 |
| wall | 0.02 | 0.02 |
| water | 3.47 | 3.5 |
| window | 2.31 | 2.4 |
| wood | 0.47 | 0.52 |
+------------+-------+-------+
Summary:

+-------+-------+-------+
| aAcc | mIoU | mAcc |
+-------+-------+-------+
| 24.44 | 15.07 | 18.89 |
+-------+-------+-------+

Thank you for sharing such a nice work.
I have some questions.
(1) What is the meaning of self.pre_assign_attn? Has it been described in the paper?

(2) Does self.assign represent equation (3), (4), and part of (5) in the paper?

Dear authors,
Thanks for sharing this nice work.
Are the training and testing in classification task in hard assignment style, while the inference in segmentation in soft assignment?

Hi, thank you for your great work.

I noticed that during the generation of segmentation masks, soft assignment matrices are used instead of hard assigment matrices (from segmentation/evaluation/group_vit_seg.py, line 166). Although the product of the soft assignment metrices is converted to one-hot before classifying pixels, it is somewhat different from your paper, which suggests that we should directly multiply hard assigment matrices.

In fact, by changing the code in line 166 from attn_masks = attn_dict['soft'] to attn_masks = attn_dict['hard'], the demo yields worse segmentation result.

Am I misunderstanding the code or missing some implementation details from your paper?

Hi,

Thank you for your great work. However, it seems the pre-training script dist_train.sh is missing. Can you plz have a check? TY.

Hi, Thank you for your great work.
I'm just wondering that the 2 stage only outputs 8 group tokens, does it mean that there are at most 8 kinds of objects to be segmented? What to do if there are more than 8 objects in an image, or if the number of object in an image are uncertain?

https://github.com/NVlabs/GroupViT#gcc3m

sed -i '1s/^/caption\turl\n/' gcc3m.tsv
img2dataset --url_list gcc3m.tsv --input_format "tsv" \
            --url_col "url" --caption_col "caption" --output_format webdataset\
            --output_folder local_data/gcc3m_shards
            --processes_count 16 --thread_count 64
            --image_size 512 --resize_mode keep_ratio --resize_only_if_bigger True \
            --enable_wandb True --save_metadata False --oom_shard_count 6
rename -d 's/^/gcc-train-/' local_data/gcc3m_shards/*

1. After checking the documentation of img2dataset, I find that there is no argument --save_metadata for img2dataset and this may cause an error.

Hi! Thank you for the great work and neat implementation.

In section C.4 of the paper you reported results from testing GroupViT pretrained model on COCO dataset, which were quit impressive but not as good as the ones on PASCAL VOC. This is probably due to the domain shift between PASCAL VOC and COCO images, classes/text descriptions.

I was wondering if it's possible to fine-tune GroupViT on COCO dataset (and out-of-domain datasets in general) by freezing the model's weights and training only the learnable group tokens on the new datasets in a few-shot manner.

If it's the case, I'm ready to implement this with you're guidance.

Hi, Thanks for sharing the great work!

In my initial guess, I think stage 2 groups the results from stage 1. And it can form a hierarchical grouping pipeline.
However, when I reading the source code, it seems that stage 2 just groups the image patches.
Is my understanding right? And why not just grouping the results from stage 1?

Thanks for your reading!

Dear Author,
Thanks for your work! I have some questions about the code:
(1）Dataset splitting
Assume we have one custom dataset called Seg_Caption Dataset which has segmentation mask annotation and caption annotation. Can we split the dataset based on the following way:
Specifically, Seg_Caption_train just includes the input image and caption, the Seg_Caption_val includes the input image, caption, and segmentation mask. The segmentation mask is used to compare the model's segmentation prediction and the ground truth segmentation mask. And we just enable the seg task during the evaluation.

train:
- Seg_Caption_train
val:
- Seg_Caption_val
-
evaluate:
task:
- seg

(2) label's definition in the code
labels = torch.arange(batch_size, dtype=torch.long, device=image_x.device) + batch_size * dist.get_rank()
loss_img = self.cross_entropy(logits_per_img * logit_scale, labels)
loss_text = self.cross_entropy(logits_per_text * logit_scale, labels)
Does this label make sense? It's hard to understand what such a loss wants to do.

(3) WebDataset format
Following this video tutorial: https://www.youtube.com/watch?v=v_PacO-3OGQ
I try to use the "tar --sort=name -cf ../dataset.tar ." to convert my dataset into the required format.
but the training will stop after the follwing procedure:
(main_group_vit.py 284): INFO Train: [0/30][0/750] eta 0:15:46 lr 0.000000 time 1.2622 (1.2622) total_loss 2.3884 (2.3884) loss 0.7358 (0.7358) multi_label_loss 1.6526 (1.6526) grad_norm nan (nan) mem 4184MB
Do you know why this will happen?

Thanks for your help!
Mengya Xu

Hi, thanks for the amazing work! I have two questions about the paper. In my understanding, the resolution of the mask is restricted by the number of patches. In the paper, it says that during testing the shorter side is resized to 448. With a patch size 16, the mask resolution is 28 for the shorter side.

1. Many of the visualization results in the paper show very smooth masks. Some have very high frequency noise (very small isolated dots in a mask). I do some rough calculation, it does not seem to have a resolution of only 28. Do you use an even larger image size for the visualization results? Or I am missing something?
2. If I understand correctly, you are using 224x224 images for training, but use 448 directly for testing without any further modification to the model. There should be a data distribution mismatch between training and testing, right? Do you have any sense about how this affects the results? For example, maybe testing on 448x448 imagenet images to see how the accuracy changes?

Thank you!
Jiahao

Hi, Thank you for your great work.
I am confused by the visualization results. In my opinion, the final assignment is image patch (pixels in same patch have same class/group).
The setting for training is an image resolution of 224x224 and a patch size of 16x16.
I assume there should have a clear grid effect in the final visualization.
But in your paper, visualization results have perfect curve.

Best.

Thanks a lot for your contribution!

Do you have IoU for each category for PASCAL VOC dataset? This would help understand how your model performs for each category and sometimes helps to interpret some results.
Also, are background class masks obtained with query "background"?

Dear Authors,

I tried training the GroupViT model on GCC + YFCC datasets using the group_vit_gcc_yfcc_30e.yml config file and with a batch size of 2048 (256x8). The results on PASCAL VOC after 30 epochs of training is roughly 5% mIoU lower (absolute) than what is reported in the paper. Additionally, I tried training with gradient accumulation with 2 steps, which did not give any improvement. Do you have any suggestions on what can cause the lower performance?

Thank you.

Thanks for your code sharing!

Normalization Step above (copied from Group-ViT arxiv pre-print) seems not be applied in the code, is that true? And I found that the normalization may make training unstable (not in ViT but other backbone), did you encounter the same problem?

Hi, @xvjiarui @shalinidemello
Thank you for presenting this good work.

I have some questions about the group tokens.
It seems that group_token is set as None for initialization in line 839 ./models/group_vit.py.
So how to make the group_token learnable? Why do not set it as a learnable parameters?

What's more, how do we get the segmentation mask through the model? It seems that the "class GroupViT" of ./models/group_vit.py only return the image features?

These make me very confused.
Could you please help me to solve my confusion?

Hello, thank you very much for your work. I found that the data processing in the code is for public data sets, and now I have some image-text pairs of data sets, and I want to make some fine-tuning on the pre-training model you published, so I would like to ask you how to deal with custom data sets. Looking forward to your reply.

The current command for installing apex
cd && apex && pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

Needs to be replaced by
cd && apex && pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

Could you help me solve this problem? libtorch_cuda_cu.so has already existed in the directory: "/home/user/anaconda3/envs/groupvit/lib/python3.7/site-packages/torch/lib"

The question is that how to generate so many prompt templates? Is the inspiration from other work?

GroupViT/datasets/imagenet_template.py

Line 27 in b4ef51b

full_imagenet_templates = [

Hi guys,

Do you have any plans on releasing the code for evaluating the mask quality (mask probing) mentioned in appendix C.2. table C.2.??
It would be really helpful. Thank you!.

Regards

Hi, thank you for your great work!
I'm just wondering how to choose the number of learnable group tokens in each stage? Is it a random number at the first stage, such as 64, and then decreases as the grouping stage goes deeper?

Thank you very much!

RedCaps12m has "raw_caption" and "caption" in its annotations. Can I know which one do you use?

Hello!
it's a great work. I have a question about zero-shot segmentation.
Are we just change cfg: segmentation/configs/base/datasets/pascal_voc12.py to "cfg: segmentation/configs/base/datasets/coco.py" for coco segmentation?

Just curious, why do you use WebDataset instead of Torch ImageDataset?

Hi, thanks for your great work.
In the process of training, I found that the memory usage gradually increased until it was out of memory. There are 252G memory in my server. The training dataset is gcc12m, the val dataset is VOC2012.
The Traceback message:
RuntimeError:DataLoader worker(pid xxxxx) is killed by signal:killed.
Could you please help me solve it?

Hi, thanks for your contribution!

I've re-run the model twice with GCC+RedCaps. For the first time, I got 51.9 on ImageNet and 11.6 on COCO, while for the second time, I got 54.3 on ImageNet and 25.8 on COCO. It seems that the results are not stable.

Is this a normal phenomenon for this model?

Hi guys,
Thanks again for such a nice work. I want to ask you why is this assertion on the image size necessary? In case it is, any ideas on how to fix it? Thank you!

Hi @xvjiarui, thanks for your great work! I'm confused about how did you generate the smooth object masks based on the grouping allocation among 16 * 16 patches per image. Why the masks are not aligned with the 16 * 16 grids?

The command is
(node0)$ ./tools/dist_mn_launch.sh main_group_vit.py configs/group_vit_gcc_yfcc_30e.yml 0 2 8 tcp://node0
(node1)$ ./tools/dist_mn_launch.sh main_group_vit.py configs/group_vit_gcc_yfcc_30e.yml 1 2 8 tcp://node0
I want to ask how can I do with the tcp://node0 ??? what is it?
Thank you very much.

Hi, I have trained GroupVIT under your default settings with RedCAP+GCC3M+GCC12M, and I ran the experiments 2 times, only achieving 26.8% mIoU and 27.9% mIoU. According to the previous discussions in the Closed Issue, I found that it may be caused by the training data size. So would you plz tell me the specific data size of RedCap, GCC3M, GCC12M, and YFC100M?

Hello, could you please tell me how can I get image embedding with size of 512?
By testing demo, I can just obtain image embedding with size 256.

Hello, when I run the code .
When it arrives the line 220 of main_group_vit.py.
for idx, samples in enumerate(data_loader):
it will show the information:

/data/miniconda3/envs/groupvit/lib/python3.7/site-packages/torchvision/transforms/functional.py:365: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.
"Argument interpolation should be of type InterpolationMode instead of int. "
/group/20016/zhangyangqi/debug/GroupViT/datasets/builder.py:105: UserWarning: LookupError("\n**********************************************************************\n Resource \x1b[93mpunkt\x1b[0m not found.\n Please use the NLTK Downloader to obtain the resource:\n\n \x1b[31m>>> import nltk\n >>> nltk.download('punkt')\n \x1b[0m\n For more information see: https://www.nltk.org/data.html\n\n Attempted to load \x1b[93mtokenizers/punkt/PY3/english.pickle\x1b[0m\n\n Searched in:\n - '/usr/local/app/nltk_data'\n - '/data/miniconda3/envs/groupvit/nltk_data'\n - '/data/miniconda3/envs/groupvit/share/nltk_data'\n - '/data/miniconda3/envs/groupvit/lib/nltk_data'\n - '/usr/share/nltk_data'\n - '/usr/local/share/nltk_data'\n - '/usr/lib/nltk_data'\n - '/usr/local/lib/nltk_data'\n - ''\n**********************************************************************\n")
warnings.warn(repr(exn))
/data/miniconda3/envs/groupvit/lib/python3.7/site-packages/torchvision/transforms/functional.py:365: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.
"Argument interpolation should be of type InterpolationMode instead of int. "
/group/20016/zhangyangqi/debug/GroupViT/datasets/builder.py:105: UserWarning: LookupError("\n**********************************************************************\n Resource \x1b[93mpunkt\x1b[0m not found.\n Please use the NLTK Downloader to obtain the resource:\n\n \x1b[31m>>> import nltk\n >>> nltk.download('punkt')\n \x1b[0m\n For more information see: https://www.nltk.org/data.html\n\n Attempted to load \x1b[93mtokenizers/punkt/PY3/english.pickle\x1b[0m\n\n Searched in:\n - '/usr/local/app/nltk_data'\n - '/data/miniconda3/envs/groupvit/nltk_data'\n - '/data/miniconda3/envs/groupvit/share/nltk_data'\n - '/data/miniconda3/envs/groupvit/lib/nltk_data'\n - '/usr/share/nltk_data'\n - '/usr/local/share/nltk_data'\n - '/usr/lib/nltk_data'\n - '/usr/local/lib/nltk_data'\n - ''\n**********************************************************************\n")
warnings.warn(repr(exn))

I do not know what it is and what should I do next?

sed -i '1s/^/caption\turl\n/' gcc3m.tsv
How to write this command under the Windows system

Hello, can the model be trained on a single GPU

Hi @xvjiarui, Thanks for your great work on GroupViT!

I notice that in your implementation of the grouping blocks here, L280-L281 and L285 are not mentioned in the paper, especially the pre-attention part.

May I ask that are these designs crucial to equip the model with grouping abilities? I've failed to reproduce your results using the information in the paper only, i.e. implementing the grouping block as simply as using a cross-attention (q: grouping tokens, kv: visual tokens) block with Gumbel Softmax and hard assignment strategy.

Hi,

Thank you for your work :)
I was curious about your work and noticed an ablation study on grouping stages(in Table 3). I was hoping if you could provide me with the checkpoints for the model trained with 1 grouping stage only as well. I wanted to finetune the models of a smaller dataset and evaluate the performance :)

Best
Ayushi

Hi, Thanks for the great work.
It seems that gumbel-softmax is not used in the hard grouping setting, will it be better to switch it on?

1. Please check default.yml, if we should add the node of test just like train and val nodes.
   
2. default.yml also did not provide the setting of pascal voc dataset and coco dataset
   

Hi, Thanks for the great work.
Why do you have to use onehot or gumbel softmax. why don't you use softmax?

NaN in loss started to appear in the 2nd epoch. Is this to be expected?

Hi, for the model that achieves 52.3% mIoU on PASCAL VOC, the paper says that GCC12M + YFCC14M are used for training. Whereas, in the config files of this repository GCC3M is also used. Which one is correct?

In case if GCC3M is also used, are the pre-trainings for fully supervised transfer in Table 5. done using GCC3M as well?

Thank you in advance.

Hi!
Currently, I'm working with your code and have been able to train in multiple GPUs without major issues. However, I had to disable the testing on PASCAL VOC because it takes forever and the code gets stuck. the last prints that I can see are the next ones:

All checkpoints founded in output/group_vit_gcc_3m_bs712x4: []
All checkpoints founded in output/group_vit_gcc_3m_bs712x4: []
All checkpoints founded in output/group_vit_gcc_3m_bs712x4: []
All checkpoints founded in output/group_vit_gcc_3m_bs712x4: []
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0


[                           ] 20/1449, 0.0 task/s, elapsed: 9609s, ETA: 686561s(base)

And

[2022-04-10 10:10:18 group_vit_gcc_3m_bs712x4](main_group_vit.py 294): INFO EPOCH 0 training takes 0:27:05
[2022-04-10 10:10:19 group_vit_gcc_3m_bs712x4](checkpoint.py 92): INFO output/group_vit_gcc_3m_bs712x4/ckpt_epoch_0.pth saving......
[2022-04-10 10:10:20 group_vit_gcc_3m_bs712x4](checkpoint.py 95): INFO output/group_vit_gcc_3m_bs712x4/ckpt_epoch_0.pth saved !!!
[2022-04-10 10:10:20 group_vit_gcc_3m_bs712x4](main_group_vit.py 160): INFO Avg loss of the network on the 2876225 train images: 15.16
[2022-04-10 10:10:20 group_vit_gcc_3m_bs712x4](main_group_vit.py 317): INFO Building zero shot classifier
[2022-04-10 10:10:25 group_vit_gcc_3m_bs712x4](main_group_vit.py 321): INFO Zero shot classifier built
[2022-04-10 10:10:39 group_vit_gcc_3m_bs712x4](main_group_vit.py 347): INFO Test: [0/17]        Time 19.063 (19.063)    Loss 6.8310 (6.8310)    Acc@1 1.545 (1.545)     Acc@5 6.180 (6.180)     Mem 71248MB
[2022-04-10 10:12:43 group_vit_gcc_3m_bs712x4](main_group_vit.py 347): INFO Test: [10/17]       Time 13.185 (13.037)    Loss 6.8311 (6.8282)    Acc@1 1.990 (1.904)     Acc@5 6.739 (6.734)     Mem 71248MB
[2022-04-10 10:13:53 group_vit_gcc_3m_bs712x4](main_group_vit.py 353): INFO Clearing zero shot classifier
[2022-04-10 10:13:53 group_vit_gcc_3m_bs712x4](main_group_vit.py 355): INFO  * Acc@1 1.903 Acc@5 6.735
[2022-04-10 10:13:53 group_vit_gcc_3m_bs712x4](main_group_vit.py 166): INFO Accuracy of the network on the 50000 test images: 1.9%
[2022-04-10 10:13:53 group_vit_gcc_3m_bs712x4](checkpoint.py 92): INFO output/group_vit_gcc_3m_bs712x4/ckpt_epoch_0_best_acc1.pth saving......
[2022-04-10 10:13:55 group_vit_gcc_3m_bs712x4](checkpoint.py 95): INFO output/group_vit_gcc_3m_bs712x4/ckpt_epoch_0_best_acc1.pth saved !!!
[2022-04-10 10:13:55 group_vit_gcc_3m_bs712x4](main_group_vit.py 173): INFO Max accuracy: 1.90%
[2022-04-10 10:13:55 group_vit_gcc_3m_bs712x4](group_vit_seg.py 142): INFO Building GroupViTSegInference with 21 classes, test_cfg=Config (path: None): {'bg_thresh': 0.95, 'mode': 'slide', 'stride': (224, 224), 'crop_size': (448, 448)}, with_bg=True

Do you have any clue about what is going on? Thanks for your help!