liewfeng / imted Goto Github PK
View Code? Open in Web Editor NEW[ICCV 2023] Integrally Migrating Pre-trained Transformer Encoder-decoders for Visual Object Detection
Home Page: https://arxiv.org/abs/2205.09613
License: Apache License 2.0
[ICCV 2023] Integrally Migrating Pre-trained Transformer Encoder-decoders for Visual Object Detection
Home Page: https://arxiv.org/abs/2205.09613
License: Apache License 2.0
Thank you for your work,i would llike to know the environment of your project, such as the version of mmcv, it seems can't work with high version.
Hello, I would like to consult. I put MAE pre-training weights, but it still gives an error. Could you help me explain it
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/mmcv/utils/registry.py", line 52, in build_from_cfg
return obj_cls(**args)
File "/dfs/data/imTED/mmdet/models/detectors/imted.py", line 20, in init
super(imTED, self).init(
File "/dfs/data/imTED/mmdet/models/detectors/two_stage.py", line 48, in init
self.init_weights(pretrained=pretrained)
File "/dfs/data/imTED/mmdet/models/detectors/two_stage.py", line 68, in init_weights
self.backbone.init_weights(pretrained=pretrained)
File "/dfs/data/imTED/mmdet/models/backbones/vision_transformer.py", line 132, in init_weights
raise ValueError(f"checkpoint path {pretrained} is invalid")
ValueError: checkpoint path pre/Weights/ is invalid
Hello, I see that you only tested the effect on coco dataset, have you experimented on PASCAL VOC? Or can you provide the weights you trained on the coco dataset? thank you
Hello, I'm sorry to bother you again. I find that your paper has no visual inspection effect. Have you done anything about that? Or is there a demo
Thank you for your contribution, did you conduct the few shot study on voc0712 dataset?
Hi, very thanks for releasing the codes for imTED.
I am very interested in the comparison between the ViT-based Faster R-CNN and the imTED, however, I did not find the codes for the baseline. Could you kindly provide the codes related to the baseline e.g., the model and the training? Thanks for your time and consideration!
I am trying to migrate this project (ImTed) to mmdetection3.x. However, I have some doubts about how to send roi features after the Multi-scale Feature Modulator to mae decoder.
I found that the bbox_feats' shape are like torch.Size([512, 384, 7, 7]),but the self.bbox_head, i.e., the mae decoder requires inputs like x.shape = [b_s, dim, W, H], how to solve this?
def _bbox_forward(self, x, rois):
"""Box head forward function used in both training and testing."""
# TODO: a more flexible way to decide which feature maps to use
if self.with_mfm:
ss_bbox_feats = self.ss_bbox_roi_extractor(
[x[-1]], rois)
x = [self.mfm_fc(x[i]) for i in range(self.ms_bbox_roi_extractor.num_inputs)]
ms_bbox_feats = self.ms_bbox_roi_extractor(
x[:self.ms_bbox_roi_extractor.num_inputs], rois) # multi scale
factor = self.mfm_factor.reshape(1, -1, 1, 1).expand_as(ms_bbox_feats)
bbox_feats = ss_bbox_feats + ms_bbox_feats * factor
else:
bbox_feats = self.bbox_roi_extractor(
x[:self.bbox_roi_extractor.num_inputs], rois)
if self.with_shared_head:
bbox_feats = self.shared_head(bbox_feats)
cls_score, bbox_pred = self.bbox_head(bbox_feats)
Hi,
On how many images did you evaluate for novel classes for few-shot object detection on the MS-COCO benchmark (60 base classes and 20 novel classes)?
And how did you obtain that from this link : http://dl.yf.io/fs-det/datasets/cocosplit/
Best
Excuse me, but I got the following error while training with imted_faster_rcnn_vit_base_2x_finetuning_10shot_coco.py. Can you help explain it? thank you
@LiewFeng
May I ask how many versions of 'mmcv' need to be downloaded? The prompt for me to download 2.0.0 is that I cannot use 'Config' and 'DictAction', as well as' mmcv. parallel 'and' mmcv. runner '
Hello, excuse me, do you have a config file that does not use the decoder with MAE pre-trained weights, but uses the decoder with randomly initialized parameters as the detection header
Hello, I am trying to reproduce your work however I can't find the weights for the ViT-S model. The weights for ViT-B and ViT-L are available but not for ViT-S.
Would it be possible to provide a link to download the ViT-S weights?
Thanks
Hello, is there any difference between the configs files of 10shot and 30shot? Or I want to test the effect in 20shot. What should I change
Checklist
Describe the bug
Apparently, the LayerDecayOptimizerConstructorBackboneFronzen is not in the optimizer builder registry. Where can I find it?
Reproduction
I use the following command
python tools/train.py configs/imted/few_shot/imted_faster_rcnn_vit_base_2x_finetuning_10shot_coco.py
Error traceback
KeyError: 'LayerDecayOptimizerConstructorBackboneFronzen is not in the optimizer builder registry'
I tried to search for the LayerDecayOptimizerConstructorBackboneFronzen in the code but I don't find it. Please tell me where can I find the code associated with this layer.
Thanks.
data = dict( samples_per_gpu=2, workers_per_gpu=2, train=dict( pipeline=train_pipeline, classes = classes, ann_file='path/of/annotations/json'), val=dict( classes = classes, ann_file='path/of/annotations/json'), test=dict( classes = classes, ann_file='path/of/annotations/json'))
Hello, I would like to ask how JSON needs to be configured,I changed to full_ Box_ 10shot_ Frisbee_ The error has been reported continuously since trainval.json, as follows:
`2023-10-29 15:52:05,608 - mmdet - INFO - load model from: pre/mae_pretrain_vit_base_full.pth
2023-10-29 15:52:05,954 - mmdet - WARNING - The model and loaded state dict do not match exactly
unexpected key in source state_dict: mask_token, decoder_pos_embed, norm.weight, norm.bias, decoder_embed.weight, decoder_embed.bias, decoder_blocks.0.norm1.weight, decoder_blocks.0.norm1.bias, decoder_blocks.0.attn.qkv.weight, decoder_blocks.0.attn.qkv.bias, decoder_blocks.0.attn.proj.weight, decoder_blocks.0.attn.proj.bias, decoder_blocks.0.norm2.weight, decoder_blocks.0.norm2.bias, decoder_blocks.0.mlp.fc1.weight, decoder_blocks.0.mlp.fc1.bias, decoder_blocks.0.mlp.fc2.weight, decoder_blocks.0.mlp.fc2.bias, decoder_blocks.1.norm1.weight, decoder_blocks.1.norm1.bias, decoder_blocks.1.attn.qkv.weight, decoder_blocks.1.attn.qkv.bias, decoder_blocks.1.attn.proj.weight, decoder_blocks.1.attn.proj.bias, decoder_blocks.1.norm2.weight, decoder_blocks.1.norm2.bias, decoder_blocks.1.mlp.fc1.weight, decoder_blocks.1.mlp.fc1.bias, decoder_blocks.1.mlp.fc2.weight, decoder_blocks.1.mlp.fc2.bias, decoder_blocks.2.norm1.weight, decoder_blocks.2.norm1.bias, decoder_blocks.2.attn.qkv.weight, decoder_blocks.2.attn.qkv.bias, decoder_blocks.2.attn.proj.weight, decoder_blocks.2.attn.proj.bias, decoder_blocks.2.norm2.weight, decoder_blocks.2.norm2.bias, decoder_blocks.2.mlp.fc1.weight, decoder_blocks.2.mlp.fc1.bias, decoder_blocks.2.mlp.fc2.weight, decoder_blocks.2.mlp.fc2.bias, decoder_blocks.3.norm1.weight, decoder_blocks.3.norm1.bias, decoder_blocks.3.attn.qkv.weight, decoder_blocks.3.attn.qkv.bias, decoder_blocks.3.attn.proj.weight, decoder_blocks.3.attn.proj.bias, decoder_blocks.3.norm2.weight, decoder_blocks.3.norm2.bias, decoder_blocks.3.mlp.fc1.weight, decoder_blocks.3.mlp.fc1.bias, decoder_blocks.3.mlp.fc2.weight, decoder_blocks.3.mlp.fc2.bias, decoder_blocks.4.norm1.weight, decoder_blocks.4.norm1.bias, decoder_blocks.4.attn.qkv.weight, decoder_blocks.4.attn.qkv.bias, decoder_blocks.4.attn.proj.weight, decoder_blocks.4.attn.proj.bias, decoder_blocks.4.norm2.weight, decoder_blocks.4.norm2.bias, decoder_blocks.4.mlp.fc1.weight, decoder_blocks.4.mlp.fc1.bias, decoder_blocks.4.mlp.fc2.weight, decoder_blocks.4.mlp.fc2.bias, decoder_blocks.5.norm1.weight, decoder_blocks.5.norm1.bias, decoder_blocks.5.attn.qkv.weight, decoder_blocks.5.attn.qkv.bias, decoder_blocks.5.attn.proj.weight, decoder_blocks.5.attn.proj.bias, decoder_blocks.5.norm2.weight, decoder_blocks.5.norm2.bias, decoder_blocks.5.mlp.fc1.weight, decoder_blocks.5.mlp.fc1.bias, decoder_blocks.5.mlp.fc2.weight, decoder_blocks.5.mlp.fc2.bias, decoder_blocks.6.norm1.weight, decoder_blocks.6.norm1.bias, decoder_blocks.6.attn.qkv.weight, decoder_blocks.6.attn.qkv.bias, decoder_blocks.6.attn.proj.weight, decoder_blocks.6.attn.proj.bias, decoder_blocks.6.norm2.weight, decoder_blocks.6.norm2.bias, decoder_blocks.6.mlp.fc1.weight, decoder_blocks.6.mlp.fc1.bias, decoder_blocks.6.mlp.fc2.weight, decoder_blocks.6.mlp.fc2.bias, decoder_blocks.7.norm1.weight, decoder_blocks.7.norm1.bias, decoder_blocks.7.attn.qkv.weight, decoder_blocks.7.attn.qkv.bias, decoder_blocks.7.attn.proj.weight, decoder_blocks.7.attn.proj.bias, decoder_blocks.7.norm2.weight, decoder_blocks.7.norm2.bias, decoder_blocks.7.mlp.fc1.weight, decoder_blocks.7.mlp.fc1.bias, decoder_blocks.7.mlp.fc2.weight, decoder_blocks.7.mlp.fc2.bias, decoder_norm.weight, decoder_norm.bias, decoder_pred.weight, decoder_pred.bias
missing keys in source state_dict: fpn1.0.weight, fpn1.0.bias, fpn1.1.weight, fpn1.1.bias, fpn1.1.running_mean, fpn1.1.running_var, fpn1.3.weight, fpn1.3.bias, fpn2.0.weight, fpn2.0.bias
2023-10-29 15:52:05,979 - mmdet - INFO - loading checkpoint for <class 'mmdet.models.roi_heads.bbox_heads.mae_bbox_head.MAEBBoxHead'>
Use load_from_local loader
2023-10-29 15:52:06,150 - mmdet - WARNING - The model and loaded state dict do not match exactly
unexpected key in source state_dict: cls_token, mask_token, decoder_norm.weight, decoder_norm.bias, decoder_pred.weight, decoder_pred.bias
missing keys in source state_dict: fc_cls.weight, fc_cls.bias, fc_reg.weight, fc_reg.bias, decoder_box_norm.weight, decoder_box_norm.bias
Use load_from_local loader
Traceback (most recent call last):
File "./tools/test.py", line 220, in
main()
File "./tools/test.py", line 177, in main
checkpoint = load_checkpoint(model, args.checkpoint, map_location='cpu')
File "/root/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/checkpoint.py", line 513, in load_checkpoint
checkpoint = _load_checkpoint(filename, map_location, logger)
File "/root/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/checkpoint.py", line 451, in _load_checkpoint
return CheckpointLoader.load_checkpoint(filename, map_location, logger)
File "/root/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/checkpoint.py", line 244, in load_checkpoint
return checkpoint_loader(filename, map_location)
File "/root/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/checkpoint.py", line 261, in load_from_local
checkpoint = torch.load(filename, map_location=map_location)
File "/root/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/serialization.py", line 585, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/root/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/serialization.py", line 755, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '\xe2'.
Traceback (most recent call last):
File "/root/miniconda3/envs/open-mmlab/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/root/miniconda3/envs/open-mmlab/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/root/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launch.py", line 261, in
main()
File "/root/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launch.py", line 257, in main
cmd=cmd)
subprocess.CalledProcessError: Command '['/root/miniconda3/envs/open-mmlab/bin/python', '-u', './tools/test.py', '--local_rank=0', 'configs/imted/few_shot/imted_faster_rcnn_vit_base_2x_finetuning_10shot_coco.py', 'work_dirs/imted_faster_rcnn_vit_base_2x_finetuning_10shot_coco/mae_vit_small_800e.pth', '--launcher', 'pytorch', '--eval', 'bbox']' returned non-zero exit status 1.`
Hi, what is the total number of parameters in your models?
I have a custom dataset with following summary:
| class | images boxes |
│ car │ 8144 │ 188839 │
│ person │ 9228 │139964│
│ Total │ 9946 | 328803 │
There are 2 base classes in my dataset. Later I prepared another class with 355 images and 2000~ boxes and merged this into my main dataset. Then I prepared my split using the prepare_coco_few_shot.py. And have the exact format as mentioned for my 3 classes.
Training:
I trained my model without the 3rd (novel class) [namely I had 2 classes while training] using theconfigs/imted/imted_faster_rcnn_vit_base_mae_3x_coco.py
and obtained a pretrained model let's call it X.
I proceeded to finetune my model on the few shot task for the novel classes. Created a configuration file containing all 3 classes with the novel one being the last. I am confused about how to fill the part below in the few shot configuration file:
pretrained = "path/to/the/X.pth"
.....
classes = ('truck', 'traffic light', 'fire hydrant', 'stop sign', 'parking meter', 'bench', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella',
'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard',
'tennis racket', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog',
'pizza', 'donut', 'cake', 'bed', 'toilet', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator',
'book', 'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'boat',
'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'bottle', 'chair', 'couch', 'potted plant', 'dining table', 'tv')
data = dict(
samples_per_gpu=2,
workers_per_gpu=2,
train=dict(
pipeline=train_pipeline,
classes = classes,
ann_file='path/of/annotations/json'),
val=dict(
classes = classes,
ann_file='path/of/annotations/json'),
test=dict(
classes = classes,
ann_file='path/of/annotations/json'))
I did the below configuration at first:
classes = ('person', 'car', 'mynovel')
data = dict(
samples_per_gpu=2,
workers_per_gpu=2,
train=dict(
pipeline=train_pipeline,
classes = classes,
ann_file= 'data/cocosplit/seed1/full_box_30shot_mynovel_trainval.json',
img_prefix='data/merged_dataset/all/'),
val=dict(
classes = classes,
ann_file='data/merged_dataset/annotations/fsod_test.json',
img_prefix='data/merged_dataset/all/'),
test=dict(
classes = classes,
ann_file= 'data/merged_dataset/annotations/fsod_test.json',
img_prefix='data/merged_dataset/all/'))
However, the results I am getting at epoch 102 is:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.001
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.004
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.003
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.043
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.043
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.043
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.003
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.030
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.120
Note that, my prior classes (car and person) got around 0.724 AP score for above metrics.
Also I left load_from = None as I didn't know what to put there because I only got one pretrained model and I am training the model that I can load right now.
I also considered training with the fsod_train but I have more than 30 shots for the novel class there. I checked the original trainvalno5k.json and it's also the same.
To summarize, I am having problems writing the configuration file for the few shot. I think, how you utilize the datasplit/{trainvalno5k, 5k}.json and the novel shots (the json files under the cocosplit/seeds) is ambiguous and would be very happy if you could shed a light on this topic.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.