Comments (4)
Thanks for your interest in our work.
This repo is mainly designed for CLIP-ViT. If you want to use RN50, you need to modify some lines as follows (take voc as example):
generate_cams_voc12.py
1. Comment out or remove line 134
CLIP-ES/generate_cams_voc12.py
Line 134
in
f0cecc7
CLIP-ES/generate_cams_voc12.py
Line 134 in f0cecc7
2. Replace image_features
with image
in line 144: input_tensor = [image, text_features_temp.to(device_id), h, w]
CLIP-ES/generate_cams_voc12.py
Line 144
in
f0cecc7
CLIP-ES/generate_cams_voc12.py
Line 144 in f0cecc7
3. Comment out or remove line 160 - line 197, line 202, line 207, line 213, uncomment line 212,
4. Change target_layers
in line 239: target_layers = [model.visual.layer4[-1]]
CLIP-ES/generate_cams_voc12.py
Line 239
in
f0cecc7
CLIP-ES/generate_cams_voc12.py
Line 239 in f0cecc7
5. Set reshape_transform
to None
in line 240: cam = GradCAM(model=model, target_layers=target_layers, reshape_transform=None)
CLIP-ES/generate_cams_voc12.py
Line 240
in
f0cecc7
CLIP-ES/generate_cams_voc12.py
Line 240 in f0cecc7
clip/model.py
1. Add a forward function for resnet50 in CLIP object, such as:
def forward_features_resnet50(self, image, text_features, h, w):
image_features = self.encode_image(image, h, w)
# normalized features
image_features = image_features / image_features.norm(dim=1, keepdim=True)
text_features = text_features / text_features.norm(dim=1, keepdim=True)
# cosine similarity as logits
logit_scale = self.logit_scale.exp()
logits_per_image = logit_scale * image_features @ text_features.t()
# shape = [global_batch_size, global_batch_size]
logits_per_image = logits_per_image.softmax(dim=-1)
attn_weight = None
return logits_per_image, attn_weight
pytorch_grad_cam/activations_and_gradients.py
1. Set self.height=H // 32, self.width=W // 32
in line 40-41.
CLIP-ES/pytorch_grad_cam/activations_and_gradients.py
Lines 40 to 41
in
f0cecc7
CLIP-ES/pytorch_grad_cam/activations_and_gradients.py
Lines 40 to 41 in f0cecc7
2. Change forward function to self.model.forward_features_resnet50(x[0], x[1], H, W)
in
from clip-es.
You can try different bg thresholds (such as 8 or 9 indicated by the result of eval_cam.py
in
Line 111 in f0cecc7
from clip-es.
感谢您耐心的回复!
我按照上述步骤改了代码,可以正常运行。但是我在 CLIP-RN50 产生的 CAM 的基础上加上了 CRF ,结果比较低。
python eval_cam_with_crf.py --eval_only
得到的结果
'Pixel Accuracy': 0.7503926354038674, 'Mean Accuracy': 0.14167843883095207, 'Frequency Weighted IoU': 0.5694430970966254, 'Mean IoU': 0.127361159247873
请问这是什么原因造成的?(用ViT-B/16产生的伪标签精度和论文一致)
from clip-es.
Thanks! It works.
from clip-es.
Related Issues (20)
- CGL损失在什么时候使用的? HOT 3
- The DeepLab config HOT 7
- Clarification on COCO Dataset Segmentation Groundtruth HOT 3
- The voc results HOT 4
- Clarification on Deeplab Model Configuration and Pre-training HOT 2
- Reproducing miou 58.6 (Initial) from Table 2 HOT 2
- eval CRF processed pseudo masks HOT 3
- Running Problems HOT 6
- Set of referring image segmentation queries HOT 2
- Cam performance HOT 4
- softmax processing on Grad CAM HOT 1
- Some details HOT 5
- "origami" and "rendering HOT 2
- Problem with the link for train_aug groundtruth HOT 1
- Thanks for you work, could you provide the setting of training Deeplabv2 on COCO dataset? HOT 2
- COCO数据集的SegmentationClass文件 HOT 2
- train with my own dataset HOT 5
- loss and gradient backpropagation HOT 2
- Request for clip_wrapper.py Compatible with open-clip for CAM Generation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clip-es.