Comments (8)
@kjwkch @LiXiang0021
After reviewing the current implementation, I noticed that besides the fusion modification, the point feature pass through the 2D learner needs to be added to the original point feature before passing through multihead_3d_classifier
, in order to match the model architecture outlined in the paper as below:
feat_learner = F.relu(self.leaners[idx](pts_feat))
# feat_learner -> voxel-wise feature after 2D learner
pts_pred_full = self.multihead_3d_classifier[idx]((pts_feat+feat_learner))
# pts_feat+feat_learner -> voxel-wise Enhanced 3D Features
# correspondence
pts_label_full = self.voxelize_labels(data_dict['labels'], data_dict['layer_{}'.format(idx)]['full_coors'])
pts_pred = self.p2img_mapping(pts_pred_full[coors_inv], point2img_index, batch_idx)
# modality fusion
feat_learner = self.p2img_mapping(feat_learner[coors_inv], point2img_index, batch_idx)
# feat_learner -> point-wise feature after 2D learner and img_mapping
feat_cat = torch.cat([img_feat, feat_learner], 1)
feat_cat = self.fcs1[idx](feat_cat)
feat_weight = torch.sigmoid(self.fcs2[idx](feat_cat))
fuse_feat = F.relu(feat_cat * feat_weight) + img_feat
Currently, the implementation takes the point feature as input directly for multihead_3d_classifier
instead of adding the point feature after the 2D learner.
Lines 89 to 93 in 80b8646
@yanx27, I would appreciate any suggestions you may have regarding this matter.
from 2dpass.
i have the same question
thanks in advance
from 2dpass.
Have you guys successively reproduced the model on Nuscenes, I did several experiments but the performance is far away from the report results. And, I also tested the provided weight on Nuscences getting results similar to the reported results. I'd like to know if I forgot to set some arguments.
from 2dpass.
This issue is code implementation, not performance. I think open code is not implemented as described in the paper.
from 2dpass.
Missing in the code.
modality fusion
feat_learner = F.relu(self.leaners[idx](pts_feat))
feat_cat = torch.cat([img_feat, feat_learner], 1)
feat_cat = self.fcs1[idx](feat_cat)
feat_weight = torch.sigmoid(self.fcs2[idx](feat_cat))
fuse_feat = F.relu(feat_cat * feat_weight)
I think that [fuse_feat = F.relu(feat_cat*feat_wieght) + img_feat] implements the formula in the paper as a code.
feat_learner = F.relu(self.leaners[idx](pts_feat))
feat_cat = torch.cat([img_feat, feat_learner], 1)
feat_cat = self.fcs1[idx](feat_cat)
feat_weight = torch.sigmoid(self.fcs2[idx](feat_cat))
fuse_feat = F.relu(feat_cat * feat_weight) + img_feat
from 2dpass.
Thanks for your reply, I will further check this issue.
from 2dpass.
I just trained the modified version as you said, and the performance did improve a little bit around 2 on mIoU. I believe there may be some other wrong implements or missing in the released code. And thank you again.
from 2dpass.
@jaywu109 does the new script changes work for you ?
from 2dpass.
Related Issues (20)
- inference time of 2DPASS (4scale-64dimension)
- Problems that arise during training
- Dataset Nuscenes HOT 1
- link dataset nuscenses HOT 1
- class noise
- Intensity HOT 1
- about pretrained model HOT 4
- use of intensity in 2DPASS
- When I swapped the 3D network for my own, some problems occurred
- multi-scales
- SPVCNN-nuscenes
- Running Inferences on Real-Time Data
- pre-trained model in Table 2 HOT 4
- the sacle of the 3dconv
- difference betwe paper and code about modali fuse HOT 1
- Size mismatch for pretrained model
- Segmentation fault(core dumped)
- ValueError: Input must be 1- or 2-d. HOT 1
- some issues about the idfference between the paper and the code HOT 1
- About the trainingTime
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from 2dpass.