@kjwkch @LiXiang0021
After reviewing the current implementation, I noticed that besides the fusion modification, the point feature pass through the 2D learner needs to be added to the original point feature before passing through multihead_3d_classifier, in order to match the model architecture outlined in the paper as below:

        feat_learner = F.relu(self.leaners[idx](pts_feat)) 
        # feat_learner -> voxel-wise feature after 2D learner

        pts_pred_full = self.multihead_3d_classifier[idx]((pts_feat+feat_learner)) 
        # pts_feat+feat_learner -> voxel-wise Enhanced 3D Features

        # correspondence
        pts_label_full = self.voxelize_labels(data_dict['labels'], data_dict['layer_{}'.format(idx)]['full_coors'])
        pts_pred = self.p2img_mapping(pts_pred_full[coors_inv], point2img_index, batch_idx)

        # modality fusion

        feat_learner = self.p2img_mapping(feat_learner[coors_inv], point2img_index, batch_idx)
        # feat_learner -> point-wise feature after 2D learner and img_mapping

        feat_cat = torch.cat([img_feat, feat_learner], 1)
        feat_cat = self.fcs1[idx](feat_cat)
        feat_weight = torch.sigmoid(self.fcs2[idx](feat_cat))
        fuse_feat = F.relu(feat_cat * feat_weight) + img_feat

Currently, the implementation takes the point feature as input directly for multihead_3d_classifier instead of adding the point feature after the 2D learner.

@yanx27, I would appreciate any suggestions you may have regarding this matter.

from 2dpass.

i have the same question

thanks in advance

from 2dpass.

Have you guys successively reproduced the model on Nuscenes, I did several experiments but the performance is far away from the report results. And, I also tested the provided weight on Nuscences getting results similar to the reported results. I'd like to know if I forgot to set some arguments.

from 2dpass.

This issue is code implementation, not performance. I think open code is not implemented as described in the paper.

from 2dpass.

Missing in the code.

modality fusion

feat_learner = F.relu(self.leaners[idx](pts_feat))
feat_cat = torch.cat([img_feat, feat_learner], 1)
feat_cat = self.fcs1[idx](feat_cat)
feat_weight = torch.sigmoid(self.fcs2[idx](feat_cat))
fuse_feat = F.relu(feat_cat * feat_weight)

I think that [fuse_feat = F.relu(feat_cat*feat_wieght) + img_feat] implements the formula in the paper as a code.

feat_learner = F.relu(self.leaners[idx](pts_feat))
feat_cat = torch.cat([img_feat, feat_learner], 1)
feat_cat = self.fcs1[idx](feat_cat)
feat_weight = torch.sigmoid(self.fcs2[idx](feat_cat))
fuse_feat = F.relu(feat_cat * feat_weight) + img_feat

from 2dpass.

Thanks for your reply, I will further check this issue.

from 2dpass.

I just trained the modified version as you said, and the performance did improve a little bit around 2 on mIoU. I believe there may be some other wrong implements or missing in the released code. And thank you again.

from 2dpass.

@jaywu109 does the new script changes work for you ?

from 2dpass.