a-suozhang / ada3d Goto Github PK

cuda memory increased with the iteration increasing,2M per/iteration on average,I find that it's owing to the mask_bn layer,can you tell the reason?

when I run python test.py --cfg ./cfgs/predictor_masked_bn_tune.yaml --ckpt ./pretrained-models/checkpoint.pth --hard_drop
shows File "/home/nvidia/way/ada3d/openpcdet/tools/../pcdet/models/detectors/centerpoint.py", line 214, in predictor_forward batch_dict, pred_heatmap = cur_module(batch_dict, enable_predictor=True, predictor_args=predictor_args) File "/home/nvidia/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/nvidia/way/ada3d/openpcdet/tools/../pcdet/models/backbones_3d/spconv_backbone.py", line 373, in forward inbox_rate, check_inbox_save_d = check_inbox(self.cfg, batch_dict, (pred_heatmap_*bev_valid_mask).squeeze(1), (pred_heatmap_valid_mask*bev_valid_mask).squeeze(1), if_save_runtime=if_save_runtime) File "/home/nvidia/way/ada3d/openpcdet/tools/../pcdet/models/backbones_3d/drop_utils.py", line 64, in check_inbox xy_meshgrid = torch.cat([xy_meshgrid,torch.zeros([N,1],device=device_),torch.ones([N,1],device=device_)], dim=1) # [N,4] RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 35024 but got size 35200 for tensor number 1 in the list.
but when I run python test.py --cfg ./cfgs/predictor_masked_bn_tune.yaml --ckpt ./pretrained-models/checkpoint.pth it works normally.

what is the difference between with --hard_drop and without --hard_drop?,how can I solve the problem?

Any plan about this?