@yukang2017
hello，I use your work to test nuscenes datasets，but the results is not very well .And the weight is not match with the model,I use the cbgs_voxel0075_voxelnext.yaml to test your pre-trained model trained with the nuscenes dataset . The result is show as follows

INFO ==> Loading parameters from checkpoint /home/mia/OpenPCDet/pretrain_model/voxelnext_nuscenes_kernel1.pth to GPU
2023-03-23 16:42:13,536 INFO Not updated weight backbone_3d.conv_input.0.weight: torch.Size([16, 3, 3, 3, 4])
2023-03-23 16:42:13,536 INFO ==> Done (loaded 541/542)

mAP: 0.1039
mATE: 0.4398
mASE: 0.2907
mAOE: 0.7989
mAVE: 0.7291
mAAE: 0.2989
NDS: 0.2962
Eval time: 86.0s

Per-class results:
Object Class AP ATE ASE AOE AVE AAE
car 0.438 0.260 0.169 0.458 0.753 0.330
truck 0.117 0.450 0.221 0.415 0.641 0.318
bus 0.155 0.500 0.205 0.591 1.913 0.432
trailer 0.000 0.776 0.255 0.926 0.446 0.199
construction_vehicle 0.000 0.804 0.489 1.412 0.140 0.331
pedestrian 0.100 0.309 0.307 1.489 0.941 0.321
motorcycle 0.036 0.279 0.272 1.022 0.724 0.346
bicycle 0.000 0.299 0.293 0.786 0.273 0.113
traffic_cone 0.016 0.286 0.408 nan nan nan
barrier 0.177 0.434 0.288 0.089 nan nan
2023-03-23 16:54:41,291 INFO ----------------Nuscene detection_cvpr_2019 results-----------------
***car error@trans, scale, orient, vel, attr | [email protected], 1.0, 2.0, 4.0
0.26, 0.17, 0.46, 0.75, 0.33 | 33.53, 41.90, 47.20, 52.54 | mean AP: 0.4379142060810768
***truck error@trans, scale, orient, vel, attr | [email protected], 1.0, 2.0, 4.0
0.45, 0.22, 0.42, 0.64, 0.32 | 4.76, 9.66, 14.33, 18.14 | mean AP: 0.11723098807042298
***construction_vehicle error@trans, scale, orient, vel, attr | [email protected], 1.0, 2.0, 4.0
0.80, 0.49, 1.41, 0.14, 0.33 | 0.00, 0.00, 0.00, 0.00 | mean AP: 0.0
***bus error@trans, scale, orient, vel, attr | [email protected], 1.0, 2.0, 4.0
0.50, 0.21, 0.59, 1.91, 0.43 | 6.65, 12.97, 19.10, 23.12 | mean AP: 0.15460599829470253
***trailer error@trans, scale, orient, vel, attr | [email protected], 1.0, 2.0, 4.0
0.78, 0.26, 0.93, 0.45, 0.20 | 0.00, 0.00, 0.00, 0.04 | mean AP: 9.730399922215814e-05
***barrier error@trans, scale, orient, vel, attr | [email protected], 1.0, 2.0, 4.0
0.43, 0.29, 0.09, nan, nan | 8.99, 16.68, 21.40, 23.60 | mean AP: 0.17667373910591053
***motorcycle error@trans, scale, orient, vel, attr | [email protected], 1.0, 2.0, 4.0
0.28, 0.27, 1.02, 0.72, 0.35 | 2.90, 3.76, 3.84, 3.95 | mean AP: 0.03611132627896975
***bicycle error@trans, scale, orient, vel, attr | [email protected], 1.0, 2.0, 4.0
0.30, 0.29, 0.79, 0.27, 0.11 | 0.00, 0.00, 0.00, 0.00 | mean AP: 0.0
***pedestrian error@trans, scale, orient, vel, attr | [email protected], 1.0, 2.0, 4.0
0.31, 0.31, 1.49, 0.94, 0.32 | 7.60, 9.86, 10.51, 12.15 | mean AP: 0.10028934559384498
***traffic_cone error@trans, scale, orient, vel, attr | [email protected], 1.0, 2.0, 4.0
0.29, 0.41, nan, nan, nan | 1.03, 1.23, 1.59, 2.72 | mean AP: 0.016428388021770043
--------------average performance-------------
trans_err: 0.4398
scale_err: 0.2907
orient_err: 0.7989
vel_err: 0.7291
attr_err: 0.2989
mAP: 0.1039
NDS: 0.2962

Thank you for the excellent research.
I have a question about the cfgs file provided in the code. When I tried to reproduce using the cbgs_voxel0075_voxelnext.yaml, I noticed that the FLOPs calculation shows a model with a spatial pruning ratio of 0 (83.8 GFLOPs), as seen in Table 3. Is there a model config file available with a spatial pruning ratio of 0.5, or is there a way to separately configure the pruning ratio?

Hi, thanks for your excellent work.
As shown in Tab. 1, the FLOPs is calculated for sparse cnn and head separately. I noticed that the FLOPs of sparse cnn and head (VoxelNeXt) is 33.6G and 5.1G. So, I want to know whether the sparse conv in 3D backbone and head (VoxelNeXt) is considered in calculation. AFAK, the mmcv and fvcore do not support the FLOPs calculation for sparse conv.
Besides, the head of CenterPoint is consists of 2D conv, which is supported by mmcv and fvcore.

Hi,

Thanks for your amazing repo.

I found the following config files for nuscenes. Could you give more information about these config files? I want to use your model for detecting vehicles in nuscenes.

Thank you

Hi, could give me some advice about visualizing the effective receptive field (like Fig. 5 in the paper) and normal receptive field?
Thanks!

Hi, yukang @yukang2017, yanwei @yanwei-li,
I am confused about the iou related loss you used in dense head(IoULossSparse and IoURegLossSparse). I reproduce your model on my devices, but I find IoURegLossSparse does not seem to going down, it keeps almost the same value in the training process. I use IoURegLoss in another model, I trap into the same problem(not decrease). Could you please show me your intention to use the loss?

Additionally, why use IoULossSparse and IoURegLossSparse repsectively? I can't understand the purpose of their use. Looking forward to your reply, thank you.

Best regards.
deze

Hi thanks for the contribution,
I was trying to find where you extract the query voxels and associate them across frames, but I can't find it in the code. Could you please point me to it?

Hello,
I have a great time exploring your code.

But I have a question which I can't understand

518 line in the above code, you change spatial_indices change [2,1]
But I don't understand why you change the order.
Does the order affect some compatibilities of other libraries such as nuscenes or waymo?
Or is it ok changing [2,1] --> [1,2] if I modify "def _get_predicted_boxes" and " def assign_target_of_single_head" to encode and decode bboxes

Thank you for reading.

Hi, thanks for publishing the code for your work!

I noticed the large model with 256 channels provides better results on Waymo, but the models for nuscenes and argoverse are not scaled up and use 128 channels. Is there a particular reason for this decision, or do you think the same scaling work on these datasets as well?

想请教一下，论文中table11的消融实验，voxel association具体是怎么做的，如何计算两个voxel query之间的相关性

Hello, I noticed that on all datasets you are not using spatial voxel pruning, i.e. VoxelResBackBone8xVoxelNeXt2DSPS or VoxelResBackBone8xVoxelNeXtSPS, so this is just a trick for weighing performance against FLOPS?

What version of the Waymo dataset are you using? Is it the latest 2.0 version?

I think it should add checking of mask_cls as follows:

if mask_cls.sum() == 0:
    continue

Hi, I followed the released config file to train VoxelNeXt on the KITTI, but the detection results don't seem to be good?

Here is the epoch with the best results trained on my machine.

2023-05-30 13:25:51,687 INFO *************** Performance of EPOCH 72 *****************
2023-05-30 13:25:51,688 INFO Generate label finished(sec_per_example: 0.0634 second).
2023-05-30 13:25:51,688 INFO recall_roi_0.3: 0.000000
2023-05-30 13:25:51,688 INFO recall_rcnn_0.3: 0.935357
2023-05-30 13:25:51,688 INFO recall_roi_0.5: 0.000000
2023-05-30 13:25:51,688 INFO recall_rcnn_0.5: 0.873562
2023-05-30 13:25:51,688 INFO recall_roi_0.7: 0.000000
2023-05-30 13:25:51,689 INFO recall_rcnn_0.7: 0.633500
2023-05-30 13:25:51,715 INFO Average predicted number of objects(3769 samples): 12.604
2023-05-30 13:26:09,365 INFO Car [email protected], 0.70, 0.70:
bbox AP:90.4678, 89.2926, 88.3760
bev AP:89.0743, 86.4952, 83.4077
3d AP:86.5793, 76.6896, 74.6050
aos AP:90.45, 89.11, 88.13
Car [email protected], 0.70, 0.70:
bbox AP:95.5828, 91.9155, 90.6527
bev AP:91.6419, 87.5357, 85.0153
3d AP:87.9198, 77.8969, 75.1770
aos AP:95.56, 91.71, 90.39
Car [email protected], 0.50, 0.50:
bbox AP:90.4678, 89.2926, 88.3760
bev AP:94.7389, 89.6084, 89.1113
3d AP:94.6626, 89.5168, 88.9023
aos AP:90.45, 89.11, 88.13
Car [email protected], 0.50, 0.50:
bbox AP:95.5828, 91.9155, 90.6527
bev AP:96.8324, 94.3452, 93.6283
3d AP:96.7766, 94.1481, 91.5978
aos AP:95.56, 91.71, 90.39
Pedestrian [email protected], 0.50, 0.50:
bbox AP:76.9811, 73.3218, 69.6398
bev AP:65.7048, 61.0455, 56.5700
3d AP:62.2981, 56.9214, 52.1742
aos AP:75.42, 71.20, 67.12
Pedestrian [email protected], 0.50, 0.50:
bbox AP:77.9146, 73.9838, 70.5503
bev AP:66.2811, 60.9713, 55.7163
3d AP:61.5019, 56.5423, 50.7263
aos AP:76.24, 71.67, 67.70
Pedestrian [email protected], 0.25, 0.25:
bbox AP:76.9811, 73.3218, 69.6398
bev AP:82.2534, 79.8235, 76.3670
3d AP:82.1528, 79.5656, 76.0224
aos AP:75.42, 71.20, 67.12
Pedestrian [email protected], 0.25, 0.25:
bbox AP:77.9146, 73.9838, 70.5503
bev AP:84.9853, 81.9345, 77.8001
3d AP:84.8836, 81.4974, 77.4063
aos AP:76.24, 71.67, 67.70
Cyclist [email protected], 0.50, 0.50:
bbox AP:87.4056, 73.9096, 71.1011
bev AP:84.3527, 69.5915, 65.2032
3d AP:81.7647, 66.3772, 62.1164
aos AP:87.32, 73.52, 70.74
Cyclist [email protected], 0.50, 0.50:
bbox AP:89.9100, 75.6070, 71.9232
bev AP:86.1329, 70.2481, 66.1244
3d AP:82.1792, 65.8815, 61.9532
aos AP:89.81, 75.18, 71.51
Cyclist [email protected], 0.25, 0.25:
bbox AP:87.4056, 73.9096, 71.1011
bev AP:86.7529, 71.9742, 69.1377
3d AP:86.7529, 71.9742, 69.1377
aos AP:87.32, 73.52, 70.74
Cyclist [email protected], 0.25, 0.25:
bbox AP:89.9100, 75.6070, 71.9232
bev AP:89.0928, 73.4686, 69.6789
3d AP:89.0928, 73.4686, 69.6789
aos AP:89.81, 75.18, 71.51

Hi Yukang,

I was trying to run the evaluation script and am getting the following error.

--cfg_file /mnt/tools/cfgs/nuscenes_models/cbgs_voxel0075_voxelnext.yaml --ckpt /mnt/VoxelNeXt/voxelnext_nuscenes_kernel1.pth
+ NGPUS=2
+ PY_ARGS='--cfg_file /mnt/tools/cfgs/nuscenes_models/cbgs_voxel0075_voxelnext.yaml --ckpt /mnt/VoxelNeXt/voxelnext_nuscenes_kernel1.pth'
+ python -m torch.distributed.launch --nproc_per_node=2 test.py --launcher pytorch --cfg_file /mnt/tools/cfgs/nuscenes_models/cbgs_voxel0075_voxelnext.yaml --ckpt /mnt/VoxelNeXt/voxelnext_nuscenes_kernel1.pth
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
Traceback (most recent call last):
  File "test.py", line 14, in <module>
    from eval_utils import eval_utils
  File "/mnt/VoxelNeXt/tools/eval_utils/eval_utils.py", line 8, in <module>
    from pcdet.models import load_data_to_gpu
  File "../pcdet/models/__init__.py", line 6, in <module>
    from .detectors import build_detector
  File "../pcdet/models/detectors/__init__.py", line 1, in <module>
    from .detector3d_template import Detector3DTemplate
  File "../pcdet/models/detectors/detector3d_template.py", line 7, in <module>
    from ...utils.spconv_utils import find_all_spconv_keys
  File "../pcdet/utils/spconv_utils.py", line 4, in <module>
    if float(spconv.__version__[2:]) >= 2.2:
AttributeError: module 'spconv' has no attribute '__version__'
Traceback (most recent call last):
  File "test.py", line 14, in <module>
    from eval_utils import eval_utils
  File "/mnt/VoxelNeXt/tools/eval_utils/eval_utils.py", line 8, in <module>
    from pcdet.models import load_data_to_gpu
  File "../pcdet/models/__init__.py", line 6, in <module>
    from .detectors import build_detector
  File "../pcdet/models/detectors/__init__.py", line 1, in <module>
    from .detector3d_template import Detector3DTemplate
  File "../pcdet/models/detectors/detector3d_template.py", line 7, in <module>
    from ...utils.spconv_utils import find_all_spconv_keys
  File "../pcdet/utils/spconv_utils.py", line 4, in <module>
    if float(spconv.__version__[2:]) >= 2.2:
AttributeError: module 'spconv' has no attribute '__version__'
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/launch.py", line 261, in <module>
    main()
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/launch.py", line 257, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', 'test.py', '--local_rank=1', '--launcher', 'pytorch', '--cfg_file', '/mnt/tools/cfgs/nuscenes_models/cbgs_voxel0075_voxelnext.yaml', '--ckpt', '/mnt/VoxelNeXt/voxelnext_nuscenes_kernel1.pth']' returned non-zero exit status 1.

I am using the docker image that you provide. docker pull djiajun1206/pcdet:pytorch1.6
When I run nvcc --version in the docker image, I get CUDA version 10.1.

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

So, with CUDA 10.1 and pytorch 1.6, I am unsure which version of spconv to install

Your help will be much appreciated.

Thank you
Naren

Can you share the code for the tracking section @yukang2017

Hello, can you share your machines and training cost time on waymo? Such as waymo_models/voxelnext2d_ioubranch.yaml and waymo_models/voxelnext_ioubranch_large.yaml)

Hi,yukang
I want to use the network trained by nuscenes to test my own data,because my data is including [x,y,z,intensity] but no ring ,so I want trained the network only use the four informations,but I didn't train a good network. My data config file as follows
I only modify the used_feature_list

POINT_FEATURE_ENCODING: {
encoding_type: absolute_coordinates_encoding,
#used_feature_list: ['x', 'y', 'z', 'intensity', 'timestamp'],
used_feature_list: ['x', 'y', 'z', 'intensity'],
src_feature_list: ['x', 'y', 'z', 'intensity', 'timestamp'],
}

After 20 epochs the loss of the network is very high, I dont know reason of my problem,or maybe you can show your log of train the network.

2023-03-25 13:41:23,834 INFO epoch: 19/20, acc_iter=67400, cur_iter=2191/3432, batch_size=12, time_cost(epoch): 39:52/22:34, time_cost(all): 21:12:05/22:34, loss=21.39836883544922, d_time=0.01(0.04), f_time=1.10(1.06), b_time=1.11(1.09), lr=2.248708600854635e-06
2023-03-25 13:41:32,579 INFO Save latest model to /home/mia/OpenPCDet/output/nuscenes_models/cbgs_voxel0075_voxelnext/default/ckpt/latest_model
2023-03-25 13:42:18,937 INFO epoch: 19/20, acc_iter=67450, cur_iter=2241/3432, batch_size=12, time_cost(epoch): 40:47/21:40, time_cost(all): 21:13:00/21:40, loss=21.58650016784668, d_time=0.07(0.04), f_time=0.97(1.06), b_time=1.03(1.09), lr=2.07206874494086e-06
2023-03-25 13:43:12,857 INFO epoch: 19/20, acc_iter=67500, cur_iter=2291/3432, batch_size=12, time_cost(epoch): 41:41/20:45, time_cost(all): 21:13:54/20:45, loss=21.092235565185547, d_time=0.01(0.04), f_time=1.05(1.06), b_time=1.06(1.09), lr=1.902672465122041e-06
2023-03-25 13:43:13,586 INFO mroglo-202 Sat Mar 25 13:43:13 2023 525.85.05
[0] NVIDIA GeForce RTX 4090 | 43°C, 0 % | 21645 / 24564 MB | mia(21296M)
[1] NVIDIA GeForce RTX 4090 | 48°C, 100 % | 21735 / 24564 MB | mia(21386M)
[2] NVIDIA GeForce RTX 4090 | 45°C, 100 % | 24339 / 24564 MB | mia(23990M)

2023-03-25 13:44:08,146 INFO epoch: 19/20, acc_iter=67550, cur_iter=2341/3432, batch_size=12, time_cost(epoch): 42:36/19:50, time_cost(all): 21:14:49/19:50, loss=21.325319290161133, d_time=0.00(0.04), f_time=0.95(1.06), b_time=0.95(1.09), lr=1.7405222256555348e-06
2023-03-25 13:45:03,269 INFO epoch: 19/20, acc_iter=67600, cur_iter=2391/3432, batch_size=12, time_cost(epoch): 43:31/18:56, time_cost(all): 21:15:44/18:56, loss=21.13665771484375, d_time=0.01(0.04), f_time=1.03(1.06), b_time=1.04(1.09), lr=1.5856203853883526e-06
2023-03-25 13:45:58,019 INFO epoch: 19/20, acc_iter=67650, cur_iter=2441/3432, batch_size=12, time_cost(epoch): 44:26/18:02, time_cost(all): 21:16:39/18:02, loss=20.8839111328125, d_time=0.10(0.04), f_time=1.07(1.06), b_time=1.17(1.09), lr=1.4379691977227395e-06
2023-03-25 13:45:58,907 INFO mroglo-202 Sat Mar 25 13:45:58 2023 525.85.05
[0] NVIDIA GeForce RTX 4090 | 43°C, 0 % | 21645 / 24564 MB | mia(21296M)
[1] NVIDIA GeForce RTX 4090 | 48°C, 100 % | 21735 / 24564 MB | mia(21386M)
[2] NVIDIA GeForce RTX 4090 | 45°C, 100 % | 24339 / 24564 MB | mia(23990M)

Hi, thanks for your excellent work! i trained model with master configs (4x4) on A100. But the performance are slightly lower than in the paper. Any suggestions?
with voxelnext2d_ioubranch.yaml

with voxelnext_ioubranch.yaml

voxelnext_ioubranch_large.yaml

Hi,

Thank you for your impressive work! May I ask how many epochs of training have you set for nuSecnes?

Many thanks,
Jiaxuan

@yukang2017 hi, yukang! I am confused about the spatial voxel pruning module in the code.(spconv_backbone_voxelnext） It seemed that you didn't use it in the 3d backbone while downsampling, were you?

Hi,

Thank you for the impressive work! I noticed that you mentioned in your paper that VoxelNext doesn't require NMS process, but I still found the NMS settings in the POST_PROCESSING part of the configs. May I ask what is the purpose of this setting or if I misunderstand it?

Many thanks,
Ruixiao

The mAP on the NuScenes val set mentioned in the official paper is 56.5, and the mAP of the given pre-trained model is 60.5.
What's the difference between this two models?

Hi,

The command for evaulation in README.md uses tools/scripts/dist_tesh.sh

However, dist_test.sh does not seem to be there in tools/scripts

Could you kindly help with this?

Thank you
Naren

Hello

Can I ask you about your library version such as pytorch or kornia?

~/3Detector/VoxelNeXt/tools: ### python3 test.py --cfg_file /home/wzy/3Detector/VoxelNeXt/tools/cfgs/nuscenes_models/cbgs_voxel0075_voxelnext.yaml --ckpt /home/wzy/3Detector/voxelnext_nuscenes_kernel1.pth

2023-08-09 20:08:21,915 INFO Start logging
2023-08-09 20:08:21,915 INFO CUDA_VISIBLE_DEVICES=ALL
2023-08-09 20:08:21,915 INFO cfg_file /home/wzy/3Detector/VoxelNeXt/tools/cfgs/nuscenes_models/cbgs_voxel0075_voxelnext.yaml
2023-08-09 20:08:21,915 INFO batch_size 4
2023-08-09 20:08:21,915 INFO workers 4
2023-08-09 20:08:21,915 INFO extra_tag default
2023-08-09 20:08:21,915 INFO ckpt /home/wzy/3Detector/voxelnext_nuscenes_kernel1.pth
2023-08-09 20:08:21,915 INFO pretrained_model None
2023-08-09 20:08:21,915 INFO launcher none
2023-08-09 20:08:21,915 INFO tcp_port 18888
2023-08-09 20:08:21,915 INFO local_rank 0
2023-08-09 20:08:21,915 INFO set_cfgs None
2023-08-09 20:08:21,915 INFO max_waiting_mins 30
2023-08-09 20:08:21,915 INFO start_epoch 0
2023-08-09 20:08:21,915 INFO eval_tag default
2023-08-09 20:08:21,915 INFO eval_all False
2023-08-09 20:08:21,915 INFO ckpt_dir None
2023-08-09 20:08:21,915 INFO save_to_file False
2023-08-09 20:08:21,915 INFO infer_time False
2023-08-09 20:08:21,915 INFO cfg.ROOT_DIR: /home/wzy/3Detector/VoxelNeXt
2023-08-09 20:08:21,915 INFO cfg.LOCAL_RANK: 0
2023-08-09 20:08:21,915 INFO cfg.CLASS_NAMES: ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']
2023-08-09 20:08:21,915 INFO ----------- DATA_CONFIG -----------
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.DATASET: NuScenesDataset
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.DATA_PATH: ../data/nuscenes
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.VERSION: v1.0-mini
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.MAX_SWEEPS: 10
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.PRED_VELOCITY: True
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.SET_NAN_VELOCITY_TO_ZEROS: True
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.FILTER_MIN_POINTS_IN_GT: 1
2023-08-09 20:08:21,915 INFO ----------- DATA_SPLIT -----------
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.DATA_SPLIT.train: train
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.DATA_SPLIT.test: val
2023-08-09 20:08:21,915 INFO ----------- INFO_PATH -----------
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.INFO_PATH.train: ['nuscenes_infos_10sweeps_train.pkl']
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.INFO_PATH.test: ['nuscenes_infos_10sweeps_val.pkl']
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.BALANCED_RESAMPLING: True
2023-08-09 20:08:21,915 INFO ----------- DATA_AUGMENTOR -----------
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder']
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'gt_sampling', 'DB_INFO_PATH': ['nuscenes_dbinfos_10sweeps_withvelo.pkl'], 'USE_SHARED_MEMORY': False, 'DB_DATA_PATH': ['nuscenes_dbinfos_10sweeps_withvelo_global.pkl.npy'], 'PREPARE': {'filter_by_min_points': ['car:5', 'truck:5', 'construction_vehicle:5', 'bus:5', 'trailer:5', 'barrier:5', 'motorcycle:5', 'bicycle:5', 'pedestrian:5', 'traffic_cone:5']}, 'SAMPLE_GROUPS': ['car:2', 'truck:2', 'construction_vehicle:2', 'bus:2', 'trailer:2', 'barrier:2', 'motorcycle:2', 'bicycle:2', 'pedestrian:2', 'traffic_cone:2'], 'NUM_POINT_FEATURES': 5, 'DATABASE_WITH_FAKELIDAR': False, 'REMOVE_EXTRA_WIDTH': [0.0, 0.0, 0.0], 'LIMIT_WHOLE_SCENE': True}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x', 'y']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.9, 1.1]}, {'NAME': 'random_world_translation', 'NOISE_TRANSLATE_STD': [0.5, 0.5, 0.5]}]
2023-08-09 20:08:21,915 INFO ----------- POINT_FEATURE_ENCODING -----------
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity', 'timestamp']
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity', 'timestamp']
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': True}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.075, 0.075, 0.2], 'MAX_POINTS_PER_VOXEL': 10, 'MAX_NUMBER_OF_VOXELS': {'train': 120000, 'test': 160000}}]
2023-08-09 20:08:21,915 INFO cfg.DATA_CONFIG.BASE_CONFIG: cfgs/dataset_configs/nuscenes_dataset.yaml
2023-08-09 20:08:21,915 INFO ----------- MODEL -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.NAME: VoxelNeXt
2023-08-09 20:08:21,916 INFO ----------- VFE -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.VFE.NAME: MeanVFE
2023-08-09 20:08:21,916 INFO ----------- BACKBONE_3D -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.BACKBONE_3D.NAME: VoxelResBackBone8xVoxelNeXt
2023-08-09 20:08:21,916 INFO ----------- DENSE_HEAD -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.NAME: VoxelNeXtHead
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.INPUT_FEATURES: 128
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.CLASS_NAMES_EACH_HEAD: [['car'], ['truck', 'construction_vehicle'], ['bus', 'trailer'], ['barrier'], ['motorcycle', 'bicycle'], ['pedestrian', 'traffic_cone']]
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.SHARED_CONV_CHANNEL: 128
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.KERNEL_SIZE_HEAD: 1
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.USE_BIAS_BEFORE_NORM: True
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.NUM_HM_CONV: 2
2023-08-09 20:08:21,916 INFO ----------- SEPARATE_HEAD_CFG -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_ORDER: ['center', 'center_z', 'dim', 'rot', 'vel']
2023-08-09 20:08:21,916 INFO ----------- HEAD_DICT -----------
2023-08-09 20:08:21,916 INFO ----------- center -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center.out_channels: 2
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center.num_conv: 2
2023-08-09 20:08:21,916 INFO ----------- center_z -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center_z.out_channels: 1
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center_z.num_conv: 2
2023-08-09 20:08:21,916 INFO ----------- dim -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.dim.out_channels: 3
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.dim.num_conv: 2
2023-08-09 20:08:21,916 INFO ----------- rot -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.rot.out_channels: 2
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.rot.num_conv: 2
2023-08-09 20:08:21,916 INFO ----------- vel -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.vel.out_channels: 2
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.vel.num_conv: 2
2023-08-09 20:08:21,916 INFO ----------- TARGET_ASSIGNER_CONFIG -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.FEATURE_MAP_STRIDE: 8
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NUM_MAX_OBJS: 500
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.GAUSSIAN_OVERLAP: 0.1
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MIN_RADIUS: 2
2023-08-09 20:08:21,916 INFO ----------- LOSS_CONFIG -----------
2023-08-09 20:08:21,916 INFO ----------- LOSS_WEIGHTS -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 0.25
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.2, 0.2, 1.0, 1.0]
2023-08-09 20:08:21,916 INFO ----------- POST_PROCESSING -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.POST_PROCESSING.SCORE_THRESH: 0.1
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.POST_PROCESSING.POST_CENTER_LIMIT_RANGE: [-61.2, -61.2, -10.0, 61.2, 61.2, 10.0]
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.POST_PROCESSING.MAX_OBJ_PER_SAMPLE: 500
2023-08-09 20:08:21,916 INFO ----------- NMS_CONFIG -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.2
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 1000
2023-08-09 20:08:21,916 INFO cfg.MODEL.DENSE_HEAD.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 83
2023-08-09 20:08:21,916 INFO ----------- POST_PROCESSING -----------
2023-08-09 20:08:21,916 INFO cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
2023-08-09 20:08:21,916 INFO cfg.MODEL.POST_PROCESSING.EVAL_METRIC: kitti
2023-08-09 20:08:21,916 INFO ----------- OPTIMIZATION -----------
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 4
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.NUM_EPOCHS: 20
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.LR: 0.001
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.MOMENTUM: 0.9
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.MOMS: [0.95, 0.85]
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.PCT_START: 0.4
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.DIV_FACTOR: 10
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45]
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.LR_DECAY: 0.1
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.LR_CLIP: 1e-07
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.LR_WARMUP: False
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.WARMUP_EPOCH: 1
2023-08-09 20:08:21,916 INFO cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10
2023-08-09 20:08:21,917 INFO cfg.TAG: cbgs_voxel0075_voxelnext
2023-08-09 20:08:21,917 INFO cfg.EXP_GROUP_PATH: home/wzy/3Detector/VoxelNeXt/tools/cfgs/nuscenes_models
2023-08-09 20:08:21,917 INFO Loading NuScenes dataset
2023-08-09 20:08:21,917 INFO Total samples for NuScenes dataset: 0
2023-08-09 20:08:25,209 INFO ==> Loading parameters from checkpoint /home/wzy/3Detector/voxelnext_nuscenes_kernel1.pth to GPU
2023-08-09 20:08:25,240 INFO ==> Done (loaded 542/542)
2023-08-09 20:08:25,251 INFO *************** EPOCH 1 EVALUATION *****************
eval: 0it [00:00, ?it/s]
2023-08-09 20:08:25,420 INFO *************** Performance of EPOCH 1 *****************
Traceback (most recent call last):
File "test.py", line 207, in
main()
File "test.py", line 203, in main
eval_single_ckpt(model, test_loader, args, eval_output_dir, logger, epoch_id, dist_test=dist_test)
File "test.py", line 65, in eval_single_ckpt
eval_utils.eval_one_epoch(
File "/home/wzy/3Detector/VoxelNeXt/tools/eval_utils/eval_utils.py", line 94, in eval_one_epoch
sec_per_example = (time.time() - start_time) / len(dataloader.dataset)

ZeroDivisionError: float division by zero

cuda 113,
python 3.8,
pytorch 1.11.0,
spconv-plus 2.1.21
gpu A100
import spconv.pytorch
Traceback (most recent call last):
File "", line 1, in
File "/home/suwei/anaconda3/envs/max_voxelnext/lib/python3.8/site-packages/spconv/pytorch/init.py", line 6, in
from spconv.pytorch.core import SparseConvTensor
File "/home/suwei/anaconda3/envs/max_voxelnext/lib/python3.8/site-packages/spconv/pytorch/core.py", line 21, in
from spconv.tools import CUDAKernelTimer
File "/home/suwei/anaconda3/envs/max_voxelnext/lib/python3.8/site-packages/spconv/tools.py", line 16, in
from spconv.cppconstants import CPU_ONLY_BUILD
File "/home/suwei/anaconda3/envs/max_voxelnext/lib/python3.8/site-packages/spconv/cppconstants.py", line 15, in
import spconv.core_cc as _ext
ImportError: arg(): could not convert default argument 'timer: tv::CUDAKernelTimer' in method '<class 'spconv.core_cc.cumm.gemm.main.GemmParams'>.init' into a Python object (type not registered yet?)

Thank you for your great work,

• I read your paper, and study your code, but I have some problem when training my custom dataset. My dataset format same as KITTI. For other backbones like PVRCNN, SECOND it works normally, but with VoxelNext It wont work and get the error bellow. It would be nice if you show me some tips to fix the error.
  

• It would be nice if you have the guide to train KITTI, or Custom dataset, that is really helpful.
  Thank you so much.

Hi, thanks for sharing the code.
I would like to know how many GPUs do you use to train the large model on the waymo data set? Using tools/cfgs/waymo_models/voxelnext_ioubranch_large.yaml, I use 3090 GPUx4 with mixed-precision training, but the model has a problem of loss nan at the beginning,
I'm thinking is it because my batch-size is too small? or the learning rate 0.003 is too large? or beacause of use-amp?( I have tried to decrease the learning rate, close the mix-precision training, but it didn't work)
so could please provide a pretrain-model if it is possible?

Hi, thanks for sharing the code, after reading paper, I see you use 32x feature to enlarge ERF, and centerpoint's max downsample is 16 from 2d_backbone, so I just wonder whether you have the performance that centerpoint also use max downsample 32 to make a fair compare。
What's more, after reading the paper, I thought that if we change centerpoint to full sparse(more downsample in 3d backbone and no 2d backbone and sparse centerhead), we should have similar performance with centerpoint, but very efficient for long range object detection, do you think is reasonable?

Hello, Thanks for your great work. Can you share some data about the training settings and training speed?

hi @yukang2017 , could you please show me where the results of CenterPoint and SST you refer?
for CenterPoint, original paper results are not the same as you provided, for SST, I can only find Veh. and Ped. results from the original paper. Looking forward to your reply.

Hi, thanks for your excellent work! I want to compare your work with mine on argoverse2 dataset, but only find mAP results in the paper. Could you please give me the complete results on av2 ? My email is [email protected], Thanks a lot!

Thank you for open sourcing your work.

I need some help to run a single inference on the model. Evaluation results match the results shown in readme.

I managed to get 10 Lidar seeps from the nuscenes dataset to pass into the data processor how do I run the model on batch size 1?

I get results with a very low score.

[{'pred_boxes': tensor([[ 5.9413e+00,  3.6396e+00,  3.4623e-02,  ...,  2.1592e+00,
          8.4747e-02, -4.7400e-02],
        [ 8.3678e+00,  2.4014e+00, -1.2982e-02,  ...,  1.9676e+00,
          1.6921e-02,  3.2265e-02],
        [ 7.1588e+00,  3.0267e+00,  8.0288e-03,  ..., -7.9587e-01,
         -1.0607e-02,  4.5947e-02],
        ...,
        [-3.2396e+01,  2.6405e+01,  1.2968e-02,  ..., -1.3317e+00,
          2.1994e-02,  1.7819e-02],
        [ 9.5994e+00,  5.0404e+01,  5.9437e-03,  ..., -1.0493e+00,
          1.6636e-02,  8.8143e-03],
        [-7.8000e+00, -8.9908e+00,  4.4057e-03,  ..., -1.0186e+00,
          4.8190e-02,  5.5851e-03]], device='cuda:0'),
  'pred_ious': [None, None, None, None, None, None],
  'pred_labels': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
        3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
        3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
        3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 5, 5,
        5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
        5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
        5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
        5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
        6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
        6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
        6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7, 9, 9, 9,
        9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
        9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
        9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
        9, 9, 9, 9, 9, 9, 9, 9], device='cuda:0'),
  'pred_scores': tensor([0.1018, 0.1016, 0.1016, 0.1015, 0.1015, 0.1014, 0.1013, 0.1013, 0.1013,
        0.1013, 0.1013, 0.1013, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012,
        0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012,
        0.1012, 0.1012, 0.1012, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011,
        0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011,
        0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011,
        0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011,
        0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011,
        0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011, 0.1011,
        0.1011, 0.1011, 0.1045, 0.1034, 0.1034, 0.1033, 0.1032, 0.1032, 0.1032,
        0.1031, 0.1031, 0.1031, 0.1031, 0.1031, 0.1030, 0.1030, 0.1030, 0.1030,
        0.1030, 0.1030, 0.1030, 0.1030, 0.1029, 0.1029, 0.1029, 0.1029, 0.1029,
        0.1029, 0.1029, 0.1029, 0.1029, 0.1029, 0.1029, 0.1029, 0.1029, 0.1029,
        0.1029, 0.1029, 0.1029, 0.1029, 0.1029, 0.1028, 0.1028, 0.1028, 0.1028,
        0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028,
        0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028,
        0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028,
        0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028, 0.1028,
        0.1028, 0.1028, 0.1028, 0.1028, 0.1032, 0.1026, 0.1023, 0.1022, 0.1021,
        0.1019, 0.1019, 0.1019, 0.1018, 0.1018, 0.1018, 0.1017, 0.1017, 0.1017,
        0.1017, 0.1016, 0.1016, 0.1016, 0.1016, 0.1016, 0.1015, 0.1015, 0.1015,
        0.1015, 0.1015, 0.1015, 0.1015, 0.1014, 0.1014, 0.1014, 0.1014, 0.1014,
        0.1014, 0.1014, 0.1014, 0.1014, 0.1014, 0.1014, 0.1014, 0.1014, 0.1014,
        0.1014, 0.1013, 0.1013, 0.1013, 0.1013, 0.1013, 0.1013, 0.1013, 0.1013,
        0.1013, 0.1013, 0.1013, 0.1013, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012,
        0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012,
        0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012,
        0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1012, 0.1026, 0.1022, 0.1022,
        0.1021, 0.1021, 0.1020, 0.1020, 0.1020, 0.1019, 0.1019, 0.1018, 0.1018,
        0.1018, 0.1017, 0.1017, 0.1017, 0.1017, 0.1016, 0.1016, 0.1016, 0.1016,
        0.1016, 0.1016, 0.1016, 0.1016, 0.1016, 0.1016, 0.1016, 0.1016, 0.1016,
        0.1016, 0.1016, 0.1016, 0.1016, 0.1016, 0.1016, 0.1015, 0.1015, 0.1015,
        0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015,
        0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015,
        0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015,
        0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015,
        0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1015, 0.1000,
        0.1039, 0.1037, 0.1036, 0.1034, 0.1034, 0.1034, 0.1034, 0.1034, 0.1034,
        0.1034, 0.1034, 0.1034, 0.1034, 0.1034, 0.1034, 0.1033, 0.1033, 0.1033,
        0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033,
        0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033,
        0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033,
        0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033,
        0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1033, 0.1032, 0.1032,
        0.1032, 0.1032, 0.1032, 0.1032, 0.1032, 0.1032, 0.1032, 0.1032, 0.1032,
        0.1032, 0.1032, 0.1032, 0.1032, 0.1032, 0.1032, 0.1032, 0.1032, 0.1032,
        0.1032, 0.1032], device='cuda:0')}]

RuntimeError: Only tuples, lists and Variables are supported as JIT inputs/outputs. Dictionaries and strings are also accepted, but their usage is not recommended. Here, received an input of unsupported type: numpy.ndarray.
Have you export VoxelNeXt to ONNX?

Background:

• config: tools/cfgs/waymo_models/voxelnext2d_ioubranch.yaml

• pytorch version: 1.9.1+cu111

• spconv version: 2.2.6

• loss log:
  Train: 12/12 (100%) [ 988/989 (100%)] Loss: 4.254 (5.10) LR: 3.015e-08 Time cost: 14:15/00:00 [2:56:45/00:00] Acc_iter 11868 Data time: 0.00(0.06) Forward time: 0.29(0.81) Batch time: 0.30(0.87)

I get zero AP/APH on all object types. And looks like I am getting a lot of weird extents

Tiny box dim seen, return 0.0 IOU.
b1: center_x: 5.7256546020507812
center_y: 11.504196166992188
center_z: 2.9886384010314941
width: 22.703926086425781
length: 560.6590576171875
height: 0.00077404256444424391
heading: -2.1097447872161865

could you hint a bit where I may do it wrong?

Hello, can you please tell me how to calculate FPS in openpcdet?

There FLOPS results in the paper. Is there any way to get the flops by myself?

Hello Yukangchen, thank you for your excellent work in the field of 3D detection and tracking. I carefully studied this repo project as soon as it was released, and I am very interested in the new backbone network VoxelNext. The network structure is concise and similar to VoxelRCNN. I tried to use VoxelNext to train my own dataset and see how it performs. My dataset is in a format similar to KITTI, with four dimensions for each point cloud frame: x, y, z, intensity, and annotation files divided into 11 classes. I have completed the related modules for the custom dataloader, and I can load the data for training normally and generate the corresponding pkl files after preprocessing. However, I encountered the following problem at the beginning of training:

assert boxes_a.shape[0] == boxes_b.shape[0]
I have checked the progress of training my own data, and the error did not occur at the beginning but when running a certain frame, where boxes_a.shape[0] is 142 and boxes_b.shape[0] is 143. This caused an error and abnormal calculation of the IOU loss. Then I considered that this might be a data issue, but it only occurred in a very small number of data. I wonder if it is possible to ignore these data and set the loss to 0 when encountering such cases, without affecting the loss calculation of other normal data.
So I modified the code to the following form:


After I made the modifications, the training was able to proceed normally. I don't know if these changes could have introduced any errors. After training a batch of data today, I obtained the trained model. However, when I tried to load the trained model for inference, I encountered a problem where the scores of the detection boxes were extremely low, less than 0.1. As a result, when I set the input score threshold for post-processing to be greater than 0.1, no detection results were outputted. I'm not sure what the reason for this is, so I was wondering if you could help me analyze it.
I will continue to follow up on your project and hope to have more discussions with you. Below are my data and model configuration files:
voxelnext_ioubranch.yaml:
`CLASS_NAMES: ['car', 'pedestrian', 'cyclist', 'tricyclist', 'bus', 'truck', 'special_vehicle', 'traffic_cone',
'small_obstacle', 'traffic_facilities', 'other']

DATA_CONFIG:
BASE_CONFIG: cfgs/dataset_configs/at128_dataset.yaml
OUTPUT_PATH: '/lpai/output/models'

MODEL:
NAME: VoxelNeXt

VFE:
    NAME: MeanVFE

BACKBONE_3D:
    NAME: VoxelResBackBone8xVoxelNeXt

DENSE_HEAD:
    NAME: VoxelNeXtHead
    IOU_BRANCH: True
    CLASS_AGNOSTIC: False
    INPUT_FEATURES: 128

    CLASS_NAMES_EACH_HEAD: [
        ['car', 'pedestrian', 'cyclist', 'tricyclist', 'bus', 'truck', 'special_vehicle', 'traffic_cone',
          'small_obstacle', 'traffic_facilities', 'other']
    ]

    SHARED_CONV_CHANNEL: 128
    USE_BIAS_BEFORE_NORM: True
    NUM_HM_CONV: 2
    SEPARATE_HEAD_CFG:
        HEAD_ORDER: ['center', 'center_z', 'dim', 'rot']
        HEAD_DICT: {
            'center': {'out_channels': 2, 'num_conv': 2},
            'center_z': {'out_channels': 1, 'num_conv': 2},
            'dim': {'out_channels': 3, 'num_conv': 2},
            'rot': {'out_channels': 2, 'num_conv': 2},
            'iou': {'out_channels': 1, 'num_conv': 2},
        }
    RECTIFIER: [0.68, 0.71, 0.65, 0.5, 0.6, 0.67, 0.45, 0.4, 0.46, 0.5, 0.5]
    TARGET_ASSIGNER_CONFIG:
        FEATURE_MAP_STRIDE: 8
        NUM_MAX_OBJS: 500
        GAUSSIAN_OVERLAP: 0.1
        MIN_RADIUS: 2

    LOSS_CONFIG:
        LOSS_WEIGHTS: {
            'cls_weight': 1.0,
            'loc_weight': 2.0,
            'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
        }

    POST_PROCESSING:
        SCORE_THRESH: 0.5
        POST_CENTER_LIMIT_RANGE: [0.0, -40, -2, 150.4, 40, 4]
        MAX_OBJ_PER_SAMPLE: 500
        NMS_CONFIG:
            NMS_TYPE: nms_gpu

NMS_THRESH: [0.8, 0.55, 0.55] #0.7

            NMS_THRESH: [0.5, 0.3, 0.3, 0.3, 0.3, 0.5, 0.5, 0.5, 0.4, 0.3, 0.3]

NMS_PRE_MAXSIZE: [2048, 1024, 1024] #[4096]

            NMS_PRE_MAXSIZE: [2048, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024]

NMS_POST_MAXSIZE: [200, 150, 150] #500

            NMS_POST_MAXSIZE: [500, 300, 400, 400, 200, 200, 200, 100, 100, 100, 100]

POST_PROCESSING:
    RECALL_THRESH_LIST: [0.5, 0.3, 0.3, 0.3, 0.3, 0.5, 0.5, 0.5, 0.4, 0.3, 0.3]

    EVAL_METRIC: kitti

OPTIMIZATION:
BATCH_SIZE_PER_GPU: 26
NUM_EPOCHS: 50

OPTIMIZER: adam_onecycle
LR: 0.003
WEIGHT_DECAY: 0.01
MOMENTUM: 0.9

MOMS: [0.95, 0.85]
PCT_START: 0.4
DIV_FACTOR: 10
DECAY_STEP_LIST: [35, 45]
LR_DECAY: 0.1
LR_CLIP: 0.0000001

LR_WARMUP: False
WARMUP_EPOCH: 1

GRAD_NORM_CLIP: 10

custom_dataset.yaml:DATASET: 'At128Dataset'
DATA_PATH: '/lpai/volumes/lpai-autopilot-root/autopilot-cloud/lidar_at128_dataset_1W/autopilot-cloud/lidar_at128_dataset_1W'

POINT_CLOUD_RANGE: [0.0, -40, -2, 150.4, 40, 4]

DATA_SPLIT: {
'train': train,
'test': val
}

INFO_PATH: {
'train': [at128_infos_train.pkl],
'test': [at128_infos_val.pkl],
}

BALANCED_RESAMPLING: True

GET_ITEM_LIST: ["points"]

DATA_AUGMENTOR:
DISABLE_AUG_LIST: ['placeholder']
AUG_CONFIG_LIST:
- NAME: gt_sampling
DB_INFO_PATH:
- at128_dbinfos_train.pkl
USE_SHARED_MEMORY: False
PREPARE: {
filter_by_min_points: ['car:12', 'pedestrian:8', 'cyclist:10', 'tricyclist:15', 'bus:40', 'truck:30', 'special_vehicle:35',
'traffic_cone:10', 'small_obstacle:7', 'traffic_facilities:11', 'other:16']
}

      SAMPLE_GROUPS: ['car:20', 'pedestrian:8', 'cyclist:10', 'tricyclist:15', 'bus:40', 'truck:30', 'special_vehicle:35',
                                'traffic_cone:10', 'small_obstacle:7', 'traffic_facilities:11', 'other:16']
      NUM_POINT_FEATURES: 4
      DATABASE_WITH_FAKELIDAR: False
      REMOVE_EXTRA_WIDTH: [0.0, 0.0, 0.0]
      LIMIT_WHOLE_SCENE: True

    - NAME: random_world_flip
      ALONG_AXIS_LIST: ['x']

    - NAME: random_world_rotation
      WORLD_ROT_ANGLE: [-0.78539816, 0.78539816]

    - NAME: random_world_scaling
      WORLD_SCALE_RANGE: [0.95, 1.05]

POINT_FEATURE_ENCODING: {
encoding_type: absolute_coordinates_encoding,
used_feature_list: ['x', 'y', 'z', 'intensity'],
src_feature_list: ['x', 'y', 'z', 'intensity'],
}

DATA_PROCESSOR:
- NAME: mask_points_and_boxes_outside_range
REMOVE_OUTSIDE_BOXES: True

- NAME: shuffle_points
  SHUFFLE_ENABLED: {
    'train': True,
    'test': False
  }

- NAME: transform_points_to_voxels
  VOXEL_SIZE: [0.1, 0.1, 0.15]
  MAX_POINTS_PER_VOXEL: 20
  MAX_NUMBER_OF_VOXELS: {
    'train': 500000,
    'test': 400000
  }`

If possible, feel free to add me on WeChat or contact me through email. Let's discuss together.
Wechat: lyb543918165
Email: [email protected]

Traceback (most recent call last):
File "train.py", line 231, in
main()
File "train.py", line 200, in main
cfg=cfg
File "dev0510/OpenPCDet/tools/train_utils/train_utils.py", line 194, in train_model
use_amp=use_amp
File "dev0510/OpenPCDet/tools/train_utils/train_utils.py", line 56, in train_one_epoch
loss, tb_dict, disp_dict = model_func(model, batch)
File "../pcdet/models/init.py", line 44, in model_func
ret_dict, tb_dict, disp_dict = model(batch_dict)
File "miniconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/miniconda3/envs/py37/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 1040, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "miniconda3/envs/py37/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 1000, in _run_ddp_forward
return module_to_run(*inputs[0], **kwargs[0])
File "miniconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "../pcdet/models/detectors/voxelnext.py", line 11, in forward
batch_dict = cur_module(batch_dict)
File "miniconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "../pcdet/models/dense_heads/voxelnext_head.py", line 535, in forward
data_dict['gt_boxes'], num_voxels, spatial_indices, spatial_shape
File "../pcdet/models/dense_heads/voxelnext_head.py", line 160, in assign_targets
min_radius=target_assigner_cfg.MIN_RADIUS,
File "../pcdet/models/dense_heads/voxelnext_head.py", line 225, in assign_target_of_single_head
inds[k] = distance.argmin()
IndexError: argmin(): Expected reduction dim to be specified for input.numel() == 0.

Hi,thanks for your great work,when training my own dataset using my own data augmentation,the ERROR always occurs randomly after training serveral epochs , so i wonder how it works and whether made by my own dataset and my own data augmentation.

Can someone provide a website for pre-training models that don't need to be downloaded from google, thank you

I am trying to start inference on Argoverse2 dataset via the next command:

python test.py --cfg_file cfgs/argo2_models/cbgs_voxel01_voxelnext.yam --ckpt ../../VoxelNeXt_Argo2.pth --set DATA_CONFIG.DATA_SPLIT.test val

However, there is an error while loading checkpoint:

KeyError: 'model_state'

This code produces as error:

def load_params_from_file(self, filename, logger, to_cpu=False, pre_trained_path=None):
    if not os.path.isfile(filename):
        raise FileNotFoundError

    logger.info('==> Loading parameters from checkpoint %s to %s' % (filename, 'CPU' if to_cpu else 'GPU'))
    loc_type = torch.device('cpu') if to_cpu else None
    checkpoint = torch.load(filename, map_location=loc_type)
    model_state_disk = checkpoint['model_state']

VoxelNeXt_Argo2.pth is the file, downloaded from your website link.
What am I doing wrong?

I have followed all the installation instructions but this is the error I get when trying to run it:

/content/drive/.shortcut-targets-by-id/1JbE4k9qNvUlpfTSZhyxGbIIgEZCGZ7Ny/18744_Intermediate_Demo/VoxelNextShruti/VoxelNeXt/tools

• NGPUS=1

• PY_ARGS='--cfg_file cfgs/nuscenes_models/.yaml --ckpt /content/drive/MyDrive/18744_Intermediate_Demo/VoxelNeXtShruti/voxelnext_nuscenes_kernel1.pth'

• python -m torch.distributed.launch --nproc_per_node=1 test.py --launcher pytorch --cfg_file cfgs/nuscenes_models/.yaml --ckpt /content/drive/MyDrive/18744_Intermediate_Demo/VoxelNeXtShruti/voxelnext_nuscenes_kernel1.pth
  /usr/local/lib/python3.9/dist-packages/torch/distributed/launch.py:181: FutureWarning: The module torch.distributed.launch is deprecated
  and will be removed in future. Use torchrun.
  Note that --use-env is set by default in torchrun.
  If your script expects --local-rank argument to be set, please
  change it to read from os.environ['LOCAL_RANK'] instead. See
  https://pytorch.org/docs/stable/distributed.html#launch-utility for
  further instructions

  warnings.warn(
  usage: test.py
  [-h]
  [--cfg_file CFG_FILE]
  [--batch_size BATCH_SIZE]
  [--workers WORKERS]
  [--extra_tag EXTRA_TAG]
  [--ckpt CKPT]
  [--pretrained_model PRETRAINED_MODEL]
  [--launcher {none,pytorch,slurm}]
  [--tcp_port TCP_PORT]
  [--local_rank LOCAL_RANK]
  [--set ...]
  [--max_waiting_mins MAX_WAITING_MINS]
  [--start_epoch START_EPOCH]
  [--eval_tag EVAL_TAG]
  [--eval_all]
  [--ckpt_dir CKPT_DIR]
  [--save_to_file]
  [--infer_time]
  test.py: error: unrecognized arguments: --local-rank=0
  ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 2) local_rank: 0 (pid: 19545) of binary: /usr/bin/python3
  Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
  return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
  exec(code, run_globals)
  File "/usr/local/lib/python3.9/dist-packages/torch/distributed/launch.py", line 196, in
  main()
  File "/usr/local/lib/python3.9/dist-packages/torch/distributed/launch.py", line 192, in main
  launch(args)
  File "/usr/local/lib/python3.9/dist-packages/torch/distributed/launch.py", line 177, in launch
  run(args)
  File "/usr/local/lib/python3.9/dist-packages/torch/distributed/run.py", line 785, in run
  elastic_launch(
  File "/usr/local/lib/python3.9/dist-packages/torch/distributed/launcher/api.py", line 134, in call
  return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/lib/python3.9/dist-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
  raise ChildFailedError(
  torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
  ============================================================
  test.py FAILED

Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time : 2023-04-09_21:34:55
host : 8d5d35163cb5
rank : 0 (local_rank: 0)
exitcode : 2 (pid: 19545)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Any guidance would be helpful

Hi, thank you for you sharing the great work!
Can you send me the nuscene and waymo test set model?
My e-mail is : [email protected]
Thanks a lot!

can you share pretrained model for voxelnext-2d on nuscenes?

Hi, thanks again for publishing the code for your work!

The waymo_dataset.yaml config file used by the Waymo models have SAMPLED_INTERVAL for train split set to 5, i.e. 20% of Waymo. However, the reported results are included in the 100% Waymo table in OpenPCDet repo.

Given that the model is only trained for 12 epochs, I assume the SAMPLED_INTERVAL needs to be set to 1, i.e. 100% Waymo, to reproduce the results?

Hi, thanks for open sourcing your work.

I've been trying to run inference (demo.py) on nuscenes and kitti data but the result doesn't look right. Is there another way or do i need to transform the dataset?

python3 demo.py --cfg_file cfgs/nuscenes_models/cbgs_voxel0075_voxelnext.yaml --ckpt ../checkpoints/voxelnext_nuscenes_kernel1.pth --data_path ~/sd_data/data/nuscenes/v1.0-mini/samples/LIDAR_TOP/n008-2018-08-30-15-16-55-0400__LIDAR_TOP__1535657118649480.pcd.bin 

I get an assertion error

File "../OpenPCDet/pcdet/datasets/processor/point_feature_encoder.py", line 50, in absolute_coordinates_encoding
    assert points.shape[-1] == len(self.src_feature_list)

To get around it i just changed the line 48 in demo.py to comply with the model feature list src_feature_list: ['x', 'y', 'z', 'intensity', 'timestamp'],

points = np.fromfile(self.sample_file_list[index], dtype=np.float32).reshape(-1, 4)
# to
points = np.fromfile(self.sample_file_list[index], dtype=np.float32).reshape(-1, 5)

the output then looks like

Fig1: nuscenes results

Ran it on KITTI and it looks reasonable but with a lot of false positives and overlapping boxes, is that expected?
Have to modify line 48 in demo.py again as

points = np.fromfile(self.sample_file_list[index], dtype=np.float32).reshape(-1, 4)
points = np.c_[points, np.zeros(points.shape[0])]

Fig2: KITTI results

Need some help to get this running on nuscenes, or could you point me in a direction/code/doc i need to look at.

Hi, when I do bash scripts/dist_train.sh 1 --cfg_file cfgs/nuscenes_models/cbgs_voxel0075_voxelnext.yaml, some question appear, If you can help me, very appreciate!

• NGPUS=1

• PY_ARGS='--cfg_file cfgs/nuscenes_models/cbgs_voxel0075_voxelnext.yaml'

• true

• PORT=13654
  ++ nc -z 127.0.0.1 13654
  ++ echo 127

• status=127

• '[' 127 '!=' 0 ']'

• break

• echo 13654
  13654

• python -m torch.distributed.launch --nproc_per_node=1 train.py --launcher pytorch --cfg_file cfgs/nuscenes_models/cbgs_voxel0075_voxelnext.yaml
  /root/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
  and will be removed in future. Use torchrun.
  Note that --use_env is set by default in torchrun.
  If your script expects --local_rank argument to be set, please
  change it to read from os.environ['LOCAL_RANK'] instead. See
  https://pytorch.org/docs/stable/distributed.html#launch-utility for
  further instructions

  warnings.warn(
  Traceback (most recent call last):
  File "train.py", line 7, in
  from test import repeat_eval_ckpt
  File "/root/VoxelNeXt/tools/test.py", line 14, in
  from eval_utils import eval_utils
  File "/root/VoxelNeXt/tools/eval_utils/eval_utils.py", line 8, in
  from pcdet.models import load_data_to_gpu
  File "/root/VoxelNeXt/tools/../pcdet/init.py", line 4, in
  from .version import version
  ModuleNotFoundError: No module named 'pcdet.version'
  ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 31174) of binary: /root/miniconda3/bin/python
  Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
  return _run_code(code, main_globals, None,
  File "/root/miniconda3/lib/python3.8/runpy.py", line 87, in _run_code
  exec(code, run_globals)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in
  main()
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
  launch(args)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
  run(args)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/run.py", line 710, in run
  elastic_launch(
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in call
  return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
  raise ChildFailedError(
  torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
  ============================================================
  train.py FAILED