Comments (7)
yukang2017
commented on May 20, 2024
你好，感觉不是很正常。我用4卡V100，BATCH_SIZE_PER_GPU=4. 训练大概2天多点。我猜测可能训练速度瓶颈在dataloader的数据读取上。
from voxelnext.
dabi4256
commented on May 20, 2024
你好，感谢你的回复。
请问这个时间是实际跑完需要2天，还是程序给出的预估时间。 我代码还没有训练完，只是看到程序给出的时间是200个小时。
而且，我用双卡3090跟单卡3090的预估时间差别不大。
from voxelnext.
yukang2017
commented on May 20, 2024
你好，我预估时间和实际时间基本差不多。
from voxelnext.
learnuser1
commented on May 20, 2024
您好，我自己用完整的nuScenes数据集训练，显卡是两张3090， BATCH_SIZE_PER_GPU=8，用的是‘cbgs_voxel0075_voxelnext.yaml’配置文件。训练时间稳定的话大概是200个小时，这是正常的吗。
你好，这个自己训练的话效果跟原论文比的话怎么样。然后这个模型跟centerpoint比的话要好很多嘛
from voxelnext.
dabi4256
commented on May 20, 2024
您好，我自己用完整的nuScenes数据集训练，显卡是两张3090， BATCH_SIZE_PER_GPU=8，用的是‘cbgs_voxel0075_voxelnext.yaml’配置文件。训练时间稳定的话大概是200个小时，这是正常的吗。你好，这个自己训练的话效果跟原论文比的话怎么样。然后这个模型跟centerpoint比的话要好很多嘛
我的机器跑的很慢，20个epoch还没跑完。但是我测试了第10个epoch，mAP已经到了59.32了。CenterPoint我没跑过，但是论文里面写的是CenterPoint的mAP为 58.6.
from voxelnext.
learnuser1
commented on May 20, 2024
您好，我自己用完整的nuScenes数据集训练，显卡是两张3090， BATCH_SIZE_PER_GPU=8，用的是‘cbgs_voxel0075_voxelnext.yaml’配置文件。训练时间稳定的话大概是200个小时，这是正常的吗。你好，这个自己训练的话效果跟原论文比的话怎么样。然后这个模型跟centerpoint比的话要好很多嘛
我的机器跑的很慢，20个epoch还没跑完。但是我测试了第10个epoch，mAP已经到了59.32了。CenterPoint我没跑过，但是论文里面写的是CenterPoint的mAP为 58.6.
好的，感谢
from voxelnext.
rockywind
commented on May 20, 2024
@learnuser1 @yukang2017 @dabi4256 @yanwei-li 大家好
我跑多卡训练，会报这个错误。
Traceback (most recent call last):
File "train.py", line 246, in <module>
Traceback (most recent call last):
File "train.py", line 246, in <module>
main()
File "train.py", line 179, in main
main()
File "train.py", line 179, in main
model = nn.parallel.DistributedDataParallel(model, device_ids=[cfg.LOCAL_RANK % torch.cuda.device_count()])
File "/opt/conda/envs/VoxelNet/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 674, in __init__
model = nn.parallel.DistributedDataParallel(model, device_ids=[cfg.LOCAL_RANK % torch.cuda.device_count()])
File "/opt/conda/envs/VoxelNet/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 674, in __init__
_verify_param_shape_across_processes(self.process_group, parameters)
File "/opt/conda/envs/VoxelNet/lib/python3.8/site-packages/torch/distributed/utils.py", line 118, in _verify_param_shape_across_processes
_verify_param_shape_across_processes(self.process_group, parameters)
File "/opt/conda/envs/VoxelNet/lib/python3.8/site-packages/torch/distributed/utils.py", line 118, in _verify_param_shape_across_processes
return dist._verify_params_across_processes(process_group, tensors, logger)
torch.distributed.DistBackendError: return dist._verify_params_across_processes(process_group, tensors, logger)NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1275, internal error, NCCL version 2.14.3
ncclInternalError: Internal check failed.
Last error:
Duplicate GPU detected : rank 1 and rank 0 both on CUDA device 8a000
torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1275, internal error, NCCL version 2.14.3
ncclInternalError: Internal check failed.
Last error:
Duplicate GPU detected : rank 0 and rank 1 both on CUDA device 8a000
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1132223) of binary: /opt/conda/envs/VoxelNet/bin/python
Traceback (most recent call last):
File "/opt/conda/envs/VoxelNet/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/VoxelNet/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/VoxelNet/lib/python3.8/site-packages/torch/distributed/launch.py", line 196, in <module>
main()
File "/opt/conda/envs/VoxelNet/lib/python3.8/site-packages/torch/distributed/launch.py", line 192, in main
launch(args)
File "/opt/conda/envs/VoxelNet/lib/python3.8/site-packages/torch/distributed/launch.py", line 177, in launch
run(args)
File "/opt/conda/envs/VoxelNet/lib/python3.8/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/opt/conda/envs/VoxelNet/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/envs/VoxelNet/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
from voxelnext.
Related Issues (20)
- VoxelNeXT-2D HOT 1
- tracking HOT 3
- How to get the flops ?
- No module named 'pcdet.version' HOT 3
- IoU related loss HOT 1
- [Question] How to run a single inference on this model? HOT 2
- Question about the settings of training
- How to use the NuScenes dataset? Please!!! HOT 2
- Can you send me the nuscene and waymo test set model?
- Question about test results
- Can't reproduce the results on waymo
- FPS
- Question about NuScenes val set
- KeyError 'model_state' while loading checkpoint HOT 1
- Could not launch VoxelNeXt due to VoxelGenerator
- Details about the code HOT 1
- The result on nuscenes test set is not good
- 3D object detection from UAV based LiDAR point clouds
- Argoverse2 trainval infos
- About the deployment
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from voxelnext.