Starting Pre-Training with gt traversals as input for decoder
Global seed set to 0
2023-12-19 12:59:32,159 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/folder_builder.py:20} Building experiment folders...
2023-12-19 12:59:32,159 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/folder_builder.py:22} Experimental folder: /home/mh/code/nuplan/exp/exp/training_gc_pgp_model/training_gc_pgp_model/2023.12.19.12.59.31
2023-12-19 12:59:32,159 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/worker_pool_builder.py:19} Building WorkerPool...
2023-12-19 12:59:32,160 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/utils/multithreading/worker_ray.py:78} Starting ray local!
2023-12-19 12:59:33,686 INFO worker.py:1664 -- Started a local Ray instance. View the dashboard at 35.3.215.205:8265
2023-12-19 12:59:34,153 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/utils/multithreading/worker_pool.py:101} Worker: RayDistributed
2023-12-19 12:59:34,155 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/utils/multithreading/worker_pool.py:102} Number of nodes: 1
Number of CPUs per node: 32
Number of GPUs per node: 1
Number of threads across all nodes: 32
2023-12-19 12:59:34,155 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/worker_pool_builder.py:27} Building WorkerPool...DONE!
2023-12-19 12:59:34,155 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/training/experiments/training.py:41} Building training engine...
2023-12-19 12:59:34,155 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/model_builder.py:18} Building TorchModuleWrapper...
2023-12-19 12:59:34,299 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/model_builder.py:21} Building TorchModuleWrapper...DONE!
2023-12-19 12:59:34,299 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/splitter_builder.py:18} Building Splitter...
2023-12-19 12:59:34,675 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/splitter_builder.py:21} Building Splitter...DONE!
2023-12-19 12:59:34,675 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/data_augmentation_builder.py:19} Building augmentors...
2023-12-19 12:59:34,685 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/data_augmentation_builder.py:28} Building augmentors...DONE!
2023-12-19 12:59:34,686 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/scenario_building_builder.py:18} Building AbstractScenarioBuilder...
2023-12-19 12:59:34,737 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/scenario_building_builder.py:21} Building AbstractScenarioBuilder...DONE!
2023-12-19 12:59:34,737 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/scenario_filter_builder.py:35} Building ScenarioFilter...
2023-12-19 12:59:34,738 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/scenario_filter_builder.py:44} Building ScenarioFilter...DONE!
Ray objects: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 32/32 [02:11<00:00, 4.12s/it]
2023-12-19 13:01:49,405 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/scenario_builder.py:171} Extracted 177435 scenarios for training
2023-12-19 13:01:49,408 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:258} WORLD_SIZE was not set.
2023-12-19 13:01:49,408 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:266} PytorchLightning Trainer gpus was set to -1, finding number of GPUs used from torch.cuda.device_count().
2023-12-19 13:01:49,408 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:277} Number of gpus found to be in use: 1
2023-12-19 13:01:49,408 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:114} World size: 1
2023-12-19 13:01:49,408 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:115} Learning rate before: 0.0001
2023-12-19 13:01:49,408 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:119} Scaling method: Equal Variance
2023-12-19 13:01:49,408 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:141} Betas after scaling: [0.9, 0.999]
2023-12-19 13:01:49,408 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:143} Learning rate after scaling: 0.0001
2023-12-19 13:01:49,487 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:172} Updating Learning Rate Scheduler Config...
2023-12-19 13:01:49,487 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:258} WORLD_SIZE was not set.
2023-12-19 13:01:49,487 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:266} PytorchLightning Trainer gpus was set to -1, finding number of GPUs used from torch.cuda.device_count().
2023-12-19 13:01:49,487 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:277} Number of gpus found to be in use: 1
2023-12-19 13:01:49,487 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:199} Updating torch.optim.lr_scheduler.MultiStepLR in ddp setting is not yet supported. Learning rate scheduler config will not be updated.
2023-12-19 13:01:49,487 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:245} Optimizer and LR Scheduler configs updated according to ddp strategy.
2023-12-19 13:01:49,494 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/training_callback_builder.py:19} Building callbacks...
Error executing job with overrides: ['seed=0', 'py_func=train', '+training=training_gc_pgp_model', 'job_name=training_gc_pgp_model', 'scenario_builder=nuplan', 'scenario_filter.num_scenarios_per_type=4000', 'cache.cache_path=/home/mh/code/nuplan/exp/mh/cache', 'cache.use_cache_without_dataset=False', 'callbacks.visualization_callback.pixel_size=0.25', '+callbacks.multimodal_visualization_callback.pixel_size=0.25', 'lightning.trainer.params.max_epochs=20', 'lightning.trainer.params.max_time=null', 'data_loader.params.batch_size=32', 'optimizer.lr=1e-4', 'lr_scheduler=multistep_lr', 'lr_scheduler.milestones=[40,50,55]', 'lr_scheduler.gamma=0.5', 'model.encoder.use_red_light_feature=TRUE', 'model.aggregator.use_route_mask=FALSE', 'model.aggregator.hard_masking=FALSE', 'model.aggregator.pre_train=true']
Traceback (most recent call last):
File "/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/run_training.py", line 89, in
main()
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/main.py", line 49, in decorated_main
_run_hydra(
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/utils.py", line 367, in _run_hydra
run_and_report(
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
raise ex
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
return func()
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/utils.py", line 368, in
lambda: hydra.run(
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 110, in run
_ = ret.return_value
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/core/utils.py", line 233, in return_value
raise self._return_value
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/core/utils.py", line 160, in run_job
ret.return_value = task_function(task_cfg)
File "/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/run_training.py", line 59, in main
engine = build_training_engine(cfg, worker)
File "/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/training/experiments/training.py", line 60, in build_training_engine
trainer = build_trainer(cfg)
File "/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/training_builder.py", line 109, in build_trainer
callbacks = build_callbacks(cfg)
File "/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/training_callback_builder.py", line 25, in build_callbacks
validate_type(callback, pl.Callback)
File "/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_type.py", line 32, in validate_type
assert isinstance(
AssertionError: Class to be of type <class 'pytorch_lightning.callbacks.base.Callback'>, but is <class 'omegaconf.dictconfig.DictConfig'>!
Starting Training with aggregator traversals as input for decoder
Global seed set to 0
2023-12-19 13:01:56,346 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/folder_builder.py:20} Building experiment folders...
2023-12-19 13:01:56,346 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/folder_builder.py:22} Experimental folder: /home/mh/code/nuplan/exp/exp/training_gc_pgp_model/training_gc_pgp_model/2023.12.19.13.01.55
2023-12-19 13:01:56,347 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/worker_pool_builder.py:19} Building WorkerPool...
2023-12-19 13:01:56,347 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/utils/multithreading/worker_ray.py:78} Starting ray local!
2023-12-19 13:01:57,778 INFO worker.py:1664 -- Started a local Ray instance. View the dashboard at 35.3.215.205:8265
2023-12-19 13:01:58,236 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/utils/multithreading/worker_pool.py:101} Worker: RayDistributed
2023-12-19 13:01:58,237 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/utils/multithreading/worker_pool.py:102} Number of nodes: 1
Number of CPUs per node: 32
Number of GPUs per node: 1
Number of threads across all nodes: 32
2023-12-19 13:01:58,237 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/worker_pool_builder.py:27} Building WorkerPool...DONE!
2023-12-19 13:01:58,237 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/training/experiments/training.py:41} Building training engine...
2023-12-19 13:01:58,237 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/model_builder.py:18} Building TorchModuleWrapper...
2023-12-19 13:01:58,376 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/model_builder.py:21} Building TorchModuleWrapper...DONE!
2023-12-19 13:01:58,376 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/splitter_builder.py:18} Building Splitter...
2023-12-19 13:01:58,760 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/splitter_builder.py:21} Building Splitter...DONE!
2023-12-19 13:01:58,760 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/data_augmentation_builder.py:19} Building augmentors...
2023-12-19 13:01:58,770 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/data_augmentation_builder.py:28} Building augmentors...DONE!
2023-12-19 13:01:58,770 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/scenario_building_builder.py:18} Building AbstractScenarioBuilder...
2023-12-19 13:01:58,821 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/scenario_building_builder.py:21} Building AbstractScenarioBuilder...DONE!
2023-12-19 13:01:58,821 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/scenario_filter_builder.py:35} Building ScenarioFilter...
2023-12-19 13:01:58,821 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/scenario_filter_builder.py:44} Building ScenarioFilter...DONE!
Ray objects: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 32/32 [02:10<00:00, 4.09s/it]
2023-12-19 13:04:12,689 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/scenario_builder.py:171} Extracted 177435 scenarios for training
2023-12-19 13:04:12,692 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:258} WORLD_SIZE was not set.
2023-12-19 13:04:12,692 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:266} PytorchLightning Trainer gpus was set to -1, finding number of GPUs used from torch.cuda.device_count().
2023-12-19 13:04:12,692 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:277} Number of gpus found to be in use: 1
2023-12-19 13:04:12,692 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:114} World size: 1
2023-12-19 13:04:12,692 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:115} Learning rate before: 0.0001
2023-12-19 13:04:12,692 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:119} Scaling method: Equal Variance
2023-12-19 13:04:12,693 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:141} Betas after scaling: [0.9, 0.999]
2023-12-19 13:04:12,693 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:143} Learning rate after scaling: 0.0001
2023-12-19 13:04:12,770 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:172} Updating Learning Rate Scheduler Config...
2023-12-19 13:04:12,770 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:258} WORLD_SIZE was not set.
2023-12-19 13:04:12,770 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:266} PytorchLightning Trainer gpus was set to -1, finding number of GPUs used from torch.cuda.device_count().
2023-12-19 13:04:12,770 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:277} Number of gpus found to be in use: 1
2023-12-19 13:04:12,770 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:199} Updating torch.optim.lr_scheduler.MultiStepLR in ddp setting is not yet supported. Learning rate scheduler config will not be updated.
2023-12-19 13:04:12,771 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_config.py:245} Optimizer and LR Scheduler configs updated according to ddp strategy.
2023-12-19 13:04:12,777 INFO {/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/training_callback_builder.py:19} Building callbacks...
Error executing job with overrides: ['seed=0', 'py_func=train', '+training=training_gc_pgp_model', 'job_name=training_gc_pgp_model', 'scenario_builder=nuplan', 'scenario_filter.num_scenarios_per_type=4000', 'cache.cache_path=/home/mh/code/nuplan/exp/mh/cache', 'cache.use_cache_without_dataset=False', 'callbacks.visualization_callback.pixel_size=0.25', '+callbacks.multimodal_visualization_callback.pixel_size=0.25', 'lightning.trainer.params.max_epochs=90', 'lightning.trainer.params.max_time=null', 'lightning.trainer.checkpoint.resume_training=true', 'data_loader.params.batch_size=32', 'optimizer.lr=1e-4', 'lr_scheduler=multistep_lr', 'model.encoder.use_red_light_feature=TRUE', 'model.aggregator.use_route_mask=FALSE', 'model.aggregator.hard_masking=FALSE', 'model.aggregator.pre_train=false']
Traceback (most recent call last):
File "/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/run_training.py", line 89, in
main()
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/main.py", line 49, in decorated_main
_run_hydra(
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/utils.py", line 367, in _run_hydra
run_and_report(
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
raise ex
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
return func()
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/utils.py", line 368, in
lambda: hydra.run(
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 110, in run
_ = ret.return_value
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/core/utils.py", line 233, in return_value
raise self._return_value
File "/home/mh/miniconda3/envs/nuplan/lib/python3.9/site-packages/hydra/core/utils.py", line 160, in run_job
ret.return_value = task_function(task_cfg)
File "/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/run_training.py", line 59, in main
engine = build_training_engine(cfg, worker)
File "/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/training/experiments/training.py", line 60, in build_training_engine
trainer = build_trainer(cfg)
File "/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/training_builder.py", line 109, in build_trainer
callbacks = build_callbacks(cfg)
File "/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/training_callback_builder.py", line 25, in build_callbacks
validate_type(callback, pl.Callback)
File "/home/mh/code/nuplan/nuplan-devkit/nuplan/planning/script/builders/utils/utils_type.py", line 32, in validate_type
assert isinstance(
AssertionError: Class to be of type <class 'pytorch_lightning.callbacks.base.Callback'>, but is <class 'omegaconf.dictconfig.DictConfig'>!