autonomousvision / navsim Goto Github PK

When I run ./run_cv_pdm_score_evaluation.sh, there is a bug when loading log file. I use the pickle files you provide.

Hi, may I ask how large the training set is and when will it be released?

Hello, while attempting to run transfuser (using ./run_transfuser_training.sh), I am getting the following error :

Traceback (most recent call last):
  File "/mnt/disks/data/sim2real/CVPR-challenge/navsim/navsim/navsim/planning/script/run_training.py", line 119, in main
    trainer.fit(
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 544, in fit
    call._call_and_handle_interrupt(
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 105, in launch
    return function(*args, **kwargs)
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 580, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 987, in _run
    results = self._run_stage()
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1033, in _run_stage
    self.fit_loop.run()
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 205, in run
    self.advance()
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 363, in advance
    self.epoch_loop.run(self._data_fetcher)
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 140, in run
    self.advance(data_fetcher)
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 212, in advance
    batch, _, __ = next(data_fetcher)
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/pytorch_lightning/loops/fetchers.py", line 133, in __next__
    batch = super().__next__()
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/pytorch_lightning/loops/fetchers.py", line 60, in __next__
    batch = next(self.iterator)
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/pytorch_lightning/utilities/combined_loader.py", line 341, in __next__
    out = next(self._iterator)
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/pytorch_lightning/utilities/combined_loader.py", line 78, in __next__
    out[i] = next(self.iterators[i])
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 635, in __next__
    data = self._next_data()
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 679, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 56, in fetch
    return self.collate_fn(data)
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 267, in default_collate
    return collate(batch, collate_fn_map=default_collate_fn_map)
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 142, in collate
    return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed]  # Backwards compatibility.
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 142, in <listcomp>
    return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed]  # Backwards compatibility.
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 127, in collate
    return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 127, in <dictcomp>
    return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 119, in collate
    return collate_fn_map[elem_type](batch, collate_fn_map=collate_fn_map)
  File "/opt/conda/envs/navsim/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 164, in collate_tensor_fn
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [3, 256, 1024] at entry 0 and [1, 256, 1024] at entry 2

I was getting an error stating : RuntimeError: Trying to resize storage that is not resizable, when i had number of workers as 4 with a prefetch factor 2, so i modified it to 0 and none. Is there a fix you could suggest for this?

Hi everyone, I notice that there was an update in the navsim repository including dataset, new baseline, visualizations, thanks for the great work. I have a question about the navtrain dataset, when I downloaded it, it seemed there is no current_split_5 and history_split_5, which included in the download script(as shown below). Did I miss something or there is no such data? And there is no corrresponding pkl file, which means we can't use it for our training, will it be provided later?

Another Question is that privously I split the mini set into train, val, test for small testing, and upload the trained model on the warmup track. But I notice that you said that the warmup_test_e2e is a scene filter for the mini split. So can I ask which part of the mini set you use for the warmup track?

Thanks a lot for answering my question!

Hi! Thank you for this incredible work! I'd like to clarify the proper method of setting up the necessary environment variables according to your README documentation located here: https://github.com/autonomousvision/navsim/blob/main/docs/install.md#2-install-the-navsim-devkit.

After following your guide to prepare the dataset and code, I have reached the stage where I need to define the paths for the environment variables. However, I am uncertain about the specific directories to use for each of the following:

export NAVSIM_DEVKIT_ROOT=/path/to/navsim/devkit  # Should this be the path to the root directory of my navsim installation?
export NUPLAN_EXP_ROOT=/path/to/navsim/exp # Is this the path pointing to the 'navsim_logs' folder that I've downloaded?
export NUPLAN_MAPS_ROOT=/path/to/nuplan/maps # The path to the downloaded 'nuplan-maps-v1.0' folder?
export OPENSCENE_DATA_ROOT=/path/to/openscene # Do I need to acquire a separate dataset specifically for this variable?

Your guidance in specifying these paths would be greatly appreciated!

Could someone please explain why the heading angle needs to be set to negative in the function convert_absolute_to_relative_se2_array？

Hi! I find this line of code calculating the Driving Direction Compliance metric a bit confusing:

In this file cache file is written like this
cache.cache_path=/home/aah1si/openscene/exp/public_test_metric_cache
I think it should use exported values instead of /home/aah1si
So wouldn't this be better?
cache.cache_path=$NAVSIM_EXP_ROOT/metric_cache

Currently, I am using ego2global transformation matrix to warp LiDAR points’ in the history to the current key frame. If this matrix isn’t available in the final test set, it might be difficult to do so. Can you provide the transformation matrix between different frames for the test set?

Hi, I would like to download the navtrain datasets by running the code download_navtrain.sh.
However, I got following outputs while downloading it.

mv: cannot move 'history_split_1/2021.06.09.14.03.17_veh-12_00159_00283' to 'trainval_sensor_blobs/trainval/2021.06.09.14.03.17_veh-12_00159_00283': Directory not empty

Note: This output was obtained for all the data in the history_split_1 directory. The one shown is just an example.

Upon checking the download logs, it seems that the data included in history_split might already be obtained when downloading the current_split.

I want to use NAVSIM with my custom dataset that provides images, LiDAR point cloud, 3D bouding box annotation and annotated HD map. Could you give me some instructions on adapting NAVSIM to my own custom dataset?

Hi, I've downloaded the private test set downloaded from https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_metadata_private_test_e2e.tgz and https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_private_test_e2e.tgz according to your script. It seems that this private set is mainly based on the previous test set from https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_metadata_test.tgz?

The previous test set contains annotations like bboxes and HD map according to OpenScene. I feel like asking how can we maintain the integrity of the challenge if these data are visible to participants?

Hello, I would like to ask what I have no problem testing on the mini dataset, but it is wrong to test on the private_test_e2e dataset?

(navsim) tongyy@server:~/TYY/navsim/scripts$ ./evaluation/run_transfuer.sh
2024-05-23 15:49:00,634 INFO {/mnt/datadisk/tongyy/TYY/navsim/navsim/planning/script/builders/worker_pool_builder.py:20} Building WorkerPool...
2024-05-23 15:49:00,770 INFO {/mnt/datadisk/tongyy/TYY/navsim/navsim/planning/utils/multithreading/worker_ray_no_torch.py:51} Not using GPU in ray
2024-05-23 15:49:00,770 INFO {/mnt/datadisk/tongyy/TYY/navsim/navsim/planning/utils/multithreading/worker_ray_no_torch.py:77} Starting ray local!
2024-05-23 15:49:02,660 INFO worker.py:1749 -- Started a local Ray instance.
2024-05-23 15:49:03,740 INFO {/mnt/datadisk/tongyy/.conda/envs/navsim/lib/python3.9/site-packages/nuplan/planning/utils/multithreading/worker_pool.py:101} Worker: RayDistributedNoTorch
2024-05-23 15:49:03,740 INFO {/mnt/datadisk/tongyy/.conda/envs/navsim/lib/python3.9/site-packages/nuplan/planning/utils/multithreading/worker_pool.py:102} Number of nodes: 1
Number of CPUs per node: 136
Number of GPUs per node: 0
Number of threads across all nodes: 136
2024-05-23 15:49:03,740 INFO {/mnt/datadisk/tongyy/TYY/navsim/navsim/planning/script/builders/worker_pool_builder.py:31} Building WorkerPool...DONE!
Loading logs: 0%| | 0/1 [00:00<?, ?it/s]
Error executing job with overrides: ['agent=transfuser_agent', 'agent.checkpoint_path=/mnt/datadisk/tongyy/TYY/navsim/exp/training_transfuser_agent/2024.05.16.20.42.58/lightning_logs/version_0/checkpoints/epoch19.ckpt', 'experiment_name=transfuser_agent_eval', 'split=private_test_e2e']
Traceback (most recent call last):
File "/mnt/datadisk/tongyy/TYY/navsim/navsim/planning/script/run_pdm_score.py", line 46, in main
scene_loader = SceneLoader(
File "/mnt/datadisk/tongyy/TYY/navsim/navsim/common/dataloader.py", line 83, in init
self.scene_frames_dicts = filter_scenes(data_path, scene_filter)
File "/mnt/datadisk/tongyy/TYY/navsim/navsim/common/dataloader.py", line 50, in filter_scenes
and len(frame_list[scene_filter.num_history_frames - 1]["roadblock_ids"]) == 0
TypeError: object of type 'NoneType' has no len()

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Hi, when i move to the whole dataset and use run_dataset_caching to cache the trainval data, it takes almost 16 hours to cache all the data. For features, I only use ego_state and front images by now. I am not sure how long it takes when i move to all the images and all the lidar data. And it also means when i want to change my input, i will have to cache all the data again. So I want to ask are there anyway to accelerate the caching process, or are there anyway that i can directly start training instead of caching the data for more than one day (probably without losing too much performance)? Did I miss something? below is the script I use when cache the data:

by the way I have num_workers = 96, and also with GPU memory = 24GB * 2, are there anyway I can fully make use of it to acc the training process?

Thanks in advance

Is the 'classic' Transfuser or Transfuser++ implemented in the repo?

Hi! Just want to ask what's the coordinate system origin of the ego vehicle? Is it the center of the rear? or something else

So there is a trajectory stored in metric cache data. I am trying to look to the path it takes but I couldn't figure out where does it start ?
Is the start position initial_ego_state.center ?

When does the test server on huggingface restart? Or can I get a fair test score offline?

The script processes the mini set successfully but fails on test split with key error 'anns'.

Thanks for your great benchmark. Now I can run_transfuser.sh on miniset. I'd like to ask how to visualize planning results as videos for each scenario. Great thanks if you would help!