This repository contains the code to reproduce DenseTNT-Intent (Section V), presented in "Towards Trustworthy Multi-Modal Motion Prediction: Evaluation and Interpretability" by Sandra Carrasco, Sylwia Majchrowska, Joakim Johnander, Christoffer Peterson and David Fernández LLorca, presented at .. 2022.
@misc{MPCarrasco2022,
doi = {10.48550/ARXIV.2210.16144},
url = {https://arxiv.org/abs/2210.16144},
author = {Carrasco Limeros, Sandra and Majchrowska, Sylwia
and Johnander, Joakim and Petersson, Christoffer
and Sotelo, Miguel Ángel and Llorca, David Fernández},
title = {Towards Trustworthy Multi-Modal Motion Prediction: Evaluation and Interpretability},
publisher = {arXiv},
year = {2022}
}
Follow the original DenseTNT Repository to set up your environment.
You can download the model weights using this link.
OUTPUT_DIR=models.densetnt.12m; \
MODEL_PATH=models.densetnt.12m/model_save/model.16.bin; \
GPU_NUM=8; \
python src/run.py --argoverse --future_frame_num 30 \
--do_eval --data_dir_for_val val/data/ --output_dir ${OUTPUT_DIR} \
--log_dir log/dir/ --hidden_size 128 --eval_batch_size 64 --use_map \
--core_num 16 --use_centerline --mode_num 12 --distributed_training ${GPU_NUM} \
--other_params \
clustering semantic_lane direction l1_loss \
goals_2D enhance_global_graph subdivide goal_scoring laneGCN point_sub_graph \
lane_scoring complete_traj complete_traj-3 \
--eval_params optimization MRminFDE cnt_sample=9 opti_time=0.1 mask_lanes p=0.2
To perform the robustness analysis, you can remove clustering
from --other_params
and add mask_agents p=0.2
for an 80% recall in detecting dynamic agents, or mask_lanes p=0.2
for lanes. Use mask_agents_frames p=0.2
to perform the temporal analysis, masking 20% of the frames in interacting agents.
To visualize the scenes, add visualize
to --other_params
.
- This is the official implementation of the paper: DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets (ICCV 2021).
- DenseTNT v1.0 was released in November 1st, 2021.
- Updates:
- July 25th, 2022: Add detailed code comments.
Requires:
- Python ≥ 3.6
- PyTorch ≥ 1.6
pip install -r requirements.txt
The latest version of Argoverse requires Python ≥ 3.7
If using Python 3.6, you can install Argoverse v1.0
https://github.com/argoai/argoverse-api
Compile a .pyx file into a C file using Cython (already installed at step 1):
cd src/ && cython -a utils_cython.pyx && python setup.py build_ext --inplace && cd ../
Results on Argoverse motion forecasting validation set:
minADE | minFDE | Miss Rate | |
---|---|---|---|
DenseTNT w/ 100ms optimization (Miss Rate) | 0.80 | 1.27 | 7.0% |
DenseTNT w/ 100ms optimization (minFDE) | 0.73 | 1.05 | 9.8% |
DenseTNT w/ goal set predictor (Miss Rate) | 0.82 | 1.37 | 7.0% |
DenseTNT w/ goal set predictor (minFDE) | 0.75 | 1.05 | 9.7% |
A miss rate between 6.9% and 7.1% is considered reasonable. A minFDE between 1.05 and 1.06 is considered reasonable.
Suppose the training data of Argoverse motion forecasting is at ./train/data/
.
OUTPUT_DIR=models.densetnt.1; \
GPU_NUM=8; \
python src/run.py --argoverse --future_frame_num 30 \
--do_train --data_dir train/data/ --output_dir ${OUTPUT_DIR} \
--hidden_size 128 --train_batch_size 64 --use_map \
--core_num 16 --use_centerline --distributed_training ${GPU_NUM} \
--other_params \
semantic_lane direction l1_loss \
goals_2D enhance_global_graph subdivide goal_scoring laneGCN point_sub_graph \
lane_scoring complete_traj complete_traj-3 \
Training takes 20 minutes per epoch and 5 hours for the default 16 epochs on 8 × 2080Ti.
Suppose the validation data of Argoverse motion forecasting is at ./val/data/
.
-
Optimize Miss Rate:
- Add
--do_eval --eval_params optimization MRminFDE cnt_sample=9 opti_time=0.1
to the end of the training command.
- Add
-
Optimize minFDE:
- Add
--do_eval --eval_params optimization MRminFDE=0.0 cnt_sample=9 opti_time=0.1
to the end of the training command.
- Add
Compared with the optimization algorithm (default setting), the set predictor has similar performance but faster inference speed.
After training DenseTNT, suppose the model path is at models.densetnt.1/model_save/model.16.bin
. The command for training the set predictor is:
OUTPUT_DIR=models.densetnt.set_predict.1; \
MODEL_PATH=models.densetnt.1/model_save/model.16.bin; \
GPU_NUM=8; \
python src/run.py --argoverse --future_frame_num 30 \
--do_train --data_dir train/data/ --output_dir ${OUTPUT_DIR} \
--hidden_size 128 --train_batch_size 64 --use_map \
--core_num 16 --use_centerline --distributed_training ${GPU_NUM} \
--other_params \
semantic_lane direction l1_loss \
goals_2D enhance_global_graph subdivide goal_scoring laneGCN point_sub_graph \
lane_scoring complete_traj \
set_predict=6 set_predict-6 data_ratio_per_epoch=0.4 set_predict-topk=0 set_predict-one_encoder set_predict-MRratio=1.0 \
set_predict-train_recover=${MODEL_PATH} \
This training command optimizes Miss Rate. To optimize minFDE, change set_predict-MRratio=1.0
in the command to set_predict-MRratio=0.0
.
To evaluate the set predictor, just add --do_eval
to the end of this training command.
If you find our work useful for your research, please consider citing the paper:
@inproceedings{densetnt,
title={Densetnt: End-to-end trajectory prediction from dense goal sets},
author={Gu, Junru and Sun, Chen and Zhao, Hang},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages={15303--15312},
year={2021}
}