Official code link: https://github.com/Tsinghua-MARS-Lab/futr3d

This repo implements the paper FUTR3D: A Unified Sensor Fusion Framework for 3D Detection. Paper - project page

1. mmcv-full>=1.3.8, <=1.4.0
2. mmdet>=2.14.0, <=3.0.0
3. mmseg>=0.14.1, <=1.0.0
4. mmdetection3d, >=v0.17.0
5. nuscenes-devkit

• mmcv-full 1.4.0
• mmdet 2.14.0
• mmdet3d 0.17.1
• mmsegmentation 0.14.1

conda create -n open-mmlab python=3.8 -y
conda activate open-mmlab

pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html

pip install mmcv-full==1.4.0

pip install mmdet==2.14.0
pip install mmsegmentation==0.14.1

git clone https://github.com/open-mmlab/mmdetection3d.git
cd mmdetection3d
git checkout v0.17.1 # Other versions may not be compatible.
pip install -e .


git clone https://github.com/nacayu/FUTR3D

For cameras with Radar setting, you should generate a meta file or say .pkl file including Radar infos.

bash tools/create_data.sh nuscenes v1.0-trainval 4

For others, please follow the mmdet3d to process the data. https://mmdetection3d.readthedocs.io/en/stable/datasets/nuscenes_det.html

For example, If your GPU memory is about 10GB, to train small FUTR3D with Radar only on 2 GPUs, please use

bash tools/dist_train.sh plugin/futr3d/configs/cam_radar/res101_radar_small.py 2

or

Train full FUTR3D from official (RTX 3090: 24GB GPU) with Radar only on 2 GPUs, please use

bash tools/dist_train.sh plugin/futr3d/configs/cam_radar/res101_radar_large.py 2

For example, to evalaute FUTR3D with Radar-cam on 2 GPUs, please use

bash tools/dist_test.sh ./plugin/futr3d/configs/cam_radar/res101_radar_large.py ./ckpts/cam_res101_radar_900q.pth --eval bbox

You should first generate .pkl results using:

bash tools/dist_test.sh ./plugin/futr3d/configs/cam_radar/res101_radar_large.py ./ckpts/cam_res101_radar_900q.pth --format-only

to generate .pkl result files and run:

python tools/misc/visualize_results.py /path/to/your_config /path/to/your_config_file_of_config

The camera-only version of FUTR3D is the same as DETR3D. Please check DETR3D for detail implementation.

For the implementation, we rely heavily on MMCV, MMDetection, MMDetection3D, and DETR3D

1. DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries
2. MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries
3. For more projects on Autonomous Driving, check out our Visual-Centric Autonomous Driving (VCAD) project page webpage

@article{chen2022futr3d,
  title={FUTR3D: A Unified Sensor Fusion Framework for 3D Detection},
  author={Chen, Xuanyao and Zhang, Tianyuan and Wang, Yue and Wang, Yilun and Zhao, Hang},
  journal={arXiv preprint arXiv:2203.10642},
  year={2022}
}

Contact: Xuanyao Chen at: [email protected] or [email protected]