This is the official implementation of the paper Depth Estimation from Monocular Images and Sparse Radar Data. In this repo, we provide code for dataset preprocessing, training, and evaluation.
Some parts of the implementation are adapted from sparse-to-dense. We thank the authors for sharing their implementation.
-
Training and evaluation code.
-
Trained models.
-
Download instructions for the processed dataset.
-
Detailed documentation for the processed dataset.
-
Code and instructions to process data from the official nuScenes dataset.
git clone https://github.com/brade31919/radar_depth.git
cd radar_depthWe provide our processed files specifically for the RGB + Radar depth estimation task. The download and setup instructions are:
mkdir DATASET_PATH # Set the path you want to use on your own PC/cluster.
cd DATASET_PATH
wget https://data.vision.ee.ethz.ch/daid/NuscenesRadar/Nuscenes_depth.tar.gz
tar -zxcf Nuscenes_depth.tar.gzcd radar_depth # Go back to the project root
pip install -r requirements.txtIf you encounter error message like "ImportError: libSM.so.6: cannot open shared object file: No such file or directory" from cv2, you can try:
sudo apt-get install libsm6 libxrender1 libfontconfig1we put important path setting in config/config_nuscenes.py. You need to modify them to the paths you use on your own PC/cluster.
In line 14 and 18, please specify your PROJECT_ROOT and DATASET_ROOT
PROJECT_ROOT = "YOUR_PATH/radar_depth"
DATASET_ROOT = "DATASET_PATH"In line 53, please specify your EXPORT_PATH (the path you want to put our processed dataset).
EXPORT_ROOT = "YOUR_EXP_PATH"We provide some pretrained models. They are not the original models used to produce the numbers on the paper but they have similar performances (I lost the original checkpoints due to some cluster issue...).
Please download the pretrained models from here, and put them to pretrained/ folder so that the directory structue looks like this:
pretrained/
├── resnet18_latefusion.pth.tar
└── resnet18_multistage.pth.tarpython main.py \
--arch resnet18_latefusion \
--data nuscenes \
--modality rgbd \
--decoder upproj \
-j 12 \
--epochs 20 \
-b 16 \
--max-depth 80 \
--sparsifier radarTo make sure that the training process is stable, we'll initialize each stage from the reset18_latefusion model. If you want to skip the trainig of resnet18_latefusion, you can use our pre-trained models.
python main.py \
--arch resnet18_multistage_uncertainty_fixs \
--data nuscenes \
--modality rgbd \
--decoder upproj \
-j 12 \
--epochs 20 \
-b 8 \
--max-depth 80 \
--sparsifier radarHere we use batch size 8 (instead of 16). This allows us to train the model on cheaper GPU models such as GTX1080Ti, GTX2080Ti, etc., and the training process is more stable.
After the training process finished, you can evaluate the model by (replace the PATH_TO_CHECKPOINT with the path to checkpoint file you want to evaluate):
python main.py \
--evaluate PATH_TO_CHECKPOINT \
--data nuscenesPlease use the following citation format if you want to reference to our paper.
@InProceedings{radar:depth:20,
author = {Lin, Juan-Ting and Dai, Dengxin and {Van Gool}, Luc},
title = {Depth Estimation from Monocular Images and Sparse Radar Data},
booktitle = {International Conference on Intelligent Robots and Systems (IROS)},
year = {2020}
}
If you use the processed dataset, remember to cite the offical nuScenes dataset.
@article{nuscenes2019,
title={nuScenes: A multimodal dataset for autonomous driving},
author={Holger Caesar and Varun Bankiti and Alex H. Lang and Sourabh Vora and
Venice Erin Liong and Qiang Xu and Anush Krishnan and Yu Pan and
Giancarlo Baldan and Oscar Beijbom},
journal={arXiv preprint arXiv:1903.11027},
year={2019}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent