This github implements the system described in the paper:

Self-supervised Depth and Ego-motion Estimation for Monocular Thermal Video using Multi-spectral Consistency Loss

Ukcheol Shin, Kyunghyun Lee, Seokju Lee, In So Kweon

Robotics and Automation Letter 2022 & ICRA 2022

[PDF] [Project webpage] [Full paper] [Youtube]

This codebase was developed and tested with python 3.7, Pytorch 1.5.1, and CUDA 10.2 on Ubuntu 16.04.

conda env create --file environment.yml

See "scripts/run_prepare_vivid_data.sh".

For ViViD Raw dataset, download the dataset provided on the official website.

For our post-processed dataset and pre-trained models, you can download after fill out a simple survey.

We will send you an e-mail with a download link.

After download post-processed dataset, generate training/testing dataset

sh scripts/run_prepare_vivid_data.sh

The "scripts" folder provides several examples for training and testing.

You can train the depth and pose model on vivid dataset by running

sh scripts/train_vivid_resnet18_indoor.sh
sh scripts/train_vivid_resnet18_outdoor.sh

Then you can start a tensorboard session in this folder by

tensorboard --logdir=checkpoints/

and visualize the training progress by opening https://localhost:6006 on your browser.

You can evaluate depth and pose by running

sh scripts/test_vivid_indoor.sh
sh scripts/test_vivid_outdoor.sh

and visualize depth by running

sh scripts/run_inference.sh

You can comprehensively see the overall results by running

sh scripts/display_result.sh

Please cite the following paper if you use our work, parts of this code, and pre-processed dataset in your research.

@article{shin2021self,
  title={Self-Supervised Depth and Ego-Motion Estimation for Monocular Thermal Video Using Multi-Spectral Consistency Loss},
  author={Shin, Ukcheol and Lee, Kyunghyun and Lee, Seokju and Kweon, In So},
  journal={IEEE Robotics and Automation Letters},
  volume={7},
  number={2},
  pages={1103--1110},
  year={2021},
  publisher={IEEE}
}

• SfMLearner-Pytorch (CVPR 2017)
• SC-SfMLearner-Pytorch (NeurIPS 2019)