Giter Club home page Giter Club logo

ms2-multispectralstereodataset's Introduction

Multi-Spectral Stereo ($MS^2$) Outdoor Driving Dataset

This is the official github page of the $MS^2$ dataset described in the following paper.

This page provides a dataloader and simple python code for $MS^2$ dataset.

If you want to download the dataset and see the details, please visit the dataset page.

Deep Depth Estimation from Thermal Image

Ukcheol Shin, Jinsun Park, In So Kweon

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

[Paper] [Dataset page]

Updates

  • 2023.03.30: Open Github page.
  • 2023.05.30: Release $MS^2$ dataset, dataloader, and demo code.

$MS^2$ Dataset Specification

MS2 dataset provides:

  • (Synchronized) Stereo RGB images / Stereo NIR images / Stereo thermal images
  • (Synchronized) Stereo LiDAR scans / GPS/IMU navigation data
  • Projected depth map (in RGB, NIR, thermal image planes)
  • Odometry data (in RGB, NIR, thermal cameras, and LiDAR coordinates)

Usage

  1. Download the datasets and place them in 'MS2dataset' folder in the following structure:
MS2dataset
├── sync_data
│   ├── <Sequence Name1>
│   ├── <Sequence Name2>
│   ├── ...
│   └── <Sequence NameN>
├── proj_depth
│   ├── <Sequence Name1>
│   ├── <Sequence Name2>
│   ├── ...
│   └── <Sequence NameN>
└── odom
    ├── <Sequence Name1>
    ├── <Sequence Name2>
    ├── ...
    └── <Sequence NameN>
  1. We provide a simple python code (demo.py) along with a dataloader to take a look at the provided dataset. To run the code, you need any version of Pytorch library.
python demo.py --seq_name <Sequence Name> --modality rgb --data_format MonoDepth
python demo.py --seq_name <Sequence Name> --modality nir --data_format StereoMatch
python demo.py --seq_name <Sequence Name> --modality thr --data_format MultiViewImg

ms2-multispectralstereodataset's People

Contributors

ukcheolshin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

ms2-multispectralstereodataset's Issues

Upload dataset on google drive

Could you upload the dataset to Google Drive, please? There is some trouble when downloading the dataset vis Dropbox. Thank you very much!

How to get the extrinsic parameters between left rgb and left thr?

Thank you for your work!
According to your code:
ext_NIR2THR = np.concatenate([calib['R_nir2thr'], calib['T_nir2thr']*0.001], axis=1) # mm -> m scale conversion.
ext_NIR2RGB = np.concatenate([calib['R_nir2rgb'], calib['T_nir2rgb']*0.001], axis=1)
ext_THR2NIR = np.linalg.inv(np.concatenate([ext_NIR2THR, [[0,0,0,1]]],axis=0))
ext_THR2RGB = np.matmul(np.concatenate([ext_NIR2RGB, [[0,0,0, 1]]],axis=0), ext_THR2NIR)
ext_RGB2NIR = np.linalg.inv(np.concatenate([ext_NIR2RGB, [[0,0,0,1]]],axis=0))
ext_RGB2THR = np.linalg.inv(ext_THR2RGB)

I get the ext_RGB2THR, can it express extrinsic parameters from left rgb to left thr?

some files seems damaged

Hi, @UkcheolShin

Thank you for your work, your work provides valuable data for multimodal visual tasks.

After downloading and decompressing the compressed file, I found that most of the files were damaged. For example, an error occurred when I decompressed to nir/right/8756.png in _2021-08-06-10-59-33 (I re-downloaded the compressed file and the same error occurred at the same location).

Could you please check the status of each compressed package?

The list of files that failed to decompress is as follows:

image:
_2021-08-06-10-59-33.tar.bz2
_2021-08-06-11-23-45.tar.bz2
_2021-08-06-11-37-46.tar.bz2
_2021-08-06-16-19-00.tar.bz2
_2021-08-06-16-45-28.tar.bz2
_2021-08-06-16-59-13.tar.bz2
_2021-08-06-17-21-04.tar.bz2
_2021-08-06-17-44-55.tar.bz2
_2021-08-13-15-46-56.tar.bz2
_2021-08-13-16-14-48.tar.bz2
_2021-08-13-16-50-57.tar.bz2
_2021-08-13-17-06-04.tar.bz2
_2021-08-13-21-18-04.tar.bz2
_2021-08-13-21-36-10.tar.bz2
_2021-08-13-22-03-03.tar.bz2

proj_depth:
_2021-08-06-10-59-33.tar.bz2
_2021-08-06-11-23-45.tar.bz2
_2021-08-06-16-45-28.tar.bz2
_2021-08-06-17-44-55.tar.bz2
_2021-08-13-15-46-56.tar.bz2
_2021-08-13-16-08-46.tar.bz2
_2021-08-13-16-31-10.tar.bz2
_2021-08-13-17-06-04.tar.bz2
_2021-08-13-21-18-04.tar.bz2
_2021-08-13-21-36-10.tar.bz2
_2021-08-13-22-03-03.tar.bz2
_2021-08-13-22-16-02.tar.bz2

Partial Dataset Missing Issue

Hello!
In the paper, it is mentioned that the dataset consists of 24 scenes, but there are only 20 scenes available in the download path. I would like to inquire if some parts of the dataset are not downloadable.

projection depth map

Thank you for releasing the dataset.

Can you tell me how to obtain the projection depth map? I use y=K_ nirL * extrinsic_lidarL2nir * x cannot obtain the same depth map as you

ThermalMonoDepth

Is it possible to use the dataset for training which in your previously published method "Self-supervised Depth and Pose from Monocular Thermal Video " models

Thermal looks all black

Hi I recently downloaded the dataset the thermal image looks all black when you view it in photo viewer can someone tell me how to fix this so i can actually view in a viewer

Possibly incorrect FOVs listed in paper

I computed HFOVs from the rectified camera matrices provided in calib.npy with the following equation:
$HFOV = 2 * \text{atand}(n_x / (2 f_x))$
where:

  • atand: inverse tangent in degrees
  • n_x: image width
  • f_x: horizontal focal length (in units of pixels)

this yields:

  • thermal left: 79 degrees (vs 45 in the paper)
  • rgb left: 77 degrees (vs 82.2 in the paper)
  • nir left: 90 degrees (vs 69 in the paper)

I checked the one camera for which I could confidently find the HFOV, the left NIR camera from the Intel RealSense D435i, and the spec sheet lists the HFOV as 91.2 degrees (see table 3-13). It appears the 69 degree HFOV listed in the paper was taken from either the color sensor (table 3-16) or the standard imager (table 3-12), while the D435i uses the wide imager (as listed in table 3-11).

My guess is that HFOV listed for the RGB camera in the paper is correct and that the rectified RGB images lose a slight bit of that FOV. But it seems the thermal and NIR FOVs listed in the paper are incorrect. Can you confirm whether this is the case?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.