Liyuan Zhu, Yuru Jia, Shengyu Huang, Nicholas Meyer, Andreas Wieser, Konrad Schindler, Jordan Aaron
ETH Zurich
This repository is the official implementation of paper: DeFlow: Self-supervised 3D Motion Estimation of Debris Flow, CVPRW 2023.
Existing work on scene flow estimation focuses on autonomous driving and mobile robotics, while automated solutions are lacking for motion in nature, such as that exhibited by debris flows. We propose \deflow, a model for 3D motion estimation of debris flows, together with a newly captured dataset. We adopt a novel multi-level sensor fusion architecture and self-supervision to incorporate the inductive biases of the scene. We further adopt a multi-frame temporal processing module to enable flow speed estimation over time. Our model achieves state-of-the-art optical flow and depth estimation on our dataset, and fully automates the motion estimation for debris flows.
First clone our repository:
git clone https://github.com/Zhu-Liyuan/DeFlow
cd DeFlow
You will need to install conda to build the environment.
conda create -n DeFlow python=3.9
conda activate DeFlow
pip install -r requirements.txt
We provide preprocessed debris flow dataset. The preprocessed dataset and checkpoint can be downloaded by running:
wget --no-check-certificate --show-progress https://share.phys.ethz.ch/~gsg/DeFlow/DeFlow_Dataset.zip
unzip DeFlow_Dataset.zip -d data
wget --no-check-certificate --show-progress https://share.phys.ethz.ch/~gsg/DeFlow/checkpoint.zip
unzip checkpoint.zip
You can also build your own dataset following the structure below
โโโ Data
โโโ Cam1
โโโ 000001.jpg
โโโ 000002.jpg
.
.
โโโ 00000X.jpg
โโโ Cam2
โโโ 000001.jpg
โโโ 000002.jpg
.
.
โโโ 00000X.jpg
โโโ LiDAR
โโโ 000001.ply
โโโ 000002.ply
.
.
โโโ 00000X.ply
โโโ Transformations
โโโ cam_intrinxics.txt
โโโ LiCam_tranformations.txt
To train a model, run:
python main.py --config_path configs/deflow_default.yaml
and you can change the mode to eval in the config file for evaluation.
If you have any questions, please let me know:
- Liyuan Zhu {[email protected]}
If you use DeFlow for any academic work, please cite our original paper.
@InProceedings{zhu2023DeFlow,
author = {Liyuan Zhu and Yuru Jia and Shengyu Huang and Nicholas Meyer and Andreas Wieser and Konrad Schindler, Jordan Aaron},
title = {DEFLOW: Self-supervised 3D Motion Estimation of Debris Flow},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2023}
}
Additionally, we thank the respective developers of the following open-source projects: