FlowFormer: A Transformer Architecture for Optical Flow
Zhaoyang Huang*, Xiaoyu Shi*, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, Hongsheng Li
ECCV 2022
Our FlowFormer++ and VideoFlow are accepted by CVPR and ICCV, which ranks 2nd and 1st on the Sintel benchmark! Please also refer to our FlowFormer++ and VideoFlow.
Similar to RAFT, to evaluate/train FlowFormer, you will need to download the required datasets.
- FlyingChairs
- FlyingThings3D
- Sintel
- KITTI2015
- TubCrowdFlow
- HD1K (optional)
By default datasets.py
will search for the datasets in these locations. You can create symbolic links to wherever the datasets were downloaded in the data
folder
├── data
├── Sintel
├── test
├── training
├── KITTI
├── testing
├── training
├── devkit
├── FlyingChairs_release
├── data
├── FlyingThings3D
├── frames_cleanpass
├── frames_finalpass
├── optical_flow
├── TUBCrowdFlow
├── gt_flow
├── gt_trajectories
├── images
├── masks
├── Wuhan_Metro
├── transfer1-1-20231231170000-20231231203000-100992192
├── 江汉路-D口地面扶梯-2-20231231170000-20231231203000-30896731
├── 江汉路-付费区B端扶梯-3-20231231170000-20231231203000-28677086
├── 江汉路-厅东闸机1-3-20231231170000-20231231203000-29300357
├── 江汉路-厅东闸机2-2-20231231170000-20231231203000-29254227
conda create --name flowformer
conda activate flowformer
conda install pytorch=1.6.0 torchvision=0.7.0 cudatoolkit=10.1 -c pytorch
pip install yacs loguru einops timm==0.4.12 imageio tqdm matplotlib tensorboard scipy opencv-python
The script will load the config according to the training stage. The trained model will be saved in a directory in logs
and checkpoints
. For example, the following script will load the config configs/default.py
. The trained model will be saved as logs/xxxx/final
and checkpoints/chairs.pth
.
python -u train_FlowFormer.py --name chairs --stage chairs --validation chairs
To finish the entire training schedule, you can run:
./run_train.sh
We provide models trained in the four stages. The default path of the models for evaluation is:
├── checkpoints
├── chairs.pth
├── sintel.pth
├── things.pth
├── kitti.pth
├── flowformer-small
├── chairs.pth
├── sintel.pth
├── things.pth
├── things_kitti.pth
The models under flowformer-small is a small version of our flowformer. things_kitti.pth is the FlowFormer# introduced in our supplementary, used for KITTI training set evaluation.
The model to be evaluated is assigned by the _CN.model
in the config file.
Evaluating the model on the Sintel training set and the KITTI training set. The corresponding config file is configs/sintel.py
.
# with tiling technique
python evaluate_FlowFormer_tile.py --eval sintel_validation
python evaluate_FlowFormer_tile.py --eval kitti_validation --model checkpoints/things_kitti.pth
# without tiling technique
python evaluate_FlowFormer.py --dataset sintel
with tile | w/o tile | |
---|---|---|
clean | 0.94 | 1.01 |
final | 2.33 | 2.40 |
Evaluating the small version model. The corresponding config file is configs/small_things_eval.py
.
# with tiling technique
python evaluate_FlowFormer_tile.py --eval sintel_validation --small
# without tiling technique
python evaluate_FlowFormer.py --dataset sintel --small
with tile | w/o tile | |
---|---|---|
clean | 1.21 | 1.32 |
final | 2.61 | 2.68 |
Generating the submission for the Sintel and KITTI benchmarks. The corresponding config file is configs/submission.py
.
python evaluate_FlowFormer_tile.py --eval sintel_submission
python evaluate_FlowFormer_tile.py --eval kitti_submission
Visualizing the sintel dataset:
python visualize_flow_image.py --eval_type sintel --keep_size
Visualizing an image sequence extracted from a video:
python visualize_flow_image.py --eval_type seq
The default image sequence format is:
├── demo_data
├── mihoyo
├── 000001.png
├── 000002.png
├── 000003.png
.
.
.
├── 001000.png
Finetuning the model on the TubCrowdFlow dataset.
python finetune_FlowFormer.py --stage tub --validation tub
The validation results with tile of FlowFormer on the TubCrowdFlow dataset are shown below.
The results are generated by the model trained on the Sintel dataset.
python evaluate_FlowFormer_tile.py --eval tub_validation --model checkpoints/sintel.pth
EPE | AE | IE | |
---|---|---|---|
IM01 | 0.421972 | 0.212379 | 51.086972 |
IM01_hDyn | 2.626816 | 1.079218 | 65.503151 |
IM02 | 0.305167 | 0.145175 | 43.489626 |
IM02_hDyn | 2.346542 | 1.013571 | 59.352042 |
IM03 | 0.238332 | 0.165726 | 66.957967 |
IM03_hDyn | 2.455814 | 1.012770 | 56.903450 |
IM04 | 0.223487 | 0.106554 | 35.952300 |
IM04_hDyn | 3.359264 | 1.145701 | 56.903450 |
IM05 | 1.112371 | 0.470768 | 74.283647 |
IM05_hDyn | 6.204594 | 1.168258 | 89.512292 |
After finetuning on the TubCrowdFlow dataset, the results are shown below.
EPE | AE | IE | |
---|---|---|---|
IM01 | 0.093563 | 0.039530 | 38.147586 |
IM01_hDyn | 1.391226 | 0.675693 | 57.790450 |
IM02 | 0.107384 | 0.037806 | 37.189272 |
IM02_hDyn | 0.942481 | 0.465499 | 53.761484 |
IM03 | 0.078868 | 0.048568 | 34.212200 |
IM03_hDyn | 1.127714 | 0.546711 | 61.210028 |
IM04 | 0.098443 | 0.034583 | 33.293113 |
IM04_hDyn | 0.670348 | 0.276773 | 48.747687 |
IM05 | 0.367640 | 0.142700 | 67.033006 |
IM05_hDyn | 2.023295 | 0.501428 | 80.836906 |
python stat_FlowFormer.py --eval tub_validation
Model | Params(M) | MACs(G) |
---|---|---|
sintel.pth | 1.057T | 16.081M |
FlowFormer is released under the Apache License
@article{huang2022flowformer,
title={{FlowFormer}: A Transformer Architecture for Optical Flow},
author={Huang, Zhaoyang and Shi, Xiaoyu and Zhang, Chao and Wang, Qiang and Cheung, Ka Chun and Qin, Hongwei and Dai, Jifeng and Li, Hongsheng},
journal={{ECCV}},
year={2022}
}
@inproceedings{shi2023flowformer++,
title={Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation},
author={Shi, Xiaoyu and Huang, Zhaoyang and Li, Dasong and Zhang, Manyuan and Cheung, Ka Chun and See, Simon and Qin, Hongwei and Dai, Jifeng and Li, Hongsheng},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={1599--1610},
year={2023}
}
@article{huang2023flowformer,
title={FlowFormer: A Transformer Architecture and Its Masked Cost Volume Autoencoding for Optical Flow},
author={Huang, Zhaoyang and Shi, Xiaoyu and Zhang, Chao and Wang, Qiang and Li, Yijin and Qin, Hongwei and Dai, Jifeng and Wang, Xiaogang and Li, Hongsheng},
journal={arXiv preprint arXiv:2306.05442},
year={2023}
}
In this project, we use parts of codes in: