Sparse-Prior-Guided-MVS

"Sparse Prior Guided Deep Multi-view Stereo" (Computers and Graphics 2022)

Abstract

Recently, the learning-based multi-view stereo (MVS) methods have achieved notable progress. Despite this, the cost volumes of those methods are not robust enough for depth inference with the existence of noise, outlier and occlusion. In this work, we aim to reduce the matching ambiguities and increase the discrimination of the cost volume. For this purpose, we first propose a sparse prior guidance strategy which incorporates the sparse points of structure-from-motion (SfM) into cost volume construction. Further, we introduce a geometry-aware regularization module to enhance the representative power of cost volume regularization, which could adaptively fit the local geometric shapes. The two modules are straightforward yet effective, resulting in robust and accurate depth estimation. The detailed experiments are conducted on two public datasets (DTU and Tanks & Temples). The experimental results show that with the two components, the representative methods (e.g. MVSNet and CasMVSNet) make a significant gain in reconstruction quality.

Framework

Overview of our sparse prior guided multi-view stereo framework. We propose 1) Cost Volume Modulation that utilizes the sparse prior to modulate the cost distribution along the depth dimension, leading to unimodal distribution peaked at correct depth hypothesis, and 2) Geometry-aware Regularization which enhances the representative power of cost aggregation by additionally learned offsets to better fit the local geometric shape.

Data preprocessing

The sparse points produced by structure-from-motion is the sparse representation of the scene. The sparse points are first projected into image plane to generate the sparse depth, then the sparse priors are used to modulate the cost volume by using the Gaussian function.

Contact

[email protected]

Acknowledgement

This code is based on the MVSNet-pytorch/CasMVSNet, thank xy-guo/Gu Xiaodong@Alibaba for their excellent code.

References

[1] Guided Stereo Matching, Matteo Poggi, Davide Pallotti, Fabio Tosi and Stefano Mattoccia, CVPR 2019.

[2] 3D Deformable Convolutions for MRI classification [paper] [code]

xyz-qiyh / sparse-prior-guided-mvs Goto Github PK