nie-lang / multi-grid-deep-homography Goto Github PK

View Code? Open in Web Editor NEW

79.0 2.0 5.0 12.7 MB

TCSVT2021 - Multi-Grid Deep Homogarphy

Python 100.00%

multi-grid-deep-homography's Introduction

Depth-Aware Multi-Grid Deep homogrpahy Estimation with Contextual Correlation (paper)

Lang Nie*, Chunyu Lin*, Kang Liao*, Shuaicheng Liu`, Yao Zhao*

* Institute of Information Science, Beijing Jiaotong University

` School of Information and Communication Engineering, University of Electronic Science and Technology of China

Requirement

python 3.6
numpy 1.18.1
tensorflow 1.13.1

For pytorch users

The official codes are based on tensorflow. We also provide a simple pytorch implementation of CCL for pytorch users, please refer to https://github.com/nie-lang/Multi-Grid-Deep-Homography/blob/main/CCL_pytorch.py.

The pytorch version has not been strictly tested. If you encounter some problems, please feel free to concat me ([email protected]).

Dataset Preparation

step 1

We use UDIS-D for training. Please download it.

step 2

We adopt a pretrained monocular depth estimation model to get the depth of 'input2' in the training set. Please download the results of depth estimation in Google Drive or Baidu Cloud(Extraction code: 1234). Then place the 'depth2' folder in the 'training' folder of UDIS-D. (Please refer to the paper for more details about the depth. )

For windows system

For windows OS users, you have to change '/' to '\\' in 'line 73 of Codes/utils.py'.

Training

Step 1: Training without depth assistance

Modidy the 'Codes/constant.py' to set the 'TRAIN_FOLDER'/'ITERATIONS'/'GPU'. In our experiment, we set 'ITERATIONS' to 300,000.

Modify the weight of shape-preserved loss in 'Codes/train_H.py' by setting 'lam_mesh' to 0.

Then, start the training without depth assistance:

cd Codes/
python train_H.py

Step 2: Finetuning with depth assistance

Modidy the 'Codes/constant.py' to set the 'TRAIN_FOLDER'/'ITERATIONS'/'GPU'. In our experiment, we set 'ITERATIONS' to 500,000.

Modify the weight of shape-preserved loss in 'Codes/train_H.py' by setting 'lam_mesh' to 10.

Then, finetune the model with depth assistance:

python train_H.py

Testing

Our pretrained model

Our pretrained homography model can be available at Google Drive or Baidu Cloud(Extraction code: 1234). And place it to 'Codes/checkpoints/' folder.

Testing with your own model

Modidy the 'Codes/constant.py'to set the 'TEST_FOLDER'/'GPU'. The path for the checkpoint file can be modified in 'Codes/inference.py'.

Run:

python inference.py

multi-grid-deep-homography's People

Contributors

Stargazers

Watchers

Forkers

llj-l pegasuszz thuniinae odiofan focueai

multi-grid-deep-homography's Issues

提问

作者您好，请问用论文中提到的预训练模型预测深度图时的具体步骤是怎样的

requirements.txt file isn't necessary??

Hi,
Thank you for sharing your code.
Don't you think that a requirements.txt file is necessary in order to have the right version for all modules that are used in your code?? For example "tensorflow.contrib.layers import conv2d" is referred to an older version of tensorflow and an older version of python I think is required in order to import this module. Can you enlighten me??

Thanks

Training depth image

Excuse me, how can I get the training depth image? It seems doesn't work when I training with my own dataset.

预测深度图

CCL_pytorch

Very nice project! but the implementation of CCL_pytorch that you have created may cause a memory overflow when used, did you try it.
just test it like this. Looking forward to your reply。

Doubt

Hi I am going through the code have some doubts.

What is the purpose of scale transformation matrix .

M = np.array([[patch_size / 2.0, 0., patch_size / 2.0],
                  [0., patch_size / 2.0, patch_size / 2.0],
                  [0., 0., 1.]]).astype(np.float32)

It seems to me Matrix M also has translation component.
2)What is the this line doing?
H1_mat = tf.matmul(tf.matmul(M_tile_inv, H1), M_tile)

(code snippets are from H_model.py,function name :H_model)

预测深度图

作者您好，请问您在用论文中提到的预训练模型预测深度图时，训练环境pytorch版本一定是原文中的0.4.1吗，

> hello,I have a question that can this learning-based method be used for video stitching?thank you for your advise.

hello,I have a question that can this learning-based method be used for video stitching?thank you for your advise.

It can be used for alignment, but it can not be used for stitching due to the Backward Multi-Grid Deformation.
Or if the mesh resolution is set to 1*1 (equal to a global homography), it can be used for stitching.

Originally posted by @nie-lang in #8 (comment)

Backward Multi-Grid Deformation

can some one tell me where is Backward Multi-Grid Deformation implemented in the code?

list index out of range

Hello, I followed your steps, but I ran into some problems？

feature flow visualization

Excuse me, how to visualize the output of contextual correlation layer with a pair of images ?

Resolution of 128x128

Hello, I would like to ask which parts of the code need to be modified to test with a resolution of 128x128. I tried to make some modifications but encountered the following error.

ValueError: Dimension 1 in both shapes must be equal, but are 16 and 64. Shapes are [1,16,16] and [1,64,64]. for 'generator/model/feature_extract/conv_block4/concat' (op: 'ConcatV2') with input shapes: [1,16,16,128], [1,64,64,64], [1,64,64,64], [1,64,64,128], [] and with computed input tensors: input[4] = <3>.

about feature flow

Dr. Nie, I would like to ask why there is one mod W and one //W in the formula for predicting the horizontal and vertical motions value of feature flow. In addition, does the formula have a more specific description, some seem to understand but not understand, thank you

question

Dear Dr. Nie! I have been reproducing your work recently and want to further apply it to complete the stitching of the target image and the reference image, so as to achieve the effect of a complete stitching, but I don't know how to do it, could you please give me an answer? Thank you.