wsh312 / s2dnet Goto Github PK

Specular-to-Diffuse Translation for Multi-View Reconstruction

Python 99.50% Shell 0.50%

s2dnet's Introduction

S2Dnet

Specular-to-Diffuse Translation for Multi-View Reconstruction
Shihao Wu¹, Hui Huang², Tiziano Portenier¹, Matan Sela³, Daniel Cohen-Or⁴, Ron Kimmel³, and Matthias Zwicker⁵
¹University of Bern, ²Shenzhen University, ³Technion - Israel Institute of Technology, ⁴Tel Aviv University, ⁵University of Maryland
European Conference on Computer Vision (ECCV), 2018

Dependencies

Python
PyTorch
TensorFlow (optional for tensorboard)
PBRT (optional for rendering)
CycleGAN (our code is based on this implementation)

Update 10/April/2019: The code has been updated to pytorch 0.4. A single-view synthetic dataset (75 GB) is provided, one can train pix2pix or cycleGAN on it.

To-do list:

Implement a single-view translation network (with multi-scale discriminator, re-convolution and pixel-normalization) and provide a testing script.

Downloading (Dropbox links)

Training example

$ python train_multi_view.py --dataroot ../huge_uni_render_rnn --logroot ./logs/job101CP --name job_submit_101C_re1_pixel --model cycle_gan --no_dropout --loadSize 512 --fineSize 512 --patchSize 256 --which_model_netG unet_512_Re1 --which_model_netD patch_512_256_multi_new --lambda_A 10 --lambda_B 10 --lambda_vgg 5 --norm pixel

Testing

Please refer to "./useful_scripts/evaluation/"

Scripts of SIFT, SMVS, and rendering are in "./useful_scripts/".

Please contact the author for more information about the code and data.

s2dnet's People

Contributors

Stargazers

Watchers

Forkers

donrv zebrajack peterzhousz wykxyz peiyi-li felizang

s2dnet's Issues

About GT and reconstruction result

Hello!
I downloaded the "Aligned 3D models for rendering" and "Some Reconstruction results of Synthetic Data" that you uploaded, but I found that GT and reconstruction results were not aligned quantitative when I compared the results.

Could you provide consistent camera pose?

Testing pretrained model

Hi!
I am having difficulties running the code, can you please help me?

I want to test your pretrained S2D network using test_rnn.py (in useful_scripts/evaluation/test_rnn.py).

I have tried to run this code this way:
python3 useful_scripts/evaluation/test_rnn.py --logroot exp_folder/logs/ --checkpoints_dir ../s2d/ --dataroot ../s2d_data/test_data_real/ --model cycle_gan --which_model_netG unet_1024

For generator model I have tried different variations of architectures: unet_512, unet_512_Re1, unet_512_Re2, unet_1024, xrnn_512. But in all this cases I got error when try to initialize and load model ( self.load_network(self.netG_A, 'G_A', which_epoch)):

RuntimeError: Error(s) in loading state_dict for UnetGenerator:
Missing key(s) in state_dict: "model.model.0.bias", "model.model.1.model.1.bias", "model.mo.1.model.3.model.1.bias", "model.model.1.model.3.model.3.model.1.bias", "model.model.1.model.3.mode.model.3.model.1.bias", "model.model.1.model.3.model.3.model.3.model.3.model.1.bias", "model.model.odel.3.model.3.model.3.model.3.model.3.model.1.bias", "model.model.1.model.3.model.3.model.3.model.odel.3.model.3.model.1.bias", "model.model.1.model.3.model.3.model.3.model.3.model.3.model.3.model.odel.1.bias", "model.model.1.model.3.model.3.model.3.model.3.model.3.model.3.model.3.model.3.bias",odel.model.1.model.3.model.3.model.3.model.3.model.3.model.3.model.5.bias", "model.model.1.model.3.el.3.model.3.model.3.model.3.model.5.bias", "model.model.1.model.3.model.3.model.3.model.3.model.5.s", "model.model.1.model.3.model.3.model.3.model.5.bias", "model.model.1.model.3.model.3.model.5.bi, "model.model.1.model.3.model.5.bias", "model.model.1.model.5.bias".
Unexpected key(s) in state_dict: "model.model.5.weight", "model.model.5.bias", "model.modelmodel.7.weight", "model.model.1.model.7.bias", "model.model.1.model.3.model.7.weight", "model.modelmodel.3.model.7.bias", "model.model.1.model.3.model.3.model.7.weight", "model.model.1.model.3.modelmodel.7.bias", "model.model.1.model.3.model.3.model.3.model.7.weight", "model.model.1.model.3.modelmodel.3.model.7.bias", "model.model.1.model.3.model.3.model.3.model.3.model.7.weight", "model.modelmodel.3.model.3.model.3.model.3.model.7.bias", "model.model.1.model.3.model.3.model.3.model.3.modelmodel.7.weight", "model.model.1.model.3.model.3.model.3.model.3.model.3.model.7.bias", "model.modelmodel.3.model.3.model.3.model.3.model.3.model.3.model.7.weight", "model.model.1.model.3.model.3.mod3.model.3.model.3.model.3.model.7.bias", "model.model.1.model.3.model.3.model.3.model.3.model.3.mod3.model.3.model.5.weight", "model.model.1.model.3.model.3.model.3.model.3.model.3.model.3.model.3.ml.5.bias".

My question is, can you please provide some instructions on how to run testing script properly on pretrained S2D model?

Best regards,
Yaroslav

How to test your pretrained network?

Thanks for your excellent work.And I just have no idea that how to test your pretrained network

I have tried a lot of times but cannot load the weights correctly.Could you offer some instructions?Thanks a lot.
By the way,I downloaded the real training data,and it was broken...

Test command

Hi,

Can you kindly provide an example of a command for running your net?
I tried:
test.py --dataroot ./datasets/mydata/ --name S2Dnet --model cycle_gan --logroot 0 --which_model_netG unet_512_Re1

Processing a single image

Hey there!
Your results look very promising so I tried to process a single image through the CycleGAN but I can't figure out how to do that.
The command "test.py --dataroot ./Image --name S2Dnet --model cycle_gan --logroot 0 --which_model_netG unet_512_Re1" throws an error because I didn't define some directorys according to my system.
Can you explain where and how to set the parameters to simply process a single image through your pretrained network?
Thank you very much!

Data about reconstruction result

伍世浩老师您好！
很抱歉打扰您，我是西北工业大学的一名研二学生。不知您是否方便提供一些S2Dnet转换后的图像的重建结果给我。
谢谢您的开源~

About text files used for training

Hello!

I wanted to run "train_multi_view.py", but I didn't know how to create the text dataset to be used here.
(Example: img-200_concat0.txt)
144 300 162 308 167 314
148 302 165 309 171 315
203 93 209 117 209 118
184 220 197 233 201 236
290 168 289 176 291 175
...

Is this data that is outputted when SIFT is run? Also, how can it be created?
I would appreciate it if you could tell me.