The im2pano3d from shurans

im2pano3d's Introduction

Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View

This repo contains training and testing code for our paper on Im2Pano3D. More information about the project can be found in our paper and project webset

Organization

The code and data are organized as follows:

    im2pano3d
         |-- matlab_code
         |-- torch_code
         |-- models
         |-- datalist
         |-- data
                |-- mpv3  
                |-- suncgpano

Installation

Requirements for torch (see: torch installation instructions)
Install torch-hdf5

sudo apt-get install libhdf5-serial-dev hdf5-tools
git clone https://github.com/deepmind/torch-hdf5
cd torch-hdf5
luarocks make hdf5-0-0.rockspec LIBHDF5_LIBDIR="/usr/lib/x86_64-linux-gnu/"

Download

Dwonload training and testing panorama data: ./download_pano_data.sh [41 G]
Dwonload models: ./download_models.sh [1.7 G]

Training

Export path

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cudnn/v5/lib64

Train on SUNCG data

cd torch_code/
name=suncg maskType=twoview   loss_xyz=1 loadOpt=rgbpns  Gtype_in=rgbpns   Gtype_out=pns  dataset=suncg  dataPath=../data/suncgpano/ DATA_ROOT=../datalist/trainlist_suncgroom8_10000.txt gpu=1 th train.lua  2>&1 | tee ./checkpoints/training_suncg.log

Train on Matterport3D data

cd torch_code/
name=mp maskType=twoview   loss_xyz=1 loadOpt=rgbpns  Gtype_in=rgbpns   Gtype_out=pns  dataset=mp  dataPath=../data/mpv3/  DATA_ROOT=../datalist/trainlist_mp3.txt gpu=1 th train.lua  2>&1 | tee ./checkpoints/training_suncg.log

Supported input output modalities:

Gtype_in	Gtype_out	loadOpt
(input)	(output)	(load type)
pns	pns	pns
rgbpn	pns	rgbpns
rgbpns	pns	rgbpns
rgb	pns	rgbpns

Supported mask types (maskType):
twoview: Input two views of the panorama and predicts the other two. (a)
1camera: one normal camera (b)
3camera: three normal cameras (c)
middlecamera: Input six RGB-D camera looking horizantoally forward (d)
upcamera: Input six RGB-D camera looking upwards (e)
random: random mask
nomask: No mask applied in the input image. Example usage, input rgb panorama output pns.(f)\

Testing

name=mp_twoview_rgbpns  maskType=twoview dataset=suncg dataPath=../data/suncgpano/ checkpoints_dir=../checkpoints  DATA_ROOT=../datalist/testlist_suncgroom8.txt how_many=480 th test.lua

Testing with pre-trained models:

name=mpft_suncg_pns_pns_twoview  maskType=twoview dataset=suncg dataPath=../data/suncgpano/  checkpoints_dir=../models DATA_ROOT=../datalist/testlist_suncgroom8.txt how_many=480 th test.lua

Depth encoding conversion

See ./matlab_code/pano_io/pano_demo.m. It shows how to \

Compute the 3D pointcloud from depth representation. \
Compute the 3D pointcloud from plane equation representation. \

Acknowledgment

Code is inspired by DCGAN, DCGAN, Context-Encoder and pix2pix.

im2pano3d's People

Contributors

Stargazers

Watchers

im2pano3d's Issues

About missing Matlab function

I tend to visualize the 3D PLY results. I notice that there exists a flag called save_visual in the function vis_result. However, when I turn it on, MATLAB will throw an exception about missing function: pn2Room.

I am not sure that I miss some dependencies or visualize in PLY won't be supported for now. Could you help me to figure out what's happened here?

What are all the included pano files?

Can you clarify what the filenaming conventions mean in the pano directorys?

Here's my best guess, but I haven't found any documentation for the most part.
Depth : *d_r0.4.png
Image : *i_r0.4.jpg
Object labels : *_label13.png
Normals : *n_r0.4.png
Planer depth described in paper? : *p_r0.4.png
Different object label set? : *s_r0.4.png
Visualization of object labels?: *sc_r0.4.png

expecting a contiguous tensor

After“name=mp maskType=twoview loss_xyz=1 loadOpt=rgbpns Gtype_in=rgbpns Gtype_out=pns dataset=mp dataPath=../data/mpv3/ DATA_ROOT=../datalist/trainlist_mp3.txt gpu=1 th train.lua 2>&1 | tee ./checkpoints/training_suncg.log”， “Tensor.lua:461: expecting a contiguous tensor”， I don't know how to solve it.Have you encountered any similar problems?Thank you！

Recommend Projects

shurans / im2pano3d Goto Github PK