Giter Club home page Giter Club logo

amir-arsalan / synthesize3dviadepthorsil Goto Github PK

View Code? Open in Web Editor NEW
163.0 12.0 31.0 141.29 MB

[CVPR 2017] Generation and reconstruction of 3D shapes via modeling multi-view depth maps or silhouettes

Home Page: http://openaccess.thecvf.com/content_cvpr_2017/papers/Soltani_Synthesizing_3D_Shapes_CVPR_2017_paper.pdf

License: MIT License

Lua 100.00%
3d-reconstruction 3d-generation depth-maps silhouette 2d-3d generative-models representation-sharing deep-learning 3d-shapes variational-autoencoder torch computer-vision 3d reconstruction 2d-to-3d perception

synthesize3dviadepthorsil's Introduction

Synthesizing 3D Shapes via Modeling Multi-View Depth Maps and Silhouettes with Deep Generative Networks

This repository provides a Torch implementation of the framework proposed in CVPR 2017 paper Synthesizing 3D Shapes via Modeling Multi-View Depth Maps and Silhouettes with Deep Generative Networks by Amir A. Soltani, Haibin Huang, Jiajun Wu, Tejas Kulkarni and Joshua Tenenbaum

Slides used for two invited talks at CVPR 2017 Vision Meets Cognition Workshop and MIT Vision Seminar (contains new results): Here

CVPR 2017 Poster: Here

Requirements

Optional Requirements

  • tSNE -- For running the tSNE experiment
  • cudnn v6 or higher -- git clone https://github.com/soumith/cudnn.torch.git -b R7 && cd cudnn.torch && luarocks make cudnn-scm-1.rockspec

Installing Torch

1- Install LuaJIT and LuaRocks

The following installs luaJIT and luarocks locally in $HOME/usr. If you want a system-wide installation, remove the -DCMAKE_INSTALL_PREFIX=$HOME/usr option.

git clone https://github.com/torch/luajit-rocks.git
cd luajit-rocks
mkdir build; cd build
cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/usr -DWITH_LUAJIT21=OFF
make -j 4
make install
cd ../..

We assume luarocks and luajit are in $PATH. If they are not - and assuming you installed them locally in $HOME/usr - you can instead run ~/usr/bin/luarocks and ~/usr/bin/luajit.

If you run into compilation issues or you are willing to use CUDA version 9.1 and higher, you may try using this Git repo to compile Torch7 as follow:

git clone https://github.com/nagadomi/distro.git ~/torch --recursive
cd ~/torch
./install-deps
./install.sh

2- Installing Torch and other Torch packages

luarocks install torch
luarocks install cudnn # for GPU support
luarocks install cunn # for GPU support

Hardware Requirements

We recommend using a machine with ~200GBs of free storage (~60GBs if you're using ModelNet40), ~10GBs of memory and a GPU with ~5GBs of memory with the default arguments. You will need less than 2GBs of free GPU memory when using the model for running experiments (4_0_runExps.lua). GPU memory and RAM requirement could be reduced by setting nCh and maxMemory arguments to a smaller values respectively.

Running

Parameters

Use main.lua to train new models or run experiments for a pre-trained model. Before running the code make sure you specify a directory name for your model-to-be-trained by setting the input arguments:

  • benchmark: Set to 1 if you want to use a benchmark data set (E.g. ModelNet40 here)
  • fromScratch: If set to 1 the code will load the 2D images to Torch tensors and save them onto disk. Make sure you have unzipped the contents of the .zip files either manually or via running the code with the zip argument set to 1
  • modelDirName: Name of the directory to save model and results of each run of the code
  • dropoutNet: Set to 1 to train or use a pre-trained DropOutNet model
  • singleVPNet: Set to 1 to trainor or use a pre-trained SingleVPNet model
  • conditional: Set to 1 for conditional models
  • silhouetteInput: Set to 1 to use silhouettes for training/testing
  • KLD: The KLD term's gradient coefficient (smaller values like 10 delivers better reconstructions after training)
  • experiment: Set to 0 when you want to train a model and to 1 when you want to use a pre-trained model
  • fromEpoch: Set to the epoch number from which you want to load a model and use it for testing (use when experiment is set to 1). Set to 0 to use one of the pre-trained models (download from here)

Setting both dropoutNet and singleVPNet to 0 means that you are working with a AllVPNet model.

architecture

Data

Using the pre-rendered data

You can download the pre-rendered data sets for ShapNet Core and ModelNet 40 from here. You will need about ~30GBs and ~8GBs for the .zip files for ShapeNet Core and ModelNet40 data sets respectively. The data sets only contain raw depth map rendering results in float or int (stored in png files). To process the data sets and store the viewpoint depth maps into Torch tensors you would need an additional ~160GBs and ~55GBs for each ShapeNet Core or ModelNet40 respectively. Set the benchmark argument to 0 or 1 to use the ShapeNet core or ModelNet40 data sets respectively.

Run the following command to store the rendered 2D images of depth maps into Torch tensors:

th main.lua -zip 1 -fromScratch 1 -maxMemory 3000 -train 0

Rendering Viewpoints

render

If you want to create your own data set you can use the rendering code provided in /depth_render_reconstruction/code/renderDepth/. The obj/ply files names should be in model_x format where x could be any integer. You need to have install Microsoft Visual C++ 2013 Redistributable. After rendering view points to create your data set make sure you follow the following pattern for creating your .zip files:

CategoryName.zip
└── CategoryName_depth_[rgb/float]
    ├── model_x_Cam_0.jpg 
    ├── model_x_Cam_0.jpg
    ├── ...
    └── model_x_Cam_19.jpg

Note that the file camPosList.txt contains the x, y and z coordinates for 20 camera positions that we used. In case you want to compile the rendering pipeline and not use the provided .exe file for rendering views, you may compile this code . You may also compile a newer version of the same code provided here.

Training:

Here are a few examples on how to train different models. The models will be trained on the ShapeNet core data set unless you set benchmark to 1:

Train an unconditional AllVPNet model using depth maps:
th main.lua  -modelDirName "AllVPNet" -dropoutNet 0 -singleVPNet 0 -silhouetteInput 0 -conditional 0 -benchmark 0

Train a conditional `DropoutNet` model using depth maps:
th main.lua  -modelDirName "DropoutNet-Conditional" -dropoutNet 1 -singleVPNet 0 -silhouetteInput 0 -conditional 1 -benchmark 0

Train a conditional SingleVPNet using silhouettes:
th main.lua  -modelDirName "SingleVPNet-Cond" -dropoutNet 0 -singleVPNet 1 -silhouetteInput 1 -conditional 1 -benchmark 0

Train an unconditional AllVPNet model for `ModelNet40` data set with silhouettes:
th main.lua  -modelDirName "AllVPNet" -dropoutNet 0 -singleVPNet 0 -silhouetteInput 0 -conditional 0 -benchmark 1

Make sure you set the testPhase argument to 0 when training or running experiments

Testing:

To load a model and use it for testing (reconstruction, sampling etc) download a pre-trained model from here and do the followings:

Make sure you have stored the depth maps into tensors (th main.lua -zip 1 -fromScratch 1 -maxMemory 3000 -train 0)
Set the optional arguments in accordance to the model downloaded
Copy model.t7 and mean_logvar.t7 to modelDirName/model/
th main.lua -modelDirName 'someModelDirName' -experiment 1 -expType 'randomSampling' -conditional 1 -sampleCategory 'chair, car, sofa, laptop, guitar, airplane'
expType could be either randomSampling, interpolation, forwardPass, NNs or tSNE

Optional:

Set the testPhase argument to 1 to only use the test set for quick runs/tests

Visualization in 3D (Point Cloud)

To generate the final 3D shape do the followings:

Unzip `depth_render_reconstruction.zip`
Compile the code in /depth_render_reconstruction/code/depthReconstruction_Ubuntu/depthReconstruction (Make sure OpenCV is installed)
Copy the built executable file `depthReconstruction` and into the results directory; camPosList.txt must be present in the same directory.
Run the 4_1_visualization.lua -inputDir 'anywhereOnDisk' -outputDir 'anywhereOnDisk' -experiment 'sampling'
The valid inputs for experiment are reconstruction, sampling, interpolation and `NN`. Use reconstruction for nyud and userData experiments.

Results

Random Sampling

th main.lua -modelDirName 'someModelDirName' -experiment 1 -expType 'randomSampling' -conditional 1 -sampleCategory 'chair, car, sofa, laptop, guitar, airplane'

For conditional models you can specify which categories to get samples for by having comma-separated category names in the sampleCategory argument. Simply remove sampleCategory argument if you want to get samples for all categories.

Here are some random samples obtained from unconditionally-trained models: random

And here are some conditional samples. The models are able to quickly learn the generative distribution for a new category through hierarchical priors and using the learned representations for other categories. For instance, we only had about 50 samples of headphones in the training set but the model generates good-looking samples of headphones after training: conditional

Nearest Neighbors

We show that our models do not memorize the training samples by showing the nearest neighbor examples (on the Z layer embeddings) from the test set. You can run the nearest neighbor experiment for both unconditional and conditional samples. You can download the samples selected manually (manual selection is done via viz.txt) from here. To get the nearest neighbor samples:

Make sure you have your selected samples here `/modelDirName/experiments/[conditionalSamples/randomSamples]-empirical/`
`th main.lua  -modelDirName "AllVPNet-Conditional-Sil"  -silhouetteInput 1 -conditional 1 -benchmark 0 -experiment 1 -fromEpoch 0 -expType 'NNs'`

If you want to use the model trained on yoru machine, make sure you set the fromEpoch argument and create a viz.txt file in each sample set directory(e.g .../randomSamples-empirical/sample1/ or .../conditionalSamples-empirical/airplane/sample1/)

nearest

Reconstruction

th main.lua -modelDirName 'someModelDirName' -experiment 1 -conditional 0 -expType 'forwardPass' -forwardPassType 'randomReconstruction' -nReconstructions 80

Run the following to reconstruct all samples in the test data set:

th main.lua -modelDirName 'someModelDirName' -experiment 1 -conditional 1 -expType 'forwardPass' -forwardPassType 'reconstructAllSamples'

reconstruction

Out-of-sample generalization

Make sure you extract the zip-ExtraData.zip into the repository directory before running the following commands. For userData the code will convert rgb images into silhouettes ifsilhouetteInput is set to 1. If you set it to 0 the code will assume you only have depth maps in /ExtraData/userData/ and are working with a model trained on depth maps.

NYUD

th main.lua -modelDirName 'someModelDirName' -experiment 1 -conditional 0 -expType 'forwardPass' -forwardPassType 'nyud'

Random silhouettes/depth maps from user

th main.lua -modelDirName 'someModelDirName' -experiment 1 -conditional 0 -silhouetteInput 0 -expType 'forwardPass' -forwardPassType 'userData'

Interpolation

th main.lua -modelDirName 'someModelDirName' -conditional 0 -experiment 1 -expType 'interpolation' -nSamples 3 -sampleCategory 'chair, car, sofa, laptop, guitar, airplane'

Remove -sampleCategory 'chair, car, sofa, laptop, guitar, airplane' if you want to get interpolation results for all categories

tSNE Visualization

Run the following command for applying the tSNE algorithm on the Z layer embeddings.

th main.lua -modelDirName 'someModelDirName' -experiment 1 -conditional 0 -silhouetteInput 0 -expType 'tSNE''

Here is the tSNE visualization for all of the ShapeNet Core training samples:

Click here to view it with the original resolution.

Representation Consistency

SingleVPNet models give similar reconstructions/IoU/classification accuracy for most viewpoints, meaning that they obtain some consistent representations. You can run the representation consistency experiment on our ShapeNet Core test set by running the following command:

th main.lua  -modelDirName "SingleVPNet-Conditional-Depth"  -silhouetteInput 0 -singleVPNet 1 -conditional 1 -benchmark 0 -experiment 1 -fromEpoch 0 -expType 'forwardPass' -forwardPassType 'reconstructAllSamples' -allViewsExp 1
th main.lua  -modelDirName "SingleVPNet-Conditional-Silhouette"  -silhouetteInput 1 -singleVPNet 1 -conditional 1 -benchmark 0 -experiment 1 -fromEpoch 0 -expType 'forwardPass' -forwardPassType 'reconstructAllSamples' -allViewsExp 1

consistency

IoU Computation

Execute the following to get the reconstructions for all test samples:

th main.lua -modelDirName 'someModelDirName' -experiment 1 -expType 'forwardPass' -forwardPassType 'reconstructAllSamples'`

For a SingleVPNet model execute the following to get reconstructions for each views separately:

th main.lua -experiment 1 -expType 'forwardPass' -forwardPassType 'reconstructAllSamples' -allViewsExp 1`

To get the IoU numbers, in a Windows machine open the README.txt in zip-IoU.zip and install the requirements. Then Run run.bat

Citation

Please cite our work if you found our paper helpful and/or use our code for your research:

@inproceedings{{3DVAE,
  title={Synthesizing 3D Shapes via Modeling Multi-View Depth Maps and Silhouettes with Deep Generative Networks},
  author={Soltani, Amir Arsalan and Huang, Haibin and Wu, Jiajun and Kulkarni, Tejas D and Tenenbaum, Joshua B}
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={1511-1519},
  year={2017}
}

synthesize3dviadepthorsil's People

Contributors

amir-arsalan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

synthesize3dviadepthorsil's Issues

Training on my own data

When I try and train on my own dataset, I include files for all 20 viewpoints per the README, but when I perform random sampling on the manifold I get results (the 5x5 grids) that are the same for each viewpoint. However when I train with the same arguments on class data from ShapeNet Core through the provided link (e.g. headphones) then I am able to get expected results with different views for each viewpoint. Could you offer any ideas of what I might be doing wrong with my data? Thanks.

Need input of dimension 4

I have my own trained models (AllVPNet and dropoutNet) trained from ShapeNet Core with my own 20 determined viewpoints with the default settings @epoch80 and batchsize of 4. Both networks are trained on depth maps.

However, I encountered the following error when I tried to output the results for "Random silhouettes/depth maps from user"

Error thrown:
Need input of dimension 4 and input.size[1] == 20 but got input to be of shape: [2 x 1 x 224 x 224]

The input of 20 depth maps are produced with the provided renderDepth program.
The error occurred when trying to use my trained models AllVPNet and also dropoutNet.

However, it worked when I used the originally provided pre-trained model "singleVPNet".
The provided pre-trained model "AllVPNet" and "dropoutNet" also didn't worked.

or could it be that using my own depth maps in "userData" only works for singleVPNet?
If so, how could I get it work with AllVPNet and/or dropoutNet?

I would like to produce results from my own trained network AllVPNet and dropoutNet from an input of 20 depth maps but could not get it to work so far.

Any help is really much appreciated

Thanks in advance

How to compute the IoU

Hello, can you give me c code for calculating IoU? Or teach me how to calculate it? I looked at the process of calculating IoU in 3d-r2n2, but I still don't understand it. Looking forward to your reply.

Question about the loss

Hello,I read the paper "Synthesizing 3D Shapes via Modeling Multi-view Depth Maps and Silhouettes with Deep Generative Networks",the loss function which mention inside is to minimize : loss = -DKL(q(Z|X)||p(Z)) + E[logp(x|Z)]
but the loss function in VAE is :
loss = DKL(q(Z|X)||p(Z)) - E[logp(x|Z)]
I am confused about it, I hope you can give me some advice,thank you

getIoU

Hi Amir,

I tried getting the IoU for quantitative evaluations.
Requirements are all met and runs without any error.
Requirements:

  1. Visual C++ Redistributable for Visual Studio 2015 (installed-but not from the provided link because a newer version is already installed when installing Visual Studio 2015,
    Version 14.0.24215 was installed instead of the provided Version 14.0.23026)
  2. Visual C++ Redistributable Packages for Visual Studio 2013 (installed from the provided link)
  3. Python 3.7 (installed)
  4. Anaconda with Python 3.6 (installed)

Paths of the .ply files in the test folder are as below:
test/model-userData-1/model-userData-1-or.ply
test/model-userData-1/model-userData-1-rec.ply
test/model-userData-2/model-userData-1-or.ply
test/model-userData-2/model-userData-1-rec.ply
test/model-userData-3/model-userData-1-or.ply
test/model-userData-3/model-userData-1-rec.ply
etc

However, I'm only getting
"----------------average------------------
-nan(ind)"

I believed that the issue might be lying somewhere in the getIoU.exe because it seems to run
without any errors till the end.
However, I could not further verify the problem since the getIoU is a binary.

Have you come across the same problem before?
Any pointers/advise is really much appreciated.

Thanks in advance

Best regards,
Safwan

Questions about trainning the network

I have two questions and I really hope to get your help.

First, I have a big problem with training data.
I want to use the ShapeNet Core dataset to repeat your experiments. So I'm going to convert the .mat file into the .ply file.I've found that I can copy the vertex array and faces array directly from the mat file into the ply file. But this approach is too complex. Do you have some simpler ways to do it?

Second, I find the following problems when I training the network of AllVP with a 3D shape(only depths from 20 views). I don't know if it's due to my incorrect input.

/install/torch/install/bin/luajit: /install/torch/install/share/lua/5.1/nn/Container.lua:67:
In 3 module of nn.Sequential:
...torch/install/share/lua/5.1/cudnn/BatchNormalization.lua:44: assertion failed!
stack traceback:
[C]: in function 'assert'
...torch/install/share/lua/5.1/cudnn/BatchNormalization.lua:44: in function 'createIODescriptors'
...torch/install/share/lua/5.1/cudnn/BatchNormalization.lua:60: in function <...torch/install/share/lua/5.1/cudnn/BatchNormalization.lua:59>
[C]: in function 'xpcall'
/install/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'
/install/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'func'
/install/torch/install/share/lua/5.1/nngraph/gmodule.lua:345: in function 'neteval'
/install/torch/install/share/lua/5.1/nngraph/gmodule.lua:380: in function 'forward'
2_train.lua:300: in function 'opfunc'
/install/torch/install/share/lua/5.1/optim/adam.lua:37: in function 'adam'
2_train.lua:406: in main chunk
[C]: in function 'dofile'
main.lua:130: in main chunk
[C]: in function 'dofile'
...tall/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405e90

Thanks for your help.

Angles in camPosList

Hello, can you tell me, how to calculate the real angles value from camPosList.txt? And where the reference point is located?
Greetings

Download question

Hello,the download link of pre-trained models is unavailable. so,could you update it again? Thank you

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.