Giter Club home page Giter Club logo

factored3d's Introduction

Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene

Shubham Tulsiani, Saurabh Gupta, David Fouhey, Alexei A. Efros, Jitendra Malik.

Project Page

Note: Also see 3D-RelNet that improves on this work by incporporating relationships among objects. Teaser Image

Demo and Pre-trained Models

Please check out the interactive notebook which shows reconstructions using the learned models. To run this, you'll first need to follow the installation instructions to download trained models and some pre-requisites.

Training and Evaluating

To train or evaluate the (trained/downloaded) models, it is first required to download the SUNCG dataset and preprocess the data. Please see the detailed README files for Training or Evaluation of models for subsequent instructions.

Citation

If you use this code for your research, please consider citing:

@inProceedings{factored3dTulsiani17,
  title={Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene},
  author = {Shubham Tulsiani
  and Saurabh Gupta
  and David Fouhey
  and Alexei A. Efros
  and Jitendra Malik},
  booktitle={Computer Vision and Pattern Regognition (CVPR)},
  year={2018}
}

factored3d's People

Contributors

dfouhey avatar s-gupta avatar shubhtuls avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

factored3d's Issues

Running Demo

Hi,
I have installed all the required items that are described in installation.md. I just want to run the demo. What I have to do next?

Bad file descriptor when num_workers > 0

Hello @shubhtuls ,

I have tracked the following error OSError: [Errno 9] Bad file descriptor to the point when the suncg dataloader is passed to enumerate() for training. It seems however it only shows up when the num_workers > 0, and suncg_dl_out_codes=True, and from there it seems like a mess to debug. Have you ever come across this?

I am using python 3.6.4 with the latest version of pytorch, and have only processed data from the first 100 houses just to get things started for myself. Below is my code:

flags.DEFINE_integer('max_rois', 5, 'If we have more objects than this per image, we will subsample.')

FLAGS = flags.FLAGS

class LayoutTrainer(train_utils.Trainer):

    def define_model(self):
        return

    def init_dataset(self):
        opts = self.opts
        split_dir = osp.join(opts.suncg_dir, 'splits')
        self.split = suncg_parse.get_split(split_dir, house_names=os.listdir(osp.join(opts.suncg_dir, 'camera')))
        self.dataloader = suncg_data.suncg_data_loader(self.split['train'], opts)

    def define_criterion(self):
        self.criterion = None

    def train(self):

        for i, batch in enumerate(self.dataloader):
            print(i)
            continue

        print('this better?')

def main(_):
    FLAGS.suncg_dl_out_layout = True
    FLAGS.suncg_dl_out_depth = True
    FLAGS.suncg_dl_out_fine_img = True
    FLAGS.suncg_dl_out_codes = True
    FLAGS.suncg_dl_out_voxels = False
    FLAGS.suncg_dl_out_proposals = False

    torch.manual_seed(0)
    trainer = LayoutTrainer(FLAGS)
    trainer.init_dataset()
    trainer.train()

if __name__ == '__main__':
    app.run(main)

The only lines in your code I needed to change from your code is line 288 of suncg_parse.py (changing .has_key() to python3 convention of key in d.keys()) as well as lines 171 and 172 of suncg_parse.py to use the updated key names in the data (i.e. ids instead of node_ids, nPixels instead of n_pixels).

Thanks in advance

Details on reproducing object_autoenc_32

Hi shubtuls,

can you provide more details on how the training setup looked like to produce the pretrained voxel auto-encoder object_autoenc_32, which we download before training the factored representation in training.md? Is there maybe even the script available for creating the auto-encoder on my own?

In the paper you mention using a setup similar to Girdhar et al., but you don't go in detail how that training is setup. Do you also construct scenes only consisting of a single object from different azimuth angles and elevation as they did?

I would be very interested in that, since I am doing research on using other bootstrapping measures to improve the vector representation trained in the bottleneck layer.

The documentation of your code here on GitHub for this paper is otherwise excellent.
Thank you in advance!

Issues training the factored3d network

Hi shubhtuls,

Sorry for the long issue, but my questions require some explaining beforehand.

I have been trying to reproduce the results you obtained for the factored object prediction, but I have been having problems with the loss function not decreasing during training. I wanted to ask if you had a similar experience when training the network. I will first describe my training setup and show some visualizations comparing my model's performance to your results. Afterwards I will specify my questions regarding the model.

Training Setup and Results

Here are my loss functions for each of the training steps (9 epochs of box3d_base, 1 epoch dwr_base, 1 epoch of dwr_shape_ft) using the same hyperparameters as mentioned in training.md. My splits ({train: 28280, val: 3769, test: 5643}) of the SUNCG dataset are slightly smaller than yours ({train: 31127, val: 4150, test: 6226}), but I would still expect to get similar results than you. My splits are not a subset of your provided splits and there is only a small overlap (around 10%-20%) between the houses in each of our validation and test sets. I use the pretrained autoencoder you provided as inital weights for the voxel decoder used in dwr_shape_ft.
(You can click on the images to increase their size in the browser)

losses_box3d 2 -dwr_base-_dwr_shape_ft

I compared my trained model with your provided model from the cachedir based on their precision-recall plots (using pr_plots.py).
The first column is my model on my test set, the second column is your provided model on my test set, and the final column are the results you provided on your test set in the paper.

pr-plots_mynetwork_providednetwork_reportedresults

There is a dip in performance when comparing your model on my test set, but my model greatly underperforms when compared to your model.

Finally, here is also a qualitative comparison of the shape prediction performance of my model (left) to your model (right).

shapepredictioncomparison

For many objects my model produces something like a mean shape (only occupied voxels in the center of the grid) and the output values are often close to 0, which can be seen by the realtively empty grid when setting the cutoff threshold for voxels at 0.5. In some cases my model produces similar results to yours (like the television prediction in the last row).

Questions

  1. Did you have the same experience of the loss function decreasing very little from one epoch to the next as shown in my loss curves above or did it decrease more steadily during you training process?

  2. I noticed that training the network does not require a lot of GPU memory (around 3 GB on a GTX Titan X). I was wondering if you maybe increased the batch size (the training.md mentions a batch size of 8) for the final model you used for evaluation?

  3. When using the same weighting for the loss components as in training.md, the shape loss only makes up a small component of the total loss (as seen in the loss curves above, where the shape loss is often greatly dominated by the rotation and translation loss). Was this also the case for you during training?

  4. Is the model you used for the evaluation in the paper the same as the one you provided in your cachedir?

  5. Do you think that training the dwr_shape_ft for more than 1 epoch can increase the voxel decoder's shape prediction performance?

Sorry for the very long issue, but I hope you can give me some pointers on what might be the problems of my training procedure.

Best regards,
Peter Mortimer

Error Running cli-demo.py

when I am running cli_demo.py I am getting following error
rm: cannot remove '/home/qasim/factored3d/demo/../cachedir/rendering/dwr_shape_ft/*.obj': No such file or directory
Traceback (most recent call last):
File "cli_demo.py", line 56, in
img_factored_cam, img_factored_novel = renderer.render_factored3d(objects, layout)
File "/home/qasim/factored3d/../factored3d/demo/demo_utils.py", line 278, in render_factored3d
return self.render_visuals(self.mesh_dir)
File "/home/qasim/factored3d/../factored3d/demo/demo_utils.py", line 270, in render_visuals
im_view1 = scipy.misc.imread(osp.join(png_dir, 'render_000.png'))
File "/home/qasim/factored3d/venv/local/lib/python2.7/site-packages/numpy/lib/utils.py", line 101, in newfunc
return func(*args, **kwds)
File "/home/qasim/factored3d/venv/local/lib/python2.7/site-packages/scipy/misc/pilutil.py", line 164, in imread
im = Image.open(name)
File "/home/qasim/factored3d/venv/local/lib/python2.7/site-packages/PIL/Image.py", line 2548, in open
fp = builtins.open(filename, "rb")
IOError: [Errno 2] No such file or directory: '/home/qasim/factored3d/demo/../cachedir/rendering/dwr_shape_ft/rendering/render_000.png'

Layout renderings are 0

The layout renderings with scn2img are all 0. Using scnview to visualize layout.json gives an empty display (dark). Below is the output of scnview.

 Time = 0.00 seconds
 # Nodes = 10
 # Lights = 0
 # Materials = 0
 # Brdfs = 0
 # Textures = 0
 # Referenced scenes = 0
Read cameras from /path/to/camera/00052c0562bde7790f8354e6123ae7ff/room_camera.txt ...
  Time = 0.00 seconds
  # Cameras = 20

Both scnview and scn2img work well with the original house.json

Error when running on CPU

Hi,
I am trying to run "cli_demo.py" on my system which does not have any gpu. Is there any way to run it on cpu only?

This is the (partial) error log I'm getting when running cli_demo.py:


Traceback (most recent call last):
  File "cli_demo.py", line 32, in <module>
    tester.init_testing()
  File "/home/jelly/factored3d/../factored3d/nnutils/test_utils.py", line 97, in init_testing
    self.define_model()
  File "/home/jelly/factored3d/../factored3d/demo/demo_utils.py", line 131, in define_model
    self.load_oc3d_model()
  File "/home/jelly/factored3d/../factored3d/demo/demo_utils.py", line 91, in load_oc3d_model
    self.oc3d_model = self.oc3d_model.cuda(device_id=self.opts.gpu_id)
.
.
.
AssertionError: 
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx

Is there anyone reproduce table 1 result?

Is there anyone reproduce table 1 result? I test on parts of the test data, and got the following AP values, which is much lower than the one from paper.

metric shape rot translation scale
method %(IoU > 0.25) %(Err < 30) %(Err < 1m) %(err<0.5)
reproduction 48.37 50.171 62.589 48.138
(paper) 70.7 74.6 87.3 85.1

My settings are using the ground truth bounding boxes as test data proposals, and shape %(IoU > 0.25) as AP(box_iou=0, shape_threshold=0.25, rotation_threshold=400, translation_threshold=1000, scale_threshold=100), rot %(Err<30) as AP(box_iou=0, shape_threshold=0, rotation_threshold=30, translation_threshold=1000, scale_threshold=100) similar for translation and scale. I'm using the dwr.py script.

Download Links to the Berkeley website are outdated

Hi Shubham,

Most of the links referred in suncg_data.md are outdated. They are still linking to your Berkeley page, although it seems that you are relocating your content over to your GitHub page.

Can you please make the following files available again to allow further research on this project:

  • suncg_split.pkl
  • layout.tar.gz
  • ModelCategoryMappingEdited.csv

Thanks a lot,
Peter

Runtime error while running demo

hi, I'm getting runtime error while running cli_demo.py, I ran on CPU by removing coda() calls and used pytorch 0.2.0 and python 2.7

Traceback (most recent call last):
File "cli_demo.py", line 55, in
objects, layout = tester.predict_factored3d()
File "/Users/santhosh/Documents/factored3d/../factored3d/demo/demo_utils.py", line 210, in predict_factored3d
(self.input_imgs_fine, self.input_imgs, self.rois))
File "/Users/santhosh/Documents/factored3d/../factored3d/nnutils/oc_net.py", line 216, in forward
roi_img_feat = self.roi_pool.forward(img_feat_fine, rois_inp)
File "/Users/santhosh/Documents/factored3d/../factored3d/nnutils/roi_pool_py.py", line 49, in forward
torch.max(data[:, hstart:hend, wstart:wend], 1)[0], 1)[0].view(-1)
File "/Users/santhosh/Desktop/iisc/san/lib/python2.7/site-packages/torch/autograd/variable.py", line 85, in setitem
return SetItem.apply(self, key, value)
File "/Users/santhosh/Desktop/iisc/san/lib/python2.7/site-packages/torch/autograd/_functions/tensor.py", line 43, in forward
i._set_index(ctx.index, value)
RuntimeError: inconsistent tensor size, expected tensor [256] and src [512] to have the same number of elements, but got 256 and 512 elements respectively at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorCopy.c:86

Runtime error with cli_demo.py

Hi, I am getting this Runtime error: sizes do not match
when I am trying to run cli_demo.py
Here is the description:

----> 1 objects, layout = tester.predict_factored3d()

/vulcan/scratch/luyu/source/others/factored3d/../factored3d/demo/demo_utils.pyc in predict_factored3d(self)
211 print(self.rois)
212 codes_pred_all, labels_pred = self.oc3d_model.forward(
--> 213 (self.input_imgs_fine, self.input_imgs, self.rois))
214 scores_pred = labels_pred.cpu().data.numpy()
215 bboxes_pred = self.rois.data.cpu().numpy()[:, 1:]

/vulcan/scratch/luyu/source/others/factored3d/../factored3d/nnutils/oc_net.pyc in forward(self, imgs_rois)
214 img_feat_fine = self.resnet_conv_fine.forward(imgs_inp_fine)
215
--> 216 roi_img_feat = self.roi_pool.forward(img_feat_fine, rois_inp)
217 roi_img_feat = roi_img_feat.view(roi_img_feat.size(0), -1)
218

/vulcan/scratch/luyu/source/others/factored3d/../factored3d/nnutils/roi_pool_py.py in forward(self, features, rois)
47 data = features[batch_ind]
48 outputs[roi_ind, :, ph, pw] = torch.max(
---> 49 torch.max(data[:, hstart:hend, wstart:wend], 1)[0], 1)[0].view(-1)
50
51 return outputs

/vulcan/scratch/luyu/anaconda3/envs/t2/lib/python2.7/site-packages/torch/autograd/variable.pyc in setitem(self, key, value)
78 else:
79 if isinstance(value, Variable):
---> 80 return SetItem(key)(self, value)
81 else:
82 return SetItem(key, value)(self)

/vulcan/scratch/luyu/anaconda3/envs/t2/lib/python2.7/site-packages/torch/autograd/_functions/tensor.pyc in forward(self, i, value)
37 else: # value is Tensor
38 self.value_size = value.size()
---> 39 i._set_index(self.index, value)
40 return i
41
RuntimeError: sizes do not match at /opt/conda/conda-bld/pytorch_1501972792122/work/pytorch-0.1.12/torch/lib/THC/THCTensorCopy.cu:31

Does it require python 2.7 or 3.X?

Hi, I built two different conda envs, one using python2.7, another 3.6.5.
With the same packages installed.

If I am using the python2.7 I am getting the same error as qasim's earlier.
"""
No such file or directory: '/vulcan/scratch/luyu/source/others/factored3d/demo/../cachedir/rendering/dwr_shape_ft/rendering/render_000.png'
"""
It seems like a python version problem based on your conversation.
However if I use the python3.6.5 environment, I get
"""
ImportError: dynamic module does not define module export function (PyInit_bbox_utils)
"""
which also seems like a version problem.

Error in running demo

while trying to run demo
~/factored3d$ python3 cli_demo.py
i get error
Traceback (most recent call last):
File "cli_demo.py", line 12, in
from factored3d.demo import demo_utils
ImportError: No module named 'factored3d'

suncg dataset for help

do you have complete suncg dataset ?i am looking for it ,if you have ,can you share for me?thank you very much!

binvox

How to use binvox? when i run the "python voxelize_objects.py", it said that " No module named 'binvox'"

The problem of generating node images with onscreen mode

I follow the instructions of preprocessing instructions to generate the node images. And I use the onscreen mode. However, too many of the generated node images are just fully black images or sometimes the random screen shot of my desktop. Is there any others encountering the same problem?
Could you please give me some suggestions?

Thanks a lot!
Best Regards

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.