https://people.eecs.berkeley.edu/~shubhtuls/ redirects to https://shubhtuls.github.io , and ofc. there's nothing to be found ;(

Hi,
I have installed all the required items that are described in installation.md. I just want to run the demo. What I have to do next?

Hello @shubhtuls ,

I have tracked the following error OSError: [Errno 9] Bad file descriptor to the point when the suncg dataloader is passed to enumerate() for training. It seems however it only shows up when the num_workers > 0, and suncg_dl_out_codes=True, and from there it seems like a mess to debug. Have you ever come across this?

I am using python 3.6.4 with the latest version of pytorch, and have only processed data from the first 100 houses just to get things started for myself. Below is my code:

flags.DEFINE_integer('max_rois', 5, 'If we have more objects than this per image, we will subsample.')

FLAGS = flags.FLAGS

class LayoutTrainer(train_utils.Trainer):

    def define_model(self):
        return

    def init_dataset(self):
        opts = self.opts
        split_dir = osp.join(opts.suncg_dir, 'splits')
        self.split = suncg_parse.get_split(split_dir, house_names=os.listdir(osp.join(opts.suncg_dir, 'camera')))
        self.dataloader = suncg_data.suncg_data_loader(self.split['train'], opts)

    def define_criterion(self):
        self.criterion = None

    def train(self):

        for i, batch in enumerate(self.dataloader):
            print(i)
            continue

        print('this better?')

def main(_):
    FLAGS.suncg_dl_out_layout = True
    FLAGS.suncg_dl_out_depth = True
    FLAGS.suncg_dl_out_fine_img = True
    FLAGS.suncg_dl_out_codes = True
    FLAGS.suncg_dl_out_voxels = False
    FLAGS.suncg_dl_out_proposals = False

    torch.manual_seed(0)
    trainer = LayoutTrainer(FLAGS)
    trainer.init_dataset()
    trainer.train()

if __name__ == '__main__':
    app.run(main)

The only lines in your code I needed to change from your code is line 288 of suncg_parse.py (changing .has_key() to python3 convention of key in d.keys()) as well as lines 171 and 172 of suncg_parse.py to use the updated key names in the data (i.e. ids instead of node_ids, nPixels instead of n_pixels).

Thanks in advance

Hi shubtuls,

can you provide more details on how the training setup looked like to produce the pretrained voxel auto-encoder object_autoenc_32, which we download before training the factored representation in training.md? Is there maybe even the script available for creating the auto-encoder on my own?

In the paper you mention using a setup similar to Girdhar et al., but you don't go in detail how that training is setup. Do you also construct scenes only consisting of a single object from different azimuth angles and elevation as they did?

I would be very interested in that, since I am doing research on using other bootstrapping measures to improve the vector representation trained in the bottleneck layer.

The documentation of your code here on GitHub for this paper is otherwise excellent.
Thank you in advance!

Thanks a ton for the amazing work. The link toward a trained mode: https://people.eecs.berkeley.edu/~shubhtuls/cachedir/factored3d/cachedir.tar.gz is borken somehow.

Hi shubhtuls,

Sorry for the long issue, but my questions require some explaining beforehand.

I have been trying to reproduce the results you obtained for the factored object prediction, but I have been having problems with the loss function not decreasing during training. I wanted to ask if you had a similar experience when training the network. I will first describe my training setup and show some visualizations comparing my model's performance to your results. Afterwards I will specify my questions regarding the model.

Training Setup and Results

Here are my loss functions for each of the training steps (9 epochs of box3d_base, 1 epoch dwr_base, 1 epoch of dwr_shape_ft) using the same hyperparameters as mentioned in training.md. My splits ({train: 28280, val: 3769, test: 5643}) of the SUNCG dataset are slightly smaller than yours ({train: 31127, val: 4150, test: 6226}), but I would still expect to get similar results than you. My splits are not a subset of your provided splits and there is only a small overlap (around 10%-20%) between the houses in each of our validation and test sets. I use the pretrained autoencoder you provided as inital weights for the voxel decoder used in dwr_shape_ft.
(You can click on the images to increase their size in the browser)

I compared my trained model with your provided model from the cachedir based on their precision-recall plots (using pr_plots.py).
The first column is my model on my test set, the second column is your provided model on my test set, and the final column are the results you provided on your test set in the paper.

There is a dip in performance when comparing your model on my test set, but my model greatly underperforms when compared to your model.

Finally, here is also a qualitative comparison of the shape prediction performance of my model (left) to your model (right).

For many objects my model produces something like a mean shape (only occupied voxels in the center of the grid) and the output values are often close to 0, which can be seen by the realtively empty grid when setting the cutoff threshold for voxels at 0.5. In some cases my model produces similar results to yours (like the television prediction in the last row).

Questions

1. Did you have the same experience of the loss function decreasing very little from one epoch to the next as shown in my loss curves above or did it decrease more steadily during you training process?

2. I noticed that training the network does not require a lot of GPU memory (around 3 GB on a GTX Titan X). I was wondering if you maybe increased the batch size (the training.md mentions a batch size of 8) for the final model you used for evaluation?

3. When using the same weighting for the loss components as in training.md, the shape loss only makes up a small component of the total loss (as seen in the loss curves above, where the shape loss is often greatly dominated by the rotation and translation loss). Was this also the case for you during training?

4. Is the model you used for the evaluation in the paper the same as the one you provided in your cachedir?

5. Do you think that training the dwr_shape_ft for more than 1 epoch can increase the voxel decoder's shape prediction performance?

Sorry for the very long issue, but I hope you can give me some pointers on what might be the problems of my training procedure.

Best regards,
Peter Mortimer

when i use cmd

mv factored3d/utils/bbox_utils.so ./

i get error mv: cannot stat 'factored3d/utils/bbox_utils.so': No such file or directory

when I am running cli_demo.py I am getting following error
rm: cannot remove '/home/qasim/factored3d/demo/../cachedir/rendering/dwr_shape_ft/*.obj': No such file or directory
Traceback (most recent call last):
File "cli_demo.py", line 56, in
img_factored_cam, img_factored_novel = renderer.render_factored3d(objects, layout)
File "/home/qasim/factored3d/../factored3d/demo/demo_utils.py", line 278, in render_factored3d
return self.render_visuals(self.mesh_dir)
File "/home/qasim/factored3d/../factored3d/demo/demo_utils.py", line 270, in render_visuals
im_view1 = scipy.misc.imread(osp.join(png_dir, 'render_000.png'))
File "/home/qasim/factored3d/venv/local/lib/python2.7/site-packages/numpy/lib/utils.py", line 101, in newfunc
return func(*args, **kwds)
File "/home/qasim/factored3d/venv/local/lib/python2.7/site-packages/scipy/misc/pilutil.py", line 164, in imread
im = Image.open(name)
File "/home/qasim/factored3d/venv/local/lib/python2.7/site-packages/PIL/Image.py", line 2548, in open
fp = builtins.open(filename, "rb")
IOError: [Errno 2] No such file or directory: '/home/qasim/factored3d/demo/../cachedir/rendering/dwr_shape_ft/rendering/render_000.png'

The layout renderings with scn2img are all 0. Using scnview to visualize layout.json gives an empty display (dark). Below is the output of scnview.

 Time = 0.00 seconds
 # Nodes = 10
 # Lights = 0
 # Materials = 0
 # Brdfs = 0
 # Textures = 0
 # Referenced scenes = 0
Read cameras from /path/to/camera/00052c0562bde7790f8354e6123ae7ff/room_camera.txt ...
  Time = 0.00 seconds
  # Cameras = 20

Both scnview and scn2img work well with the original house.json

Hi,
I am trying to run "cli_demo.py" on my system which does not have any gpu. Is there any way to run it on cpu only?

This is the (partial) error log I'm getting when running cli_demo.py:

Traceback (most recent call last):
  File "cli_demo.py", line 32, in <module>
    tester.init_testing()
  File "/home/jelly/factored3d/../factored3d/nnutils/test_utils.py", line 97, in init_testing
    self.define_model()
  File "/home/jelly/factored3d/../factored3d/demo/demo_utils.py", line 131, in define_model
    self.load_oc3d_model()
  File "/home/jelly/factored3d/../factored3d/demo/demo_utils.py", line 91, in load_oc3d_model
    self.oc3d_model = self.oc3d_model.cuda(device_id=self.opts.gpu_id)
.
.
.
AssertionError: 
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx

Is there anyone reproduce table 1 result? I test on parts of the test data, and got the following AP values, which is much lower than the one from paper.

My settings are using the ground truth bounding boxes as test data proposals, and shape %(IoU > 0.25) as AP(box_iou=0, shape_threshold=0.25, rotation_threshold=400, translation_threshold=1000, scale_threshold=100), rot %(Err<30) as AP(box_iou=0, shape_threshold=0, rotation_threshold=30, translation_threshold=1000, scale_threshold=100) similar for translation and scale. I'm using the dwr.py script.

Hi Shubham,

Most of the links referred in suncg_data.md are outdated. They are still linking to your Berkeley page, although it seems that you are relocating your content over to your GitHub page.

Can you please make the following files available again to allow further research on this project:

• suncg_split.pkl
• layout.tar.gz
• ModelCategoryMappingEdited.csv

Thanks a lot,
Peter

Hi,
I am new to Linux. When I try to run the command for downloading the above dataset, I get following error

wget https://download.pytorch.org/models/resnet18-5c106cde.pth -O ~/.torch/models/resnet18-5c106cde.pth
/home/qasim/.torch/models/resnet18-5c106cde.pth: No such file or directory

hi, I'm getting runtime error while running cli_demo.py, I ran on CPU by removing coda() calls and used pytorch 0.2.0 and python 2.7

Traceback (most recent call last):
File "cli_demo.py", line 55, in
objects, layout = tester.predict_factored3d()
File "/Users/santhosh/Documents/factored3d/../factored3d/demo/demo_utils.py", line 210, in predict_factored3d
(self.input_imgs_fine, self.input_imgs, self.rois))
File "/Users/santhosh/Documents/factored3d/../factored3d/nnutils/oc_net.py", line 216, in forward
roi_img_feat = self.roi_pool.forward(img_feat_fine, rois_inp)
File "/Users/santhosh/Documents/factored3d/../factored3d/nnutils/roi_pool_py.py", line 49, in forward
torch.max(data[:, hstart:hend, wstart:wend], 1)[0], 1)[0].view(-1)
File "/Users/santhosh/Desktop/iisc/san/lib/python2.7/site-packages/torch/autograd/variable.py", line 85, in setitem
return SetItem.apply(self, key, value)
File "/Users/santhosh/Desktop/iisc/san/lib/python2.7/site-packages/torch/autograd/_functions/tensor.py", line 43, in forward
i._set_index(ctx.index, value)
RuntimeError: inconsistent tensor size, expected tensor [256] and src [512] to have the same number of elements, but got 256 and 512 elements respectively at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorCopy.c:86

Hi, I am getting this Runtime error: sizes do not match
when I am trying to run cli_demo.py
Here is the description:

----> 1 objects, layout = tester.predict_factored3d()

/vulcan/scratch/luyu/source/others/factored3d/../factored3d/demo/demo_utils.pyc in predict_factored3d(self)
211 print(self.rois)
212 codes_pred_all, labels_pred = self.oc3d_model.forward(
--> 213 (self.input_imgs_fine, self.input_imgs, self.rois))
214 scores_pred = labels_pred.cpu().data.numpy()
215 bboxes_pred = self.rois.data.cpu().numpy()[:, 1:]

/vulcan/scratch/luyu/source/others/factored3d/../factored3d/nnutils/oc_net.pyc in forward(self, imgs_rois)
214 img_feat_fine = self.resnet_conv_fine.forward(imgs_inp_fine)
215
--> 216 roi_img_feat = self.roi_pool.forward(img_feat_fine, rois_inp)
217 roi_img_feat = roi_img_feat.view(roi_img_feat.size(0), -1)
218

/vulcan/scratch/luyu/source/others/factored3d/../factored3d/nnutils/roi_pool_py.py in forward(self, features, rois)
47 data = features[batch_ind]
48 outputs[roi_ind, :, ph, pw] = torch.max(
---> 49 torch.max(data[:, hstart:hend, wstart:wend], 1)[0], 1)[0].view(-1)
50
51 return outputs

/vulcan/scratch/luyu/anaconda3/envs/t2/lib/python2.7/site-packages/torch/autograd/variable.pyc in setitem(self, key, value)
78 else:
79 if isinstance(value, Variable):
---> 80 return SetItem(key)(self, value)
81 else:
82 return SetItem(key, value)(self)

/vulcan/scratch/luyu/anaconda3/envs/t2/lib/python2.7/site-packages/torch/autograd/_functions/tensor.pyc in forward(self, i, value)
37 else: # value is Tensor
38 self.value_size = value.size()
---> 39 i._set_index(self.index, value)
40 return i
41
RuntimeError: sizes do not match at /opt/conda/conda-bld/pytorch_1501972792122/work/pytorch-0.1.12/torch/lib/THC/THCTensorCopy.cu:31

Hi, I built two different conda envs, one using python2.7, another 3.6.5.
With the same packages installed.

If I am using the python2.7 I am getting the same error as qasim's earlier.
"""
No such file or directory: '/vulcan/scratch/luyu/source/others/factored3d/demo/../cachedir/rendering/dwr_shape_ft/rendering/render_000.png'
"""
It seems like a python version problem based on your conversation.
However if I use the python3.6.5 environment, I get
"""
ImportError: dynamic module does not define module export function (PyInit_bbox_utils)
"""
which also seems like a version problem.

while trying to run demo
~/factored3d$ python3 cli_demo.py
i get error
Traceback (most recent call last):
File "cli_demo.py", line 12, in
from factored3d.demo import demo_utils
ImportError: No module named 'factored3d'

do you have complete suncg dataset ?i am looking for it ,if you have ,can you share for me?thank you very much!

Hello, I want to test your code with some images. I have run the demo ;however, I couldn't find any way to calculate proposal.mat file for a custom image file. Is it possible ?

How to use binvox? when i run the "python voxelize_objects.py", it said that " No module named 'binvox'"

I follow the instructions of preprocessing instructions to generate the node images. And I use the onscreen mode. However, too many of the generated node images are just fully black images or sometimes the random screen shot of my desktop. Is there any others encountering the same problem?
Could you please give me some suggestions?

Thanks a lot!
Best Regards