3dlg-hcvc / plan2scene Goto Github PK

View Code? Open in Web Editor NEW

425.0 9.0 55.0 2.63 MB

Official implementation of the paper Plan2Scene.

License: MIT License

Python 96.94% C++ 0.26% Cuda 1.75% Shell 1.04%

texture-synthesis indoor-reconstruction 3d-reconstruction computer-vision machine-learning

plan2scene's Introduction

Plan2Scene

Official repository of the paper:

Plan2Scene: Converting floorplans to 3D scenes

Madhawa Vidanapathirana, Qirui Wu, Yasutaka Furukawa, Angel X. Chang , Manolis Savva

[Paper, Project Page, Google Colab Demo]

In the Plan2Scene task, we produce a textured 3D mesh of a residence from a floorplan and set of photos.

Dependencies

We use a conda environment initialized as described here.
Setup the command line library of Embark Studios texture-synthesis project.
1. You can download a pre-built binary available here. Alternatively, you may build from the source.
2. Download the seam mask available here.
3. Rename ./conf/plan2scene/seam_correct-example.json to 'seam_correct.json' and update the paths to the texture synthesis command line library binary, and the seam mask.

Use 'code/src' as the source root when running python scripts.

export PYTHONPATH=./code/src

Data

Rent3D++ dataset

Download and copy the Rent3D++ dataset to the [PROJECT_ROOT]/data directory. The data organization is described here.
[Optional] We have provided 3D scenes pre-populated with CAD models of objects. If you wish to re-populate these scenes using the Object Placement approach we use, follow the instructions here.
To replicate our results, you should use the pre-extracted crops we provide. These crops are provided with the Rent3D++ dataset and are copied to the ./data/processed/surface_crops directory.

[Optional] If you wish to extract new crops instead of using these provided crops, following these instructions.

Select ground truth reference crops and populate photo room assignment lists.

# Select ground truth reference crops.
python code/scripts/plan2scene/preprocessing/generate_reference_crops.py ./data/processed/gt_reference/train ./data/input/photo_assignments/train train
python code/scripts/plan2scene/preprocessing/generate_reference_crops.py ./data/processed/gt_reference/val ./data/input/photo_assignments/val val
python code/scripts/plan2scene/preprocessing/generate_reference_crops.py ./data/processed/gt_reference/test ./data/input/photo_assignments/test test 

# We evaluate Plan2Scene by simulating photo un-observations.
# Generate photoroom.csv files considering different photo un-observation ratios.
python code/scripts/plan2scene/preprocessing/generate_unobserved_photo_assignments.py ./data/processed/photo_assignments/train ./data/input/photo_assignments/train ./data/input/unobserved_photos.json train
python code/scripts/plan2scene/preprocessing/generate_unobserved_photo_assignments.py ./data/processed/photo_assignments/val ./data/input/photo_assignments/val ./data/input/unobserved_photos.json val
python code/scripts/plan2scene/preprocessing/generate_unobserved_photo_assignments.py ./data/processed/photo_assignments/test ./data/input/photo_assignments/test ./data/input/unobserved_photos.json test

[Optional] Stationary Textures Dataset - We use one of the following datasets to train the texture synthesis model. Not required if you are using pre-trained models.
- Version 1: We use this dataset in our CVPR paper. Details are available here.
- Version 2: Updated textures dataset which provides improved results on the Rent3D++ dataset. Details are available here.
[Optional] Substance Mapped Textures dataset. Only used by the retrieve baseline.

Pretrained models

Pretrained models are available here.

Inference on Rent3D++ dataset

Download and pre-process the Rent3D++ dataset as described in the data section.
Setup a pretrained model or train a new Plan2Scene network.

Synthesize textures for observed surfaces using the VGG textureness score.

# For test data without simulating photo unobservations. (drop = 0.0)
python code/scripts/plan2scene/preprocessing/fill_room_embeddings.py ./data/processed/texture_gen/test/drop_0.0 test --drop 0.0
python code/scripts/plan2scene/crop_select/vgg_crop_selector.py ./data/processed/vgg_crop_select/test/drop_0.0 ./data/processed/texture_gen/test/drop_0.0 test --drop 0.0
# Results are stored at ./data/processed/vgg_crop_select/test/drop_0.0

Propagate textures to unobserved surfaces using our texture propagation network.

python code/scripts/plan2scene/texture_prop/gnn_texture_prop.py ./data/processed/gnn_prop/test/drop_0.0 ./data/processed/vgg_crop_select/test/drop_0.0 test GNN_PROP_CONF_PATH GNN_PROP_CHECKPOINT_PATH --keep-existing-predictions --drop 0.0

To preview results, follow the instructions below.

Previewing outputs

Complete inference steps.

Correct seams of predicted textures and make them tileable.

# For test data without simulating photo unobservations.
python code/scripts/plan2scene/postprocessing/seam_correct_textures.py ./data/processed/gnn_prop/test/drop_0.0/tileable_texture_crops ./data/processed/gnn_prop/test/drop_0.0/texture_crops test --drop 0.0

Generate .scene.json files with embedded textures using embed_textures.py. A scene.json file describes the 3D geometry of a house. It can be previewed via a browser using the 'scene-viewer' of SmartScenesToolkit (You will have to clone and build the SmartScenesToolkit).

# For test data without simulating photo unobservations.
python code/scripts/plan2scene/postprocessing/embed_textures.py ./data/processed/gnn_prop/test/drop_0.0/archs ./data/processed/gnn_prop/test/drop_0.0/tileable_texture_crops test --drop 0.0
# scene.json files are created in the ./data/processed/gnn_prop/test/drop_0.0/archs directory.

Render .scene.json files as .pngs using render_house_jsons.py.

Download and build the SmartScenesToolkit.
Rename ./conf/render-example.json to ./conf/render.json and update its fields to point to scene-toolkit.

Run the following command to generate previews.

CUDA_VISIBLE_DEVICES=0 python code/scripts/plan2scene/render_house_jsons.py ./data/processed/gnn_prop/test/drop_0.0/archs --scene-json
# A .png file is created for each .scene.json file in the ./data/processed/gnn_prop/test/drop_0.0/archs directory.

Generate qualitative result pages with previews using preview_houses.py.

python code/scripts/plan2scene/preview_houses.py ./data/processed/gnn_prop/test/drop_0.0/previews ./data/processed/gnn_prop/test/drop_0.0/archs ./data/input/photos test --textures-path ./data/processed/gnn_prop/test/drop_0.0/tileable_texture_crops 0.0
# Open ./data/processed/gnn_prop/test/drop_0.0/previews/preview.html

Test on Rent3D++ dataset

[Optional] Download a pre-trained model or train the substance classifier used by the Subs metric. Training instructions are available here. Pre-trained weights are available here. Skip this step to omit the Subs metric.

Generate overall evaluation report at 60% photo unobservations. We used this setting in paper evaluations.

# Synthesize textures for observed surfaces using the VGG textureness score.
# For the case: 60% (i.e. 0.6) of the photos unobserved. 
python code/scripts/plan2scene/preprocessing/fill_room_embeddings.py ./data/processed/texture_gen/test/drop_0.6 test --drop 0.6
python code/scripts/plan2scene/crop_select/vgg_crop_selector.py ./data/processed/vgg_crop_select/test/drop_0.6 ./data/processed/texture_gen/test/drop_0.6 test --drop 0.6

# Propagate textures to un-observed surfaces using our GNN.
# For the case: 60% (i.e. 0.6) of the photos unobserved.
python code/scripts/plan2scene/texture_prop/gnn_texture_prop.py ./data/processed/gnn_prop/test/drop_0.6 ./data/processed/vgg_crop_select/test/drop_0.6 test GNN_PROP_CONF_PATH GNN_PROP_CHECKPOINT_PATH --keep-existing-predictions --drop 0.6

# Correct seams of texture crops and make them tileable.
# For test data where 60% of photos are unobserved.
python code/scripts/plan2scene/postprocessing/seam_correct_textures.py ./data/processed/gnn_prop/test/drop_0.6/tileable_texture_crops ./data/processed/gnn_prop/test/drop_0.6/texture_crops test --drop 0.6

# Generate overall results at 60% simulated photo unobservations.
python code/scripts/plan2scene/test.py ./data/processed/gnn_prop/test/drop_0.6/tileable_texture_crops ./data/processed/gt_reference/test/texture_crops test

Generate evaluation report for observed surfaces. No simulated unobservation of photos. We used this setting in paper evaluations.

# Run inference on using drop=0.0.
python code/scripts/plan2scene/preprocessing/fill_room_embeddings.py ./data/processed/texture_gen/test/drop_0.0 test --drop 0.0
python code/scripts/plan2scene/crop_select/vgg_crop_selector.py ./data/processed/vgg_crop_select/test/drop_0.0 ./data/processed/texture_gen/test/drop_0.0 test --drop 0.0

# Correct seams of texture crops and make them tileable by running seam_correct_textures.py.
python code/scripts/plan2scene/postprocessing/seam_correct_textures.py ./data/processed/vgg_crop_select/test/drop_0.0/tileable_texture_crops ./data/processed/vgg_crop_select/test/drop_0.0/texture_crops test --drop 0.0

# Generate evaluation results for observed surfaces.
python code/scripts/plan2scene/test.py ./data/processed/vgg_crop_select/test/drop_0.0/tileable_texture_crops ./data/processed/gt_reference/test/texture_crops test

Generate evaluation report for unobserved surfaces at 60% photo unobservations. We used this setting in the paper evaluations.

# It is assumed that the user has already generated the overall report at 0.6 drop fraction.

# Generate results on unobserved surfaces at 60% simulated photo unobservations.
python code/scripts/plan2scene/test.py ./data/processed/gnn_prop/test/drop_0.6/tileable_texture_crops ./data/processed/gt_reference/test/texture_crops test --exclude-prior-predictions ./data/processed/vgg_crop_select/test/drop_0.6/texture_crops

Generate evaluation report on FID metric as described here.

Inference on custom data

If you have scanned images of floorplans, you can use raster-to-vector to convert those floorplan images to a vector format. Then, follow the instructions here to create textured 3D meshes of houses.

If you have floorplan vectors in another format, you can convert them to the raster-to-vector annotation format. Then, follow the same instructions as before to create textured 3D meshes of houses. The R2V annotation format is explained with examples in the data section of the raster-to-vector repository.

Training a new Plan2Scene network

Plan2Scene consists of two trainable components, 1) the texture synthesis stage and 2) the texture propagation stage. Each stage is trained separately. The training procedure is as follows.

Train the texture synthesis stage as described here.
Train the texture propagation stage as described here.

Baseline Models

The baseline models are available here.

plan2scene's People

Contributors

Stargazers

Watchers

plan2scene's Issues

Error occurred while executing open colab file

Hello.

I'm incredibly happy with what you've been working on.

First of all I am constantly failing to run the provided colab file to understand your work.

Should I use only 1.7.1 and 0.8.2 for torch and torchvision versions?

If I want to use it on a local PC, is there a guide document for this?

Pre-trained model (weights) link is broken?

Hi guys! It seems that the trained model link is broken, do you mind updating it?
Hoped to use your model for the hackathon (don't need to be a SOTA best model but something would be very handy).

Also, I might have missed it, but how long/hard is it to train your model, and what hardware do you use?
Thank you in advance!

Rent3D dataset no longer available

While I was able to download the Rent3D++ dataset, the original Rent3D dataset link at http://www.cs.toronto.edu/~fidler/projects/rent3D.html seems to be broken. If my understanding is correct, that data would be necessary to retrain this model. Does anyone have an alternative download link?

I've already e-mailed the Rent3D researchers through their e-mail links on that page and if I get a response I'll update this issue with a new link.

compatible versions for google colab code

please share the compatible versions of python, torch and torchvision to run the google colab code
not able to run below attached colab with python 3.7.16 and torch and torchvision of given version
https://colab.research.google.com/drive/1lDkbfIV0drR1o9D0WYzoWeRskB91nXHq?usp=sharing#scrollTo=fgRuHJJpYOKT

How to run pre-trained model on custom data?

I need to run the pre-trained models on some custom data I have. How do I do that? Do I just replace the data in appropriate folders of Rent3D++ data set? Reading the README, I could not understand how to use this repo out of the box to generate 3d plans for custom data. Awesome work btw:)

For example:
Suppose I already have vectorised floor plan data along with photos mapped to some subset of rooms, and now want to use plan2scene using a pre-trained model for now, how can I do that?

colab

Hi, great work, can you please add a google colab for inference?

SmartScenesToolkit

Hello! Why cannot the SmartScenesToolkit display 3D models of doors, windows and objects?

python code/scripts/plan2scene/place_object_cad_models.py ./data/processed/full_archs/test ./data/processed/archs_with_hole_models/test ./data/input/object_aabbs/test/

Side Issue - SSTK Docker?

It's obviously beyond this work/repo responsibility, but I had such a great experience with your colab notebook - unless I got the result and tried to view it using smart scene toolkit - and it is hard to install and the existing Dockerfile fails to compile. Can you share your docker image, by any chance>

I can't get the dataset.

Hi, I have filled out the form but have not received a response. How can I get the dataset please?

Google Colab notebook is not working

@madhawav
What is the result of checking Google Colab notebook?

In Google Colab notebook I checked the following way, but without success.

python==3.7.13 torch==1.7.1+cu110 torchvision==0.8.2+cu110
--> Failed building wheel for torch-scatter, torch-sparse, but Successfully built torch-geometric
--> I can't proceed any further.
python==3.6.9 torch==1.8.0+cu111 torchvision==0.9.0+cu111
--> Successfully installed torch-scatter-2.0.9,Successfully installed torch-sparse-0.6.13,Successfully installed torch-geometric-2.0.4
--> Fail to import torch_sparse,torch_geometric
--> I can't proceed any further.
python==3.7.13 torch==1.11.0+cu113 torchvision==0.12.0+cu113
-->Successfully installed torch-scatter,torch_geometric,torch_sparse
-->No module named 'torchvision.models.utils' in /plan2scene/code/src/plan2scene/texture_gen/nets/vgg.py. I got this error and fixed it with torch.hub because 'torchvision.models.utils' is deprecated in torch 1.11.0.
--> When data is uploaded in"Task:Upload rectified surface crops extracted from photos." step, the photo_file_name directory is created under rectified_crops and copied. After moving the data to rectified_crops, you can see the texture in the step of "Task: Let's preview the data you have provided."
--> "# Compute texture embeddings for observed surfaces (Code adapted from ./code/scripts/preprocessing/fill_room_embeddigs.py)" step have error below like.
--> I can't proceed any further.

/usr/local/lib/python3.7/dist-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2228.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-31-42fcee0d7001>](https://localhost:8080/#) in <module>()
      4       for candidate_key, image_description in room.surface_textures[surface].items():
      5         image = image_description.image
----> 6         emb, loss = tg_predictor.predict_embs([image])
      7         room.surface_embeddings[surface][candidate_key] = emb
      8         room.surface_losses[surface][candidate_key] = loss

10 frames
[/content/plan2scene/code/src/plan2scene/texture_gen/predictor.py](https://localhost:8080/#) in predict_embs(self, sample_image_crops)
     81             predictor_result = self.predict(unsigned_images.to(self.conf.device),
     82                                             unsigned_hsv_images.to(self.conf.device),
---> 83                                             self.get_position(), combined_emb=None, train=False)
     84 
     85             # Compute loss between synthesized texture and conditioned image

[/content/plan2scene/code/src/plan2scene/texture_gen/predictor.py](https://localhost:8080/#) in predict(self, unsigned_images, unsigned_hsv_images, sample_pos, train, combined_emb)
    272             network_input, base_color = self._compute_network_input(unsigned_images, unsigned_hsv_images, additional_params)
    273             network_out, network_emb, substance_out = self.net(network_input, sample_pos.to(self.conf.device),
--> 274                                                                self.seed)
    275         else:
    276             # Predict using the combined_emb. Skip encoder.

[/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

[/content/plan2scene/code/src/plan2scene/texture_gen/nets/neural_texture/texture_gen.py](https://localhost:8080/#) in forward(self, image_gt, position, seed, weights_bottleneck)
     87 
     88         input_mlp = torch.cat([z_encoding, noise], dim=1)
---> 89         image_out = self.decoder(input_mlp)
     90         image_out = torch.tanh(image_out)
     91 

[/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

[/content/plan2scene/code/src/plan2scene/texture_gen/nets/neural_texture/mlp.py](https://localhost:8080/#) in forward(self, input)
     32     def forward(self, input):
     33 
---> 34         input_z = self.first_conv(input)
     35         output = input_z
     36         for idx, block in enumerate(self.res_blocks):

[/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

[/content/plan2scene/code/src/plan2scene/texture_gen/nets/core_modules/standard_block.py](https://localhost:8080/#) in forward(self, input, style)
     67             output = self.norm(output, style)
     68         else:
---> 69             output = self.layer(input)
     70 
     71             # output = self.norm(output)

[/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

[/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py](https://localhost:8080/#) in forward(self, input)
    445 
    446     def forward(self, input: Tensor) -> Tensor:
--> 447         return self._conv_forward(input, self.weight, self.bias)
    448 
    449 class Conv3d(_ConvNd):

[/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py](https://localhost:8080/#) in _conv_forward(self, input, weight, bias)
    442                             _pair(0), self.dilation, self.groups)
    443         return F.conv2d(input, weight, bias, self.stride,
--> 444                         self.padding, self.dilation, self.groups)
    445 
    446     def forward(self, input: Tensor) -> Tensor:

TypeError: conv2d() received an invalid combination of arguments - got (Tensor, Parameter, Parameter, tuple, tuple, tuple, int), but expected one of:
 * (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, tuple of ints padding, tuple of ints dilation, int groups)
      didn't match because some of the arguments have invalid types: (Tensor, !Parameter!, !Parameter!, !tuple!, !tuple!, !tuple!, int)
 * (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, str padding, tuple of ints dilation, int groups)
      didn't match because some of the arguments have invalid types: (Tensor, !Parameter!, !Parameter!, !tuple!, !tuple!, !tuple!, int)

Could you recheck colab notebook?

Originally posted by @charlescho64 in #28 (comment)

Vectorization process

Hello! Thank you for sharing your work, incredible!

I am curious if you could help me understand where the vectorization happens in the code and whether this is on top of a top down image of the layout or an actual cad file.

Thanks!

dataset download

Hi guys! I have submitted the application form but have not received any response. How to download the dataset?

Unable to use Google Colab. Says Python version is outdated

I am stuck trying to setup plan2scene since my local machine isn't powerful enough I tried with colab, but I'm not knowledgeable enough in python. Colab doesn't allow for older versions of python and its libraries. I bet I just need to fix syntax and it would work... Could it be updated?
Even if colab is updated I want to know what are my options for renting cloud, I heard about anaconda, could plan2scene be setup on their cloud servers?

P.S. Mostly a RoR Web Developer but I gained much interest this year around creating something that would use AI and I want to gain knowledge on it. Plan is to use this in addition to three.js and try to render apartments with WebGPU. A lot to learn, I know. Thank you in advance, I appreciate any informational help.