computational-imaging / acorn Goto Github PK

ACORN: Adaptive Coordinate Networks for Neural Scene Representation | SIGGRAPH 2021

License: MIT License

Python 97.18% Cython 2.82%

acorn's Introduction

ACORN: Adaptive Coordinate Networks for Neural Scene Representation
SIGGRAPH 2021

Project Page | Video | Paper

PyTorch implementation of ACORN.
ACORN: Adaptive Coordinate Networks for Neural Scene Representation
Julien N. P. Martel*, David B. Lindell*, Connor Z. Lin, Eric R. Chan, Marco Monteiro, Gordon Wetzstein
Stanford University
*denotes equal contribution
in SIGGRAPH 2021

Quickstart

To setup a conda environment, download example training data, begin the training process, and launch Tensorboard, follow the below commands. As part of this you will also need to register for and install an academic license for the Gurobi optimizer (this is free for academic use).

conda env create -f environment.yml
# before proceeding, install Gurobi optimizer license (see above web link)
conda activate acorn 
cd inside_mesh
python setup.py build_ext --inplace
cd ../experiment_scripts
python train_img.py --config ./config_img/config_pluto_acorn_1k.ini
tensorboard --logdir=../logs --port=6006

This example will fit 1 MP image of Pluto. You can monitor the training in your browser at localhost:6006.

Adaptive Coordinate Networks

An adaptive coordinate network learns an adaptive decomposition of the signal domain, allowing the network to fit signals faster and more accurately. We demonstrate using ACORN to fit large-scale images and detailed 3D occupancy fields.

Datasets

Image and 3D model datasets should be downloaded and placed in the data directory. The datasets used in the paper can be accessed as follows.

Public domain image of Pluto is included in the repository (NASA/Johns Hopkins University Applied Physics Laboratory/Southwest Research Institute/Alex Parker)
Gigapixel image of Tokyo (Trevor Dobson CC BY-NC-ND 2.0 image resized from original)
Public domain Gigapixel image of Mars (NASA/JPL-Caltech/MSSS)
Blender engine model .obj, .blend (ChrisKuhn CC-BY)
Lucy dataset .ply (Stanford 3D Scanning Repository)
Thai Statue dataset .ply (Stanford 3D Scanning Repository)
Dragon dataset (TurboSquid)

Training

To use ACORN, first set up the conda environment and build the Cython extension with

conda env create -f environment.yml
conda activate acorn 
cd inside_mesh
python setup.py build_ext --inplace

Then, download the datasets to the data folder.

We use Gurobi to perform solve the integer linear program used in the optimization. A free academic license can be installed from this link.

To train image representations, use the config files in the experiment_scripts/config_img folder. For example, to train on the Pluto image, run the following

python train_img.py --config ./config_img/config_pluto_1k.ini
tensorboard --logdir=../logs/ --port=6006

After the image representation has been trained, the decomposition and images can be exported using the following command.

python train_img.py --config ../logs/<experiment_name>/config.ini --resume ../logs/<experiment_name> <iteration #> --eval

Exported images will appear in the ../logs/<experiment_name>/eval folder, where <experiment_name> is the subdirectory in the log folder corresponding to the particular training run.

To train 3D models, download the datasets, and then use the corresponding config file in experiment_scripts/config_occupancy. For example, a small model representing the Lucy statue can be trained with

python train_occupancy.py --config ./config_occupancy/config_lucy_small_acorn.ini

Then a mesh of the final model can be exported with

python train_occupancy.py --config ../logs/<experiment_name>/config.ini --load ../logs/<experiment_name> --export

This will create a .dae mesh file in the ../logs/<experiment_name> folder.

Citation

@article{martel2021acorn,
  title={ACORN: {Adaptive} coordinate networks for neural scene representation},
  author={Julien N. P. Martel and David B. Lindell and Connor Z. Lin and Eric R. Chan and Marco Monteiro and Gordon Wetzstein},
  journal={ACM Trans. Graph. (SIGGRAPH)},
  volume={40},
  number={4},
  year={2021},
}

Acknowledgments

We include the MIT licensed inside_mesh code in this repo from Lars Mescheder, Michael Oechsle, Michael Niemeyer, Andreas Geiger, and Sebastian Nowozin, which is originally included in their Occupancy Networks repository.

J.N.P. Martel was supported by a Swiss National Foundation (SNF) Fellowship (P2EZP2 181817). C.Z. Lin was supported by a David Cheriton Stanford Graduate Fellowship. G.W. was supported by an Okawa Research Grant, a Sloan Fellowship, and a PECASE by the ARO. Other funding for the project was provided by NSF (award numbers 1553333 and 1839974).

Errata

The 3D shape fitting metrics were reported in the paper as calculated using the Chamfer-L1 distance. The metric should have been labeled Chamfer-L2, which is consistent with the implementation in this repository.

acorn's People

Contributors

Stargazers

Watchers

acorn's Issues

Flat outputs result for demo 2D representation

Hi, I am trying to run ACORN for demo 2D representation, after several hours of training for pluto.jpg, I got results as follows,

The left image is ground truth, the right one is predicted images, outputs from official ACORN.

And I also implement my personal version of ACORN according to the paper with a simple version of Octree, I meet the same issue, the output result is just a flat image.
The loss is not decent either,

Do you have any idea about this? Moreover, I also wonder that how to propagate gradient by interpolation function such as grid_sample? It seems non-differentiable (i.e., it is a function of position or coordinate).

features_out = torch.nn.functional.grid_sample(features_in, sample_coords,
                                                       mode='bilinear',
                                                       padding_mode='border',
                                                       align_corners=True).reshape(b_size, n_channels, np.prod(self.patch_size))

Any suggestion or idea for indicating the problems?

Train occupancy on point cloud

Hi!

Is it possible to train occupancy model on point cloud? As I can see, training heavily relies on OccupancyDataset which needs a mesh to evaluate occupancy of query points. Is there any workaround to use point clouds or this model can be used for meshes only?

o attribute 'num_scales'

dataio.py", line 493, in getitem
scales = 2*scales / (self.num_scales-1) - 1
AttributeError: 'Block3DWrapperMultiscaleAdaptive' object has no attribute 'num_scales'

Tokyo/mars gigapixel too large, process killed

Hi,

I've tried setting up the ACRON environment to train the Tokyo gigapixel ("tokyo.tif") but I cannot get the process to train after the data loader finished 50%. Even I changed the PIL.Image.MAX_IMAGE_PIXELS = none , the process still gets killed due to large memory use. I wonder if you have ever encountered this issue and would like to know how you deal with such a large image without exceeding the memory.

Thanks in advance.

Training speed

Hi, I'm training a 3d model (engine) by your code. And I completely followed the steps in README. But the code runs too slow (more than 1000 hours to finish). So where the problem is ?
(I used a GeForce RTX 2080 Ti)

Error of merging and splitting blocks

Thanks for your great job. The performance is solid and consistent in my own experiments. But I still have some confusion about the calculation of "error of merging and splitting blocks" in your code and paper.

In the paper, the error of merging blocks is Ns times the current error if parent's error is unavailable. However, all the siblings will contribute Ns times error if merging, which means the error of merging blocks will be Ns^2 compared with the current error in total. So, how could the error be consistent across scales?
In the paper, the error of splitting blocks is the sum of children's errors if children's error is available. But in your code, it seems to be Ns times of the sum of children's errors as shown below.

err_children = np.sum([child.err for child in self.children])
err_split = area * err_children    # here, area is parent's area

Why the block volume is still block_size**2 in OctTree in the code?

Hope you can correct my understanding of this work. Thank you very much!

How to choose res/patch size for new images

Hi,

Currently, I want to run ACORN for several new images. We encountered an issue with this image

that has '3250 x 4333' res. Since 3250 and 4333 has no common factor, I don't know how to choose patch size for that. I tried to resize it with the following config

but during the training, we encountered an unexpected error related to tensor shapes:

Any idea why this happened? Thanks in advance.

Training without Gurobi optimizer

Hi,

Is it possible to train/test without the Gurobi optimizer?

Regards,
MJay

Metrics

Hi, thanks for your great work!
Could you please release your code of metrics computation? Especially I want to know how many points you used to compute chamfer distance?

About the number of coords sampled within each block.

Hi, I have checked your paper and codes, and find that your sampling strategy is different from that in SIREN. You only sample nl points within each block. But I do not find the exact number of this nl. Could you please provide more explanations?