This is a modified implementation of Neuralangelo: High-Fidelity Neural Surface Reconstruction to demonstrate the usage of DFD in NeuS-like multi-view reconstruction. We accelerate the training process by using DFD and patch-based sampling to compute the gradients of the signed distance field (SDF) in the forward rendering.

In short, we use SDF samples on a patch of rays for both SDF gradient computation and volume rendering. This avoids redundant SDF samples in the forward rendering, as used in Neuralangelo. This acceleration strategy is orthogonal to the multi-resolution hash encoding, fast ray marching, and CUDA implementation. More details can be found in our paper SuperNormal: Neural Surface Reconstruction via Multi-View Normal Integration.

Requirements and usage is the same as Neuralangelo. You can switch between the original pixel-based rendering and our patch-based rendering by setting model.render.render_mode in the configuration file projects/neuralangelo/configs/base.yaml.

Note: This is a quick and dirty implementation during the review process. Please refrain from asking for further feature updates.

Note 2: Even with DFD acceleration, the training process is still slow because

• Deep MLPs are used for the SDF and color network, and
• Coarse-to-fine ray sampling is used, which requires multiple feedforward passes of the SDF network in each iteration.

Following is the original README from Neuralangelo.

This is the official implementation of Neuralangelo: High-Fidelity Neural Surface Reconstruction.

Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H. Taylor, Mathias Unberath, Ming-Yu Liu, Chen-Hsuan Lin
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023

We offer two ways to setup the environment:

1. We provide prebuilt Docker images, where

   • docker.io/chenhsuanlin/colmap:3.8 is for running COLMAP and the data preprocessing scripts. This includes the prebuilt COLMAP library (CUDA-supported).
   • docker.io/chenhsuanlin/neuralangelo:23.04-py3 is for running the main Neuralangelo pipeline.

   The corresponding Dockerfiles can be found in the docker directory.

2. The conda environment for Neuralangelo. Install the dependencies and activate the environment neuralangelo with

   conda env create --file neuralangelo.yaml
   conda activate neuralangelo

For COLMAP, alternative installation options are also available on the COLMAP website.

Please refer to Data Preparation for step-by-step instructions.
We assume known camera poses for each extracted frame from the video. The code uses the same json format as Instant NGP.

EXPERIMENT=toy_example
GROUP=example_group
NAME=example_name
CONFIG=projects/neuralangelo/configs/custom/${EXPERIMENT}.yaml
GPUS=1  # use >1 for multi-GPU training!
torchrun --nproc_per_node=${GPUS} train.py \
    --logdir=logs/${GROUP}/${NAME} \
    --config=${CONFIG} \
    --show_pbar

Some useful notes:

• This codebase supports logging with Weights & Biases. You should have a W&B account for this.
  • Add --wandb to the command line argument to enable W&B logging.
  • Add --wandb_name to specify the W&B project name.
  • More detailed control can be found in the init_wandb() function in imaginaire/trainers/base.py.
• Configs can be overridden through the command line (e.g. --optim.params.lr=1e-2).
• Set --checkpoint={CHECKPOINT_PATH} to initialize with a certain checkpoint; set --resume to resume training.
• If appearance embeddings are enabled, make sure data.num_images is set to the number of training images.

Use the following command to run isosurface mesh extraction:

CHECKPOINT=logs/${GROUP}/${NAME}/xxx.pt
OUTPUT_MESH=xxx.ply
CONFIG=logs/${GROUP}/${NAME}/config.yaml
RESOLUTION=2048
BLOCK_RES=128
GPUS=1  # use >1 for multi-GPU mesh extraction
torchrun --nproc_per_node=${GPUS} projects/neuralangelo/scripts/extract_mesh.py \
    --config=${CONFIG} \
    --checkpoint=${CHECKPOINT} \
    --output_file=${OUTPUT_MESH} \
    --resolution=${RESOLUTION} \
    --block_res=${BLOCK_RES}

Some useful notes:

• Add --textured to extract meshes with textures.
• Add --keep_lcc to remove noises. May also remove thin structures.
• Lower BLOCK_RES to reduce GPU memory usage.
• Lower RESOLUTION to reduce mesh size.

1. Q: CUDA out of memory. How do I decrease the memory footprint?
   A: Neuralangelo requires at least 24GB GPU memory with our default configuration. If you run out of memory, consider adjusting the following hyperparameters under model.object.sdf.encoding.hashgrid (with suggested values):

   GPU VRAM	Hyperparameter
   8GB	dict_size=20, dim=4
   12GB	dict_size=21, dim=4
   16GB	dict_size=21, dim=8

   Please note that the above hyperparameter adjustment may sacrifice the reconstruction quality.

   If Neuralangelo runs fine during training but CUDA out of memory during evaluation, consider adjusting the evaluation parameters under data.val, including setting smaller image_size (e.g., maximum resolution 200x200), and setting batch_size=1, subset=1.

2. Q: The reconstruction of my custom dataset is bad. What can I do?
   A: It is worth looking into the following:

   • The camera poses recovered by COLMAP may be off. We have implemented tools (using Blender or Jupyter notebook) to inspect the COLMAP results.
   • The computed bounding regions may be off and/or too small/large. Please refer to data preprocessing on how to adjust the bounding regions manually.
   • The video capture sequence may contain significant motion blur or out-of-focus frames. Higher shutter speed (reducing motion blur) and smaller aperture (increasing focus range) are very helpful.

If you find our code useful for your research, please cite

@inproceedings{li2023neuralangelo,
  title={Neuralangelo: High-Fidelity Neural Surface Reconstruction},
  author={Li, Zhaoshuo and M\"uller, Thomas and Evans, Alex and Taylor, Russell H and Unberath, Mathias and Liu, Ming-Yu and Lin, Chen-Hsuan},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
  year={2023}
}