Neural 3D Reconstruction in the Wild
Jiaming Sun, Xi Chen, Qianqian Wang, Zhengqi Li, Hadar Averbuch-Elor, Xiaowei Zhou, Noah Snavely
SIGGRAPH 2022 (Conference Proceedings)
- Training (i.e., reconstruction) code.
- Toolkit and pipeline to reproduce the evaluation results on the proposed Heritage-Recon dataset.
- Config for reconstructing generic outdoor/indoor scenes.
conda env create -f environment.yaml
conda activate neuconw
scripts/download_sem_model.shDownload the Heritage-Recon dataset and put it under data. You can also use gdown to download it in command line:
mkdir data && cd data
gdown --id 1eZvmk4GQkrRKUNZpagZEIY_z8Lsdw94vGenerate ray cache for all four scenes:
for SCENE_NAME in brandenburg_gate lincoln_memorial palacio_de_bellas_artes pantheon_exterior; do
scripts/data_generation.sh data/heritage-recon/${SCENE_NAME}
doneTo train scenes in our Heritage-Recon dataset:
# Subsutitude `SCENE_NAME` with the scene you want to reconstruct.
scripts/train.sh $EXP_NAME config/train_${SCENE_NAME}.yaml $NUM_GPU $NUM_NODEFirst, extracting mesh from a checkpoint you want to evaluate:
scripts/sdf_extract.sh $EXP_NAME config/train_${SCENE_NAME}.yaml $CKPT_PATH 10The reconstructed meshes will be saved to PROJECT_PATH/results.
Then run the evaluation pipeline:
scripts/eval_pipeline.sh $SCENE_NAME $MESH_PATHEvaluation results will be saved in the same folder with the evaluated mesh.
The code takes a standard COLMAP workspace format as input, a script is provided for automatically convert a colmap workspace into our data format:
scripts/preprocess_data.shMore instructions can be found in scripts/preprocess_data.sh
However, if you wish to select a better bounding box (i.e., reconstruction region) manually, do the following steps.
Generate semantic maps:
python tools/prepare_data/prepare_semantic_maps.py --root_dir $WORKSPACE_PATH --gpu 0Create a file config.yaml into workspace to write metadata. The target scene needs to be normalized into a unit sphere, which require manual selection. One simple way is to use SFM key-points points from COLMAP to determine the origin and radius. Also, a bounding box is required, which can be set to [origin-raidus, origin+radius], or only the region you're interested in.
{
name: brandenburg_gate, # scene name
origin: [ 0.568699, -0.0935532, 6.28958 ],
radius: 4.6,
eval_bbx: [[-14.95992661, -1.97035599, -16.59869957],[48.60944366, 30.66258621, 12.81980324]],
voxel_size: 0.25,
min_track_length: 10,
# The following configuration is only used in evaluation, can be ignored for your own scene
sfm2gt: [[1, 0, 0, 0],
[ 0, 1, 0, 0],
[ 0, 0, 1, 0],
[ 0, 0, 0, 1]],
}Run the following command with a WORKSPACE_PATH specified:
scripts/data_generation.sh $WORKSPACE_PATHAfter completing above steps, whether automatically or manually, the COLMAP workspace should be looking like this;
└── brandenburg_gate
└── brandenburg_gate.tsv
├── cache_sgs
└── splits
├── rays1_meta_info.json
├── rgbs1_meta_info.json
├── split_0
├── rays1.h5
└── rgbs1.h5
├── split_1
├──.....
├── config.yaml
├── dense
└── sparse
├── cameras.bin
├── images.bin
├── points3D.bin
└── semantic_maps
├── 99119670_397881696.jpg
├── 99128562_6434086647.jpg
├── 99250931_9123849334.jpg
├── 99388860_2887395078.jpg
├──.....Change DATASET.ROOT_DIR to COLMAP workspace path in config/train.yaml, and run:
scripts/train.sh $EXP_NAME config/train.yaml $NUM_GPU $NUM_NODEAdditionally, NEUCONW.SDF_CONFIG.inside_outside should be set to True if training an indoor scene (refer to config/train_indoor.yaml).
scripts/sdf_extract.sh $EXP_NAME config/train.yaml $CKPT_PATH $EVAL_LEVELThe reconstructed meshes will be saved to PROJECT_PATH/results.
If you find this code useful for your research, please use the following BibTeX entry.
@inproceedings{sun2022neuconw,
title={Neural {3D} Reconstruction in the Wild},
author={Sun, Jiaming and Chen, Xi and Wang, Qianqian and Li, Zhengqi and Averbuch-Elor, Hadar and Zhou, Xiaowei and Snavely, Noah},
booktitle={SIGGRAPH Conference Proceedings},
year={2022}
}Part of our code is derived from nerf_pl and NeuS, thanks to their authors for the great works.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent