NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images

1. Install basic required packages.

git clone https://github.com/liuyuan-pal/NeRO.git
cd NeRO
pip install -r requirements.txt

2. Install nvdiffrast. Please follow instructions here https://nvlabs.github.io/nvdiffrast/#installation.
3. Install raytracing. Please follow instructions here https://github.com/ashawkey/raytracing.

• NeRO Models and datasets all can be found here.
• The Blender dataset used for testing comes from NeRF and Ref-NeRF.

This project is also compatible with the NeRO dataset. In this project, we have placed the setting of dataset_dir in the .yaml file.

For example, if the path of the lego dataset is ~/NeRO/data/nerf_synthetic/lego, then in the .yaml file, dataset_dir: ~/NeRO/data/nerf_synthetic.

To run Blender Dataset, we need to change the .yaml files. The .yaml file for the Blender dataset differs from NeRO in two ways.

• The category of database_name is nerf.
• It is necessary to add is_nerf: true in the .yaml file.

Examples of the .yaml files can be seen in configs/shape/nerf and configs/material/nerf.

1. In the NeRO directory, ensure that you have the following data:

NeRO
|-- data
    |-- GlossyReal
        |-- bear 
            ...
    |-- GlossySynthetic
        |-- bell
        
    |-- Blender
    	|-- lego
            ...

2. Run the training script

# reconstructing the "bell" of the Glossy Synthetic dataset
python run_training.py --cfg configs/shape/syn/bell.yaml

# reconstructing the "bear" of the Glossy Real dataset
python run_training.py --cfg configs/shape/real/bear.yaml

# reconstructing the "lego" of the Blender dataset
python run_training.py --cfg configs/shape/nerf/lego.yaml

Intermediate results will be saved at data/train_vis. Models will be saved at data/model.

3. Extract mesh from the model.

python extract_mesh.py --cfg configs/shape/syn/bell.yaml
python extract_mesh.py --cfg configs/shape/real/bear.yaml
python extract_mesh.py --cfg configs/shape/nerf/lego.yaml

The extracted meshes will be saved at data/meshes.

1. In the NeRO directory, ensure that you have the following data:

NeRO
|-- data
    |-- GlossyReal
        |-- bear 
            ...
    |-- GlossySynthetic
        |-- bell
        
    |-- Blender
    	|-- lego
            ...
    |-- meshes
        | -- bell_shape-300000.ply
        | -- bear_shape-300000.ply
        | -- lego_shape-300000.ply
             ...

2. Run the training script:

# estimate BRDF of the "bell" of the Glossy Synthetic dataset
python run_training.py --cfg configs/material/syn/bell.yaml

# estimate BRDF of the "bear" of the Glossy Real dataset
python run_training.py --cfg configs/material/real/bear.yaml

# estimate BRDF of the "lego" of the Blender dataset
python run_training.py --cfg configs/material/nerf/lego.yaml

Intermediate results will be saved at data/train_vis. Models will be saved at data/model.

3. Extract materials from the model.

python extract_materials.py --cfg configs/material/syn/bell.yaml
python extract_materials.py --cfg configs/material/real/bear.yaml
python extract_materials.py --cfg configs/material/nerf/lego.yaml

The extracted materials will be saved at data/materials.

1. In the NeRO directory, ensure that you have the following data:

NeRO
|-- data
    |-- GlossyReal
        |-- bear 
            ...
    |-- GlossySynthetic
        |-- bell
            ...
    |-- meshes
        | -- bell_shape-300000.ply
        | -- bear_shape-300000.ply
             ...
    |-- materials
        | -- bell_material-100000
            | -- albedo.npy
            | -- metallic.npy
            | -- roughness.npy
        | -- bear_material-100000
            | -- albedo.npy
            | -- metallic.npy
            | -- roughness.npy
    |-- hdr
        | -- neon_photostudio_4k.exr

2. Run relighting script

python relight.py --blender <path-to-your-blender> \
                  --name bell-neon \
                  --mesh data/meshes/bell_shape-300000.ply \
                  --material data/materials/bell_material-100000 \
                  --hdr data/hdr/neon_photostudio_4k.exr \
                  --trans
                  
python relight.py --blender <path-to-your-blender> \
                  --name bear-neon \
                  --mesh data/meshes/bear_shape-300000.ply \
                  --material data/materials/bear_material-100000 \
                  --hdr data/hdr/neon_photostudio_4k.exr

The relighting results will be saved at data/relight with the directory name of bell-neon or bear-neon. This command means that we use neon_photostudio_4k.exr to relight the object.

Refer to custom_object.md.

Refer to eval.md.

In this repository, we have used codes from the following repositories. We thank all the authors for sharing great codes.

• NeuS
• NvDiffRast
• NvDiffRec
• Ref-NeRF
• RayTracing
• COLMAP

@inproceedings{liu2023nero,
  title={NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images},
  author={Liu, Yuan and Wang, Peng and Lin, Cheng and Long, Xiaoxiao and Wang, Jiepeng and Liu, Lingjie and Komura, Taku and Wang, Wenping},
  booktitle={SIGGRAPH},
  year={2023}
}