r00tman / nerf-osr Goto Github PK

Hi Viktor et al - very excited about this code, but in all the test*.sh files there's a parameter --test_env e.g.:
python ddp_test_nerf.py --config configs/europa/final.txt --render_splits static_path1_blend --test_env data/europa/final/static_path1_blend/envmaps
...but "--test-env" doesn't seem to be a supported parameter - I can't see it added anywhere in the code, and ddp_test_nerf.py never gets past the fact it doesn't recognise one or more parameters. Am I missing something obvious?

Excellent work! How can I use your code to train on my own dataset?

Hi there,

This is a very promising project, thanks for open-sourcing it.
We are trying to evaluate the project, but the model and data download link is really slow (< 100 kbps).
In the best interest of the project, can the data and models please be uploaded to some other cloud storage?

Thanks

Hi @r00tman ,
Thanks for sharing the code and the wonderful work.
I 've downloaded all the models and data. But when I try to run the tesh *sh scripts, there is an error message like:

Traceback (most recent call last):
File "/home/mi/anaconda3/envs/nerf/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
fn(i, *args)
File "/home/mi/PycharmProjects/NeRF-OSR/ddp_test_nerf.py", line 48, in ddp_test_nerf
start, models = create_nerf(rank, args)
File "/home/mi/PycharmProjects/NeRF-OSR/ddp_train_nerf.py", line 348, in create_nerf
with open(f) as file:
FileNotFoundError: [Errno 2] No such file or directory: 'logs/trevi_final_masked_flipxzinitenv/train_images.json'

I want to know how can I get these json file or how to generate them?

no envmap found for validation/rgb/21-08_16_00_IMG_4592-JPG

Thank you for the open source, but the width and height of the trevi dataset images in the dataset you provided are not fixed, can you please provide a version with fixed width and height?

Hello, I have a question about iterations.
In your paper, you train the model for 5 * 10^5 iterations. However, N_iters in config.txt is set to 5000001. What is the correct number of parameters?

Hello. Thank you for sharing your excellent research and code.

I have a question about the dataset. Your paper mentions the following: "Furthermore, even though NeRF-OSR can synthesise the sky and vegetation (e.g., trees), it is not possible to evaluate their predictions due to their highly varying appearance, especially when recordings sessions span different weather seasons. Hence, we also estimate the masks of these regions and exclude them from our evaluation."

However, I noticed that there are no masks available under the 'scene_name/final/test' directory in the dataset, and the masks found in the 'scene_name/final/mask' directory do not seem to include masks for trees and the sky. In order to conduct a fair evaluation, could you please release the masks that you used for evaluation?

The following error is generated when running python ddp_mesh_nerf.py --config configs/lk2/final.txt. Any ideas how to resolve it?

2023-02-12 15:56:18,287 [INFO] root: Found ckpts: ['logs/lk2_final_masked_flipxzinitenv/model_420000.pth']
2023-02-12 15:56:18,287 [INFO] root: Reloading from: logs/lk2_final_masked_flipxzinitenv/model_420000.pth
Traceback (most recent call last):
  File "ddp_mesh_nerf.py", line 102, in <module>
    mesh()
  File "ddp_mesh_nerf.py", line 97, in mesh
    join=True)
  File "/projappl/project_2007011/miniconda3/envs/nerfosr/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 200, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/projappl/project_2007011/miniconda3/envs/nerfosr/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes
    while not context.join():
  File "/projappl/project_2007011/miniconda3/envs/nerfosr/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/projappl/project_2007011/miniconda3/envs/nerfosr/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/scratch/project_2007011/NeRF-OSR/ddp_mesh_nerf.py", line 32, in ddp_mesh_nerf
    start, models = create_nerf(rank, args)
  File "/scratch/project_2007011/NeRF-OSR/ddp_train_nerf.py", line 420, in create_nerf
    models[name].load_state_dict(to_load[name])
  File "/projappl/project_2007011/miniconda3/envs/nerfosr/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1045, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DistributedDataParallel:
	Missing key(s) in state_dict: "module.env_params.01-08_07_30_IMG_6645-JPG", "module.env_params.01-08_07_30_IMG_6650-JPG",

I Use
python ddp_test_nerf.py --config configs/lk2/final.txt --render_splits static_path1_blend --test_env data/lk2/final/static_end_blend/envmaps

Obtain output

2023-07-22 19:33:57,538 [INFO] root: Reloading from: logs/lk2_final_masked_flipxzinitenv/model_1165000.pth
2023-07-22 19:33:57,657 [INFO] root: raw intrinsics_files: 0
2023-07-22 19:33:57,659 [INFO] root: raw pose_files: 0
2023-07-22 19:33:57,659 [INFO] root: Split static_path1_blend, # views: 0

Hi~ Thanks for your awesome work!
I am using your dataset to train my own nerf. I see min_depth and max_depth in your dataloder. But I have not seen these in your provided dataset. Are min_depth and max_depth nesseary? Will it cause error in your code when min_depth is None?

Hope for your reply, Thank you very much.

Hi, thank you for the great dataset.

Provided test (siggraph) data contains only 1 image for LK2 site although in your paper you mention 5 images from 5 different recording sessions.
https://nextcloud.mpi-klsb.mpg.de/index.php/s/mGXYKpD8raQ8nMk?path=%2FData%2Flk2%2Ffinal%2Fsiggraph1200

I see all 5 images for st and lwp sites. Could you please update the dataset?

The algorithm does not have an ideal effect on the indoor dataset.
As shown in the figure below, the effect of relighting is not very good. There is a "hole" on the surface of the chair.
I guss this is due to some inaccuracies in geometry estimation.
I would appreciate it if receiving your reply.
@r00tman

ori_img:

relighting:

Hi!
Thanks for sharing your code and I think this is a wonderful work to solve the problems in relighting outdoor scenes by a NeRF framework.
I have an issue with the env params consisting of 9*3 variables. I clearly know the network how to process the training images and optimize the env params from default values. However, I find that the network also takes the default values ​​as testing env params and do not optimize them. if i dont misunderstand. Is this reasonable? How do u get the default values?

Looking forward to your answer,
Thanks again!