primecai / pix2nerf Goto Github PK

Hi,

The author list in your page has a small mistake, Noah Snavely should be replaced by luc Van Gool? The link is right.

(Pix2NeRF) bash-4.4$ CUDA_VISIBLE_DEVICES=0,1 python3 train_con.py --curriculum=celeba --output_dir='/data/yshan/data/Img/pix2NerfOutput' --dataset_dir='/data/yshan/data/Img/img_align_celeba' --encoder_type='CCS' --recon_lambda=5 --ssim_lambda=1 --vgg_lambda=1 --pos_lambda_gen=15 --lambda_e_latent=1 --lambda_e_pos=1 --cond_lambda=1 --load_encoder=1
Namespace(n_epochs=3000, sample_interval=1000, output_dir='/data/yshan/data/Img/pix2NerfOutput', load_dir='/data/yshan/data/Img/pix2NerfOutput', curriculum='celeba', eval_freq=5000, port='12354', set_step=None, model_save_interval=200, pretrained_dir='', wandb_name='', wandb_entity='', wandb_project='', recon_lambda=5.0, ssim_lambda=1.0, vgg_lambda=1.0, dataset_dir='/data/yshan/data/Img/img_align_celeba', pos_lambda_gen=15.0, sn=0, lambda_e_latent=1.0, lambda_e_pos=1.0, encoder_type='CCS', cond_lambda=1.0, ema=1, load_encoder=1)
Lock not found
Traceback (most recent call last):
File "/data/yshan/Pix2NeRF/train_con.py", line 684, in
mp.spawn(train, args=(num_gpus, opt), nprocs=num_gpus, join=True)
File "/home/yshan/anaconda3/envs/Pix2NeRF/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 239, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/yshan/anaconda3/envs/Pix2NeRF/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 197, in start_processes
while not context.join():
File "/home/yshan/anaconda3/envs/Pix2NeRF/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/home/yshan/anaconda3/envs/Pix2NeRF/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
fn(i, *args)
File "/data/yshan/Pix2NeRF/train_con.py", line 85, in train
setup(rank, world_size, opt.port, opt.output_dir)
File "/data/yshan/Pix2NeRF/train_con.py", line 46, in setup
dist.init_process_group('gloo', init_method=file_lock, rank=rank, world_size=world_size)
File "/home/yshan/anaconda3/envs/Pix2NeRF/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 907, in init_process_group
default_pg = _new_process_group_helper(
File "/home/yshan/anaconda3/envs/Pix2NeRF/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1009, in _new_process_group_helper
backend_class = ProcessGroupGloo(backend_prefix_store, group_rank, group_size, timeout=timeout)
RuntimeError: flock: No locks available

Hi, I have trained the celeba in a single 3090. But when I finish 30k iterations, the program reports KeyError: 'batch_size', do I need to modify the parameters? Would you like to tell me why?

First of all, thank you for your work!
When I used code to train my data, such problems occurred, which could not be solved. I hope you can make valuable suggestions.
Specific errors are as follows:
CUDA_VISIBLE_DEVICES=2 python train_con.py --curriculum=carla --output_dir=PATH_TO_OUTPUT --dataset_dir=dataset/exp0/train/.png --encoder_type='CCS' --recon_lambda=5 --ssim_lambda=1 --vgg_lambda=1 --pos_lambda_gen=15 --lambda_e_latent=1 --lambda_e_pos=1 --cond_lambda=1 --load_encoder=1
Namespace(cond_lambda=1.0, curriculum='carla', dataset_dir='dataset/exp0/train/.png', ema=1, encoder_type='CCS', eval_freq=5000, lambda_e_latent=1.0, lambda_e_pos=1.0, load_dir='', load_encoder=1, model_save_interval=200, n_epochs=3000, output_dir='PATH_TO_OUTPUT', port='12354', pos_lambda_gen=15.0, pretrained_dir='', recon_lambda=5.0, sample_interval=1000, set_step=None, sn=0, ssim_lambda=1.0, vgg_lambda=1.0, wandb_entity='', wandb_name='', wandb_project='')
Lock not found
terminate called after throwing an instance of 'std::system_error'
what(): open(/process_group_sync.lock): Permission denied
Traceback (most recent call last):
File "train_con.py", line 686, in
mp.spawn(train, args=(num_gpus, opt), nprocs=num_gpus, join=True)
File "/home/nn/anaconda3/envs/nerf/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/nn/anaconda3/envs/nerf/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
while not context.join():
File "/home/nn/anaconda3/envs/nerf/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/home/nn/anaconda3/envs/nerf/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/home/nn/code/Pix2NeRF/train_con.py", line 87, in train
setup(rank, world_size, opt.port, opt.output_dir)
File "/home/nn/code/Pix2NeRF/train_con.py", line 48, in setup
dist.init_process_group('gloo', init_method=file_lock, rank=rank, world_size=world_size)
File "/home/nn/anaconda3/envs/nerf/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 433, in init_process_group
timeout=timeout)
File "/home/nn/anaconda3/envs/nerf/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 508, in _new_process_group_helper
timeout=timeout)
RuntimeError: open(/process_group_sync.lock): Permission denied

RuntimeError: CUDA out of memory. Tried to allocate 768.00 MiB (GPU 0; 6.00 GiB total capacity; 3.51 GiB already allocated; 391.91 MiB free; 3.57 GiB reserved in total by PyTorch)

ModuleNotFoundError: No module named 'moviepy'

pip install moviepy

RuntimeError: CUDA error: no kernel image is available for execution on the device

pip install torch==1.12.0+cu113 torchvision==0.13.0+cu113 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu113

AssertionError: Cannot find installation of real FFmpeg (which comes with ffprobe).

sudo apt-get install ffmpeg

There is no 'train.py'. Would you like to tell me how to train the pixel2nerf? Thank you.

Hi, would you like to tell me how to re-train the pix2nerf. Thank you.

1. Render novel views of the given image:
   the source image:
   

the new view:

the reconstruction:

the hybrid optimization:

Does anyone know how to improve it

Hi, Dr. Cai, would you like first release the train code of the Pixel2NeRF？Thank you.

Thanks for your great work!

I'm trying to test the pre-trained model on celeba dataset and cannot find how you split the dataset in the provided code or csv file. How could I get the test set for celeba as mentioned in the paper ( "We hold out 8k images as the test set. ").

Thanks again for you work and help.

Hello, thanks for sharing your great job!
Can we use the pretrain model to generate multi-views images with the background? e.g. like this

Thanks for your suggestions!

Hi there,
I have used 2 T4 GPU to retrain the model using the train_con.py script. The issue(cleanup function) persists across 2 different datasets, CARLA and CelebA. I would like to ask if this happened to you while training the model or if it's some error while executing the script.

Lots of thanks!

Hi, thanks for your contributions on "Single Image to Neural Radiance Fields Translation". I want to know when you will open sourced the "train" script about the pix2nerf? Thank you.

Thank you for your work! I'd like to ask how the PSNR of the srnchairs part of the paper compares. Because the pose angle defined in the code is different from the actual distribution in the data set, how do you compare the PSNR values of the rendered new view images? If it's convenient, could you provide code or some suggestions to me. Thanks!

as mentioned

Hi, many thanks for your great work！

May I ask you a question?

• What is the 'CelebA_pos.csv' used for in the training phase?
• Is it compulsory in training？ If not, how to remove it?

IS there anyone who can share the code for calculating IS 、 FID scores？Thank you!

Hi, I have re-train the pix2nerf with a single 3090 GPU on synchairs, but I have not achieve the view synthesis under the condition of a single view. If I utilize a single 3090 GPU to train the pix2nerf, would I change the batch size from 9 to 36? Thank you very much.

I means that if i know the image of 30°, i could easily get the rendered image of 40° by given a absolute angles of 40°.
However, i donnot know how the pose d which is a latent code achieve this.
That is, could i modify the latent pose d to get the clear direction?

Thanks for your excellent work！
Could you please provide checkpoints for shapenet car?
Thanks for your help!

input an image and I want to output my own 3D model

Hi, I have re-trained the pix2NeRF in a single 3090. I got similar results of IS and FID as your paper but a low result of KID. Would you like to tell me why?
The re-trained results of Pix2NeRF:
{'inception_score_mean': 2.207920169906786, 'inception_score_std': 0.04608231534322652, 'frechet_inception_distance': 28.55671719303598, 'kernel_inception_distance_mean': 0.022525952379879886, 'kernel_inception_distance_std': 0.001017455628829365}
Thank you very much.

When will the code be released? There are long time after the acceptance of the CVPR 2022.

Hi, First of all, I was impressed with your paper.
I read your paper in CVPR 2022 , but it doesn't mention about training environment and time. Can i get information about it?
if you are reluctant to disclose, plz contact me email or other connection