xharlie / pointnerf Goto Github PK

I get an error when run train_ft.py. Why it happens?

Traceback (most recent call last):
File "train_ft.py", line 1086, in
main()
File "train_ft.py", line 869, in main
add_xyz, add_embedding, add_color, add_dir, add_conf = probe_hole(model, prob_dataset, Visualizer(prob_opt), prob_opt, None, test_steps=total_steps, opacity_thresh=opt.prob_thresh)
File "train_ft.py", line 497, in probe_hole
gt_image = torch.zeros((height * width, 3), dtype=torch.float32, device=prob_maps["ray_mask"].device)
KeyError: 'ray_mask'
end loading

Hi Xharlie,

thank you very much for publishing your code!

I try to apply your method to some own data and I would like to train the MVSNet on it. Could you please provide some guidance on how to do this with your repo?

Your help is very much appreciated.

Hi! Great work. But I think there might be some mistake between the above two files. The codes inside them are entirely the same.

Hello . Thanks for your amazing work.

However , When I run bash/dev_scripts/hotdog_test.sh ， it goes with FileNotFoundError ,in line 248 load_blender_cloud load_blender_cloud.py , where is the pointcloud saved/dumped in your scripts . I can't find the this.

Hi. Thanks for your great work. I am creating a data-set collecting by myself, I want to know how to write a fine-tune_pair.txt in the nerfsynth data-set. Looking forward to your reply.

Hi ,thanks for your amazing work.

However , I'm curious about the result in Nerf LLFF forward scenes as I haven't seen the result in your paper . Have you tested the performance on this data , is it still good on outdoor scene ? Sorry to trouble you for this ,because it is a bit difficult for me to reproduce the result on LLFF dataset.

Hello authors, thank you very much for your paper and for releasing code! I am very excited about this work.
I have gotten point cloud data in a ply format somehow, how do I use this data as input to point_nerf. I don't seem to find it in "READEME.md".

Dear authro,

Can you release the full environment setup?
such as environment file in conda.

Thanks

Hi.
Could you also upload the dtu dataset to baidu pan, I can't download it from google drive anyway, it's too big.

Hi @Xharlie,

thanks for the great work! The link to scores and images on all the datasets is currently missing DTU dataset results. Can you please upload it? Thanks!

-Sergey

Dear Author,

It seems the code does not supper batch_size >1? Because the code here?
https://github.com/Xharlie/pointnerf/blob/master/models/neural_points/query_point_indices_worldcoords.py#:~:text=R%20%3D%20torch.max(torch.sum(ray_mask_tensor.to(torch.int32))).cpu().numpy()

Bests

Hello authors, thank you very much for your paper and for releasing code!

I am trying to run the script "bash dev_scripts/w_tt_ft/barn_test.sh". Is the final output a 3D model? I was confused.

This is the log file recorded at runtime.

out.txt

Hi, thanks for your great job! I have the same problem when I run "bash dev_scripts/w_n360/ship_test.sh". It goes with "FileNotFoundError: [Errno 2] No such file or directory" in line 118 of load_blender.py. My datapath is pointnerf/data_src/nerf/nerf_synthetic/ship，and it includes 3 .json files and 3 folders containing some .png pictures. At the same time, the checkpoints folder only include some .pth files. It doesn't seem to contain the saved point cloud.

Could you please tell me where the "point_path" is? Thank you~

for i in $(seq 1 $prob_freq $maximum_step)

What does the above code do？

Hi everyone,
I have a problem when launching bash scripts like "bash dev_scripts/w_n360/chair.sh", the argument 'name' is not detected and the terminal returns me the error:
train_ft.py: error: the following arguments are required: --name
Did any of you had the same problem?

Hi, @Xharlie

Thanks for your code and data. I tried to rebuild the nerf_synthetic data using the default settings of colmap dense rebuild pipeline, but I found that the results are poor and the quality of the generated point cloud is far from the one you provided. Could you please share your reconstruction settings or provide an automatic reconstruction script?

For example, on the lego data, it is clear that the key points of the two sides of the tracks were matched together, causing an error in the reconstruction.

hi, thanks for your great work.
what device do you use to train the model? I get segmentation fault (core dumped) when i train on p100 using bash dev_scripts/w_n360/chair.sh python 3.7.5 pytorch 1.8.1 pycuda 2021.1 torch_scatter2.0.8

hi @Xharlie @zexxu

What's the difference of neural_points_volumetric_model.py and mvs_points_volumetric_model.py

best,
Yingjie

Thanks for sharing the code! PointNerf's synthesis result is really amazing!

I've read the code and confused about how to compute the ray_dist. In vanilla NeRF, the sigma is usually computed using ray-distance, but PointNerf seems using depth instead of ray distance to compute sigma?

After running ‘bash dev_scripts/w_scannet_etf/scane241_test.sh’ and 'bash dev_scripts/w_n360/chair_test.sh' I got this PyCUDA error:

Does anybody know how to solve it?

Hi there, I followed all of your instructions and now can train from the scratch. However, I met some difficulties when trying point nerf on my own data. I think some more instructions may be needed for data preparation. Thanks a lot!

thanks for your briliant project. I want to use pointnerf on my own dataset. should i retrain the mvsNet on my own dataset??

In base_rendering_model, loss = self.l2loss(masked_output, masked_gt) * masked_gt.shape[1], why multiply masked_gt.shape[1]?

Thank you for your work!
There is a value error when I test the per-scene optimized Point-NeRFs.
For example, run:
bash dev_scripts/w_n360/chair_test.sh

I checked the line 92 in visualizer.py and find the total_step is not a integer but a string.

Therefore, the filename cannot be generated and will bring this error because of mismatched variable type.
You can fix this bug by converting total_step to int.
Thank you for your time.

Have you provided extracting mesh function?

I am confused about the training and testing split. Please see

In the above code, when self.opt.test_num_step is 1, the whole self.all_id_list is returned as testing list. Since self.all_id_list contains all the frame id of the dataset, the codes evaluate methods on all frames of a scene, including the training frames.

Hi,
Thanks for the awesome work. However I have a doubt about the parameters Neural_points.points_color, why they are negative? So what does this 'color' represent?

Thanks for you brilliant work. I have a question about sampling rays. Since I want to render a patch, I set the parameter 'random_sample' in scene101.sh and in train_ft.py line 917 as 'patch', then run per-scene optimization on scannet scene0101_04. In this way, the performance is inferior (~2 drop on PSNR) to randomly sampling rays. Is my setup correct? Have you ever tried the patch sampling? Thanks

Hi, @Xharlie

Thanks for your interesting work and for sharing the code. I love this work pretty much. While I noticed there is a point growing module of the network and Figure. 5 in the paper shows the module can grow points from extremely sparse point clouds, then why do we need another MVS Network to predict dense point clouds? Since we can obtain sparse point clouds from differentiable triangulation if camera poses are given. I would also like to know if there are any ablation studies showing the difference. Besides, it's interesting to show on which cases the point growing module will fail.

Hi , when I redownload the code repo and run from scratch ,It still encounters with the problem of "No such file or directory" when I run scene101.sh ,However it works fine if I run from scratch in the dataset of Nerf Synth. Is there some possiblity that I should change some setting for the scannet scene.
#7 (comment)

Hello authors, thank you very much for your paper and for releasing code! I am very excited about this work.

I am trying to run inference on a model that has been trained for point initialization (not with per-scene optimization). The README indicates that it should be possible to perform inference on DUT with bash dev_scripts/dtu_test_inf/inftest_scan8.sh.

When I run this script, I get an error:

File "/path/to/pointnerf/run/../models/neural_points_volumetric_model.py", line 270, in forward
    decoded_features, ray_valid, weight, conf_coefficient = self.aggregator(sampled_color, sampled_Rw2c, sampled_dir, sampled_conf, sampled_embedding, sampled_xyz_pers, sampled_xyz, sample_pnt_mask, sample_loc, sample_loc_w, sample_ray_dirs, vsize, grid_vox_sz)
...
File "/path/to/python3.8/site-packages/torch/nn/functional.py", line 1847, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta,
 c, ldc)`

I am using CUDA 11.1 and I believe that everything CUDA-related is installed correctly.

I was also confused by something else: this script calls train_ft.py, not test.py as I would have expected. In fact, there is not a single script in dev_scripts that calls test.py. Am I misunderstanding something here, or is test.py actually never used?

Hi @Xharlie and @zexxu ,

I do not understand about "point feature initial" part.
Does the model under "init" folder for all dataset? Or for different dataset, I need to train a VGG 2D feature extracting network firstly and seperately? If need, how to train for different dataset.

Can the model under "init" used by other dataset except for dtu? For example, scannet and kitti.

Best,
Yingjie

Thank you for your work and contribution.
I saw that you uploaded the code for COLMAP and it seems to only need the colmap_results/dense/fused.ply file.
I'm having trouble making my own dataset, I have a batch of images now, how do I make my transforms_{}.json. the camera_angle_x and rotation in it I don't know where to get from, the transform_matrix seems to be available from COLMAP. Do you have a script to generate these files please? I'm new at this.

I run this command to do Per-scene optimize from scratch:
bash dev_scripts/w_n360/lego.sh

In the paper, the speed should be 2min / 1K iters. However, on my single 2080Ti, it takes 100min / 1K iters. Anyone meets this problem?

Hi @zexxu,

Which tool did you use to create this illustration?

In the training process(point growing), how to obtain the confidence of the newly added neural points, and how to optimize the confidence of these points.

Thanks for your ncie work.
However, I have no idea with this bug，really need help.

Thanks for your great work!
I would like to know what premlp(https://github.com/Xharlie/pointnerf/blob/master/models/mvs/mvs_points_model.py line22) does, just turn 63-dimensional input into 32-dimensional? How is the weight of this part set?

Thanks for sharing this work. i am a little confused about the loss item. i know that 'coarse_raycolor' is corresponding to L_render in paper, what is 'ray_masked_loss' and 'ray_miss_loss'?
And if i understand correctly, 'zero_one_loss_items' is the L_sparse in paper. but since we already set the neural points as input data and didn't load mvsnet model, can the parameters of mvsnet also be updated? (when i inspect the variable , the only model is "ray_marching") And you set the neural points to nn.parameter to update, any reasons behind it?

Hello,

Thanks for the awesome work. However, I am not able to download the DTU training dataset from the gdrive link shared in the repo. It seems that the link is dead. Would it be possible for you to reactivate the link?

Thanks,
Aditya Vora

Hello, I just meet some problem when checking the code of intializing neural points from COLMAP points.
In your paper, you said that

We first rule out all the views where the point is occluded by other points, then we find the view of which the camera is the closest to the point.

,but I can't find the code of rule out all the views where the point is occluded by other points. In line 709 - line 720 of train_ft.py, you just find the the view of which the camera is the closest to the point.

I'm also troubled by the formula for calculating the distance between camera and view which is implemented as

dists_norm = torch.norm(dists, dim=-1) # N, M
dists_dir = dists / (dists_norm[...,None]+1e-6) # N, M, 3
dists = dists_norm / 200 + (1.1 - torch.sum(dists_dir * raydir[None, :],dim=-1))

I wonder that what does number 200 and 1.1 mean in this formula. And how does this formula ensures that the point is on the camera plane(the pixel coordinates of the point is between [0,img_size])?

After training on the dtu dataset, is it possible to also render all the test scenes in the dtu dataset without optimizing each scene? So pointnerf also seems to be a generic nerf like pixelnerf?
Then after per-scene optimization, can pointnerf get better performance than Nerf trained with multiple views?

Hello , sorry to bother again. I saw "--SR" is the max shading points number each ray, and when process the rays, the rays have more than 80 shading points are deleted.
I wonder why we need this upper bound setting, any reason behind it?

Thank you for your work and contribution!
I want to generate a depth map, in the "neural_points_volumetric_model.py " file, I set the self.return_depth is True, however, there is a warning that 'ray_ts' is not defined. And I want to know what is the 'ray_ts' in your code, looking forward to your reply.

Hello,
Thanks for the awesome work. However when I run the Per-scene Optimization on NeRF-Synthetics I found some error below:

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Debug Mode
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/numpy/core/shape_base.py:420: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
arrays = [asanyarray(arr) for arr in arrays]
/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1646755953518/work/aten/src/ATen/native/TensorShape.cpp:2228.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
dataset total: train 100
dataset [NerfSynthFtDataset] was created
../checkpoints/nerfsynth/lego/*_net_ray_marching.pth
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Continue training from 200000 epoch
Iter: 200000
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
opt.act_type!!!!!!!!! LeakyReLU
self.points_embeding torch.Size([1, 479862, 32])
querier device cuda:3 3
Traceback (most recent call last):
File "train_ft.py", line 1084, in
main()
File "train_ft.py", line 637, in main
model = create_model(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/init.py", line 39, in create_model
instance.initialize(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/base_rendering_model.py", line 369, in initialize
self.create_network_models(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/mvs_points_volumetric_model.py", line 44, in create_network_models
super(MvsPointsVolumetricModel, self).create_network_models(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points_volumetric_model.py", line 157, in create_network_models
params = self.get_additional_network_params(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points_volumetric_model.py", line 142, in get_additional_network_params
self.neural_points = NeuralPoints(opt.point_features_dim, opt.num_point, opt, self.device, checkpoint=checkpoint_path, feature_init_method=opt.feature_init_method, reg_weight=0., feedforward=opt.feedforward)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points/neural_points.py", line 331, in init
self.querier = self.lighting_fast_querier(device, self.opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points/query_point_indices_worldcoords.py", line 39, in init
self.claim_occ, self.map_coor2occ, self.fill_occ2pnts, self.mask_raypos, self.get_shadingloc, self.query_along_ray = self.build_cuda()
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points/query_point_indices_worldcoords.py", line 524, in build_cuda
""", no_extern_c=True)
File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 358, in init
include_dirs,
File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 298, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 87, in compile_plain
checksum.update(preprocess_source(source, options, nvcc).encode("utf-8"))
File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 59, in preprocess_source
"nvcc preprocessing of %s failed" % source_path, cmdline, stderr=stderr
pycuda.driver.CompileError: nvcc preprocessing of /tmp/tmpeytiys5x.cu failed
[command: nvcc --preprocess -arch sm_86 -I/mnt/data1/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/cuda /tmp/tmpeytiys5x.cu --compiler-options -P]
[stderr:
b'cc1plus: fatal error: cuda_runtime.h: No such file or directory\ncompilation terminated.\n']
end loading

PyCUDA ERROR: The context stack was not empty upon module cleanup.

A context was still active when the context stack was being
cleaned up. At this point in our execution, CUDA may already
have been deinitialized, so there is no way we can finish
cleanly. The program will be aborted now.
Use Context.pop() to avoid this problem.

So How can I fix it? Thanks.

Hi @Xharlie @zexxu ,

Could you help me about the following error?

Hi @Xharlie and thanks for the awesome work. I have been trying to bias NeRF with geometry lately and I'm glad to see such a good work published at CVPR 2022.

I have a question about the evaluation process on Tanks & Temples. Let's take the truck scene as an example. You report a PSNR of 25.36 for basic NeRF, that actually matches the value in NSVF (I assume it's just taken from there). In that paper, they say: "We label the object masks ourselves with the software of Altizure".

Do you also use object masks and compute the metrics only on pixels from that mask? It seems plausible, as the results shown in your paper and your repo contain only the object, without the background.

I'm asking this because the PSNR of NeRF is significantly smaller, when computed on the whole image. It drops down to 20.85 as reported in NeRF++ and I have been able to reach a bit more than 21 with some tuning, but not more.

Thank you in advance for the answer!

Why '/ 4'?

Hello authors, thank you very much for your paper and for releasing code!

I am trying to run : bash ./dev_scripts/w_n360/ship.sh

When I run this script, I get the follow log and error:

-----------------------------------Generate Points-----------------------------------
model [MvsPointsVolumetricModel] was created
opt.resume_iter!!!!!!!!! best
loading mvs from ../checkpoints/init/dtu_dgt_d012_img0123_conf_agg2_32_dirclr20/best_net_mvs.pth
cannot load ../checkpoints/init/dtu_dgt_d012_img0123_conf_agg2_32_dirclr20/best_net_mvs.pth
------------------- Networks -------------------
[Network mvs] Total number of parameters: 0.382M

0%| | 0/543 [00:00<?, ?it/s]/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:3829: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
"Default grid_sample and affine_grid behavior has changed "
100%|
543/543 [02:26<00:00, 3.71it/s]
100%|
543/543 [31:52<00:00, 3.52s/it]
xyz_world_all torch.Size([0, 3]) torch.Size([0, 1]) torch.Size([0])
%%%%%%%%%%%%% getattr(dataset, spacemin, None) None
vishull_mask torch.Size([0])
alpha masking xyz_world_all torch.Size([0, 3]) torch.Size([0, 1])
Traceback (most recent call last):
File "train_ft.py", line 1084, in
main()
File "train_ft.py", line 639, in main
points_xyz_all, points_embedding_all, points_color_all, points_dir_all, points_conf_all, img_lst, c2ws_lst, w2cs_lst, intrinsics_all, HDWD_lst = gen_points_filter_embeddings(train_dataset, visualizer, opt)
File "train_ft.py", line 142, in gen_points_filter_embeddings
xyz_world_all, sparse_grid_idx, sampled_pnt_idx = mvs_utils.construct_vox_points_closest(xyz_world_all.cuda() if len(xyz_world_all) < 99999999 else xyz_world_all[::(len(xyz_world_all)//99999999+1),...].cuda(), opt.vox_res)
File "/root/pointnerf/run/../models/mvs/mvs_utils.py", line 541, in construct_vox_points_closest
xyz_min, xyz_max = torch.min(xyz, dim=-2)[0], torch.max(xyz, dim=-2)[0]
RuntimeError: cannot perform reduction function min on tensor with no elements because the operation does not have an identity
end loading

I add some log to debug this issue as follow , found out: all item of confidence_all equals to 0.0208.

torch.set_printoptions(profile="full")
print(cam_xyz_all)
_, xyz_world_all, confidence_filtered_all = filter_utils.filter_by_masks_gpu(cam_xyz_all, intrinsics_all, extrinsics_all, confidence_all, points_mask_all, opt, vis=True, return_w=True, cpu2gpu=cpu2gpu, near_fars_all=near_fars_all)

Thanks for your great work! I wonder about the time cost, iteration numbers, and device when training on DTU dataset. Would you please tell me this important information?