about depth using in init_net.py about neuray HOT 3 CLOSED

Hi, thanks for your interests. The CostVolumeInitNet does not require depth actually though I have passed it into the model. You can just implement a new database and return a dummy depth map like np.zeros([h,w],dtype=float32) in get_depth().

from neuray.

Thank you for your reply, which solves my problem. Then the new question is:

1. How can I output a depth map when rendering, like the figure15 in your paper;
2. I found that on my own dataset, the neuray method will lose some objects. I want to ask which parameters I should modify to improve the image quality?

the neuray results: some objects losed

Thank you very much.

from neuray.

For your new questions:

1. I have added an interface to render depth:

   NeuRay/render.py

   Line 195 in fb3188d

   parser.add_argument('--depth', action='store_true', dest='depth', default=False)

   and
   

   NeuRay/network/renderer.py

   Line 202 in fb3188d

   outputs['render_depth'] = torch.sum(hit_prob_nr * que_depth, -1) # qn,rn

   
   Results will be saved in data/render.
2. Possible reasons for missing objects: 1) depth range does not include the object. 2) too few images seeing the object so that it cannot be correctly matched to construct the radiance field.
3. Possible improvements: The MVSNet is only trained on the DTU dataset and it is fixed during the generalization training for NVS so it is not very accurate for outdoor scenes or the forward-facing scenes according to my experience. 1) I recommend reconstructing the scene by COLMAP and using the DepthInitNet, which performs better. 2) You may finetune the MVSNet in the NVS training, which however may consume lots of GPU memory.

Thank you!

from neuray.