Converting rendered depth to 3D point cloud in world coordinates about torch-ngp HOT 9 CLOSED

@kevin-thankyou-lin Hi, if you want to extract the 3D point clouds in world coordinate, you just need to query the sdf with coordinates. A similar functionality, extracting the mesh, is provided here. You can simply modify this function by removing the marching cubes step, and instead converting to valid points.
Besides, the depth plotted currently is a relative depth value (scaled to 0-1), so it maybe not suitable for point cloud generation.

from torch-ngp.

Thanks for the great info! To be more specific, I'm trying to see whether a particular ray (defined by rays_o and rays_d) lands within a bounding volume defined in world coordinates.

I was wondering if there was a good way to convert e.g. the result of rays_o + depth * rays_d to a point in world coords?

So I know that (relative) depth is calculated as follows:

depth = torch.sum(weights * ori_z_vals, dim=-1)

I was thinking to calculated the unnormalized depth

unnormed_depth = torch.sum(median_mask * z_vals, dim=-1) # [N]

to then get rays_o + unnormed_depth * rays_d. I'm not 100% sure how to convert this result into world coordinates - any pointers on this would be great!

from torch-ngp.

@kevin-thankyou-lin Oh, I got it. I already tested this at debugging, just uncomment this line here and you can get both the point samples and a unit sphere (you can replace it with a bounding box, of course) in world coord.

from torch-ngp.

Thanks for the pointer! Maybe I can be more specific about what I mean with world-coords.

I'm taking a few photos and then convert the camera poses (camera uses OpenGL conventions) to an ngp pose via: pose_ngp = nerf_matrix_to_ngp(pose)

I then get the transformation matrix T_nerf_ngp = pose_ngp @ np.linalg.inv(pose_nerf). Then, my
c2w matrix, I think is c2w = torch.from_numpy(pose_ngp)[None, ...].to(self.device).float().

I get rays and render the model:

rays_o, rays_d, select_inds = get_rays(c2w, self.intrinsics[None, ...], self.H, self.W, N_rays=512)
with torch.cuda.amp.autocast(enabled=self.opt.fp16):
       outputs = model.render(rays_o, rays_d, **vars(self.opt))
       pred_unnormed_depth = outputs['unnormed_depth']

I visualize where the rays terminate in world (nerf) coordinates instead of ngp coordinates:

pred_3D_ngp = rays_o[0] + pred_unnormed_depths[..., None] * rays_d[0]

pred_3D_ngp_homog = torch.cat([pred_3D_ngp, torch.ones((pred_3D_ngp.shape[0], 1), device=self.device)], axis=1)
pred_3D_nerf = (torch.inverse(torch.from_numpy(T_nerf_ngp).to(self.device).float()) @ pred_3D_ngp_homog.T).T

However, when I plot pred_3D_nerf, these points seem to be in a location opposite to the direction of the ray (i.e. they're 'behind' the camera). I'm not 100% sure what I'm doing wrong here - perhaps I don't understand what the nerf_matrix_to_ngp is doing? I know it's switching some axes and perfoming some scaling --- I'm not 100% sure how to 'un-scale' the model outputs and 'switch axes' too,

May be irrelevant: a discrepancy between torch-ngp and the original nerf code is that the original code's get_rays() sets directions' last z coordinates to be -1 whereas torch-ngp is +1.

from torch-ngp.

@kevin-thankyou-lin

1. I think nerf_matrix_to_ngp should work exactly the same as instant-ngp? (negate the 2nd/3rd cols and rotate rows from xyz --> yzx, do not miss the cycle)
2. as the camera convention is different from nerf to ngp, you can check the get rays in instant-ngp here, which uses +1.
3. so you need the points in original nerf world coord sys, instead of ngp world coord sys? Some transformations will surely achieve this, but I'm also not very sure of it... maybe you could refer to this discussion.

from torch-ngp.

Thanks for the pointers, these are great!

Yes, I need the points in the original nerf world coord sys instead of ngp world coord sys. I think I've figured it out!

def ngp_ray_to_nerf(rays_o, rays_d, scale=0.33):
    rays_o = rays_o / scale

    # convert yzx to xyz
    tmp = rays_o.detach().clone()
    rays_o[..., 0] = tmp[..., 2]
    rays_o[..., 1] = tmp[..., 0]
    rays_o[..., 2] = tmp[..., 1]

    tmp = rays_d.detach().clone()
    rays_d[..., 0] = tmp[..., 2]
    rays_d[..., 1] = tmp[..., 0]
    rays_d[..., 2] = tmp[..., 1]
    return rays_o, rays_d

I'm taking the 'inverse' of their nerf_ray_to_ngp function - I've also visualized it for the case where scale=1 and things check out in the original nerf world coords there. Thanks for all your prompt responses and help!

I'm now trying to do the equivalent of taking the (median) unnormed depth value of a ray except for run_cuda. For run I have access to z_vals:

        cumsum_weights = torch.cumsum(weights, dim=-1) # [N, T]
        median_mask = cumsum_weights > 0.5 # [N, T]
        median_mask[..., 1:] = torch.logical_xor(median_mask[..., :-1], median_mask[..., 1:]) # 1st non-zero
        unnormed_median_depth = torch.sum(median_mask * z_vals, dim=-1) # [N]

Would you know how I also get the unnormed z_vals inside run_cuda? Seems like things are abstracted inside raymarching but I can't seem to find the equivalent z_vals there.

from torch-ngp.

@kevin-thankyou-lin Glad to hear that!
If I understand correctly, you are locating the first point in each ray where weight_sum > 0.5?
This can be easily done in the CUDA side, by a small modificaiton around here, like

bool flag_first = true;
....
if (weight_sum > 0.5 && flag_first) {
    depth[0] = t; // here t is in fact the accumulated z
    flag_first = false;
}

Another problem is that depth is currently not calculated in run_cuda during training (since it requires extra memory & calculation), but only in inference. You can check the difference of deltas between training and inference kernels (e.g., here).

from torch-ngp.

Yes! I'm essentially implementing DS-NeRF (which would use the expected depth), but now I think I'll do so first in pure-pytorch and then see if I need the CUDA speed boost. Thanks for all your help! I'll reach out again if I get stuck - thanks again!!

from torch-ngp.

Closed for now.

from torch-ngp.