Running iMAP demo about vmap HOT 5 CLOSED

And reduce the mesh grid resolution here will also speed up the meshing speed.

from vmap.

Hi, thanks for your interest in our work!

1. Yeah, it is achieved by pytorch multiprocessing, in the live demo, we have a visualization thread running, taking the newest map (MLP) send from the mapping thread and rendering 2D & 3D results. We didn't optimise the marching cube implementation which could be potentially more efficient by adopting pytorch3d. The released code is a single thread one for simplicity.
2. I'm not getting the question for "my gt_depth and gt_rgb are always randomized images with different dimensions from the input". The loss is the depth, RGB, and "obj mask" error between the rendering and the GT. And we randomly sample pixels from the keyframe buffer which always includes the latest one. The reason behind this is to keep the memory of the historical observation to avoid forgetting.

Please let me know if you need further help!

from vmap.

Thank you for the response, I'll just clarify what I mean for question 2.

Basically I wanted to visualize the depth and RGB images used for computing loss. When I look at gt_depth and gt_rgb by adding the following code:

                plt.subplot(2, 2, 1)
                plt.title('gt rgb image')
                plt.imshow(gt_rgb.cpu())
                plt.subplot(2, 2, 2)
                plt.title('gt depth image')
                plt.imshow(gt_depth.cpu())
                plt.subplot(2, 2, 3)
                plt.title('rgb image')
                plt.imshow(rgb.cpu())
                plt.subplot(2, 2, 4)
                plt.title('depth image')
                plt.imshow(depth.cpu())
                plt.show()

I get the attached output which shows garbled images for gt_depth and gt_rgb. I would have expected images generated by the MLP to look closer to the input images.

from vmap.

The training samples are obtained from function get_training_samples. The gt pixels are sampled from a subset of pixels (number = cfg.n_samples_per_frame) from training frames. And the training frames (number = cfg.win_size) are sampled from a keyframe buffer. Therefore, the visualisation of training samples wouldn't look like an image, and will actually be a group of pixels from the historical observations instead.
If you want to visualise a rendering image with the gt rgb, you need to render a whole image by a given pose.

from vmap.

Ah ok I think I understand now, thanks

from vmap.