when I use python setup.py install to install NumpyMarchingCubes,it raise the following error:

   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
marching_cubes/src/marching_cubes.cpp: In function ‘void get_voxel(const vec3f&, const npy_accessor&, float, float&, int&)’:
marching_cubes/src/marching_cubes.cpp:83:32: error: ‘numeric_limits’ is not a member of ‘std’
   83 |                 if (d != -std::numeric_limits<float>::infinity() && fabs(d) < truncation) w = 1;
      |                                ^~~~~~~~~~~~~~
marching_cubes/src/marching_cubes.cpp:83:47: error: expected primary-expression before ‘float’
   83 |                 if (d != -std::numeric_limits<float>::infinity() && fabs(d) < truncation) w = 1;
      |                                               ^~~~~
marching_cubes/src/marching_cubes.cpp:83:47: error: expected ‘)’ before ‘float’
   83 |                 if (d != -std::numeric_limits<float>::infinity() && fabs(d) < truncation) w = 1;
      |                    ~                          ^~~~~
      |                                               )
marching_cubes/src/marching_cubes.cpp:87:27: error: ‘numeric_limits’ is not a member of ‘std’
   87 |                 d = -std::numeric_limits<float>::infinity();
      |                           ^~~~~~~~~~~~~~
marching_cubes/src/marching_cubes.cpp:87:42: error: expected primary-expression before ‘float’
   87 |                 d = -std::numeric_limits<float>::infinity();
      |                                          ^~~~~
error: command '/usr/bin/gcc' failed with exit code 1

How can I fix this problem?

Hi Hengyi,

thanks for your excellent job! About smooth term, I have following serveral questions.

1. The explanation in the paper is not very clear.Can you explain more wht it works? Why we hope feature metric difference beween adjacent sampled vertices on the hash-grid trend to zero？Let's assume, one sample point is actual on the surface, its adjacent points in and out surface could be empty air. If we propose feature metric difference beween them is zero, this will destroy the reconstruction on this surface point. (I have replaced feature plane by hash grid in Eslam, and adding this smooth term on it with different weight. I find we will loss some reconstructed surface compared without using this term, especially on Scannet.)

2. How do you estimate the bound of scene? I notice for Replica room0, the bound size is different in different papers. In nice-slam [[-2.9,8.9],[-3.2,5.5],[-3.5,3.3]], in eslam [[-1.9,7.9],[-2.2,4.5],[-2.5,2.3]], in coslam [[-1.0,7.0],[-1.3,3.7],[-1.7,1.4]].

The effect of smooth term will be influnced by the bound. The following show the result if I use bound size [[-1.9,7.9],[-2.2,4.5],[-2.5,2.3]]. These are some artifect out of room.

3. In GoSurf you propose SDF gradient based smoothness term. Compared to that, what advantage does this feature metric smoothness term bring? By the way, why this term is only used in mapping process?

Thank you very much in advance!

Dear Authors,

Thank you for sharing this outstanding work!

I was wondering how can we separate the overall runtime that you define below:

https://github.com/JingwenWang95/neural_slam_eval/tree/main/benchmark/replica

as:

"The runtime is measured as the total processing time from the program starts until the final checkpoint is saved."

into Tracking, Mapping, and FPS as shown in Tables 1 and 2 in the main paper.

Thank you in advance for your time and effort in answering my issue.

Regards,
Karim,

Thanks very much.

Hello, I tried to calculate the total parameter of Co-SLAM by calculating all the parameters in decoder, but I found the result is much smaller than the data in the paper. I thought it was because I didn't take the learnable parameters outside the decoder into consideration. I'd like to ask how you calculate all the parameter in Co-SLAM. Thanks a lot!

Thanks for your work. In the function sdf2weights, if all the predicted SDF values of a ray are positive, the inds of the ray is 0 since the the mechanism of torch.argmax() that if all inputs are the same value, 0 will be output.
inds = torch.argmax(mask, axis=1)
So the corresponding z_min will be the config["cam"]["near"]. When the predicted SDF is all positive, it should indicates the z_min should be infinite, which makes me confused. What's more, this will make the elements in acc_map in function raw2output always equal 1.

Hi @HengyiWang, I apologize for bothering you, but I am having trouble understanding this behavior.
In particular, I attached an RGB-D camera to my drone in a Webots environment simulation. The goal was to reconstruct the map and navigate the environment using CO-SLAM. I thought this would be possible, but the issue is related to the reconstructed color on the mesh.

Co-SLAM mesh

Webots environment
My question is whether this result could be due to the fact that the images are taken from a simulated environment, and therefore, Co-SLAM has issues using them, or if I am doing something wrong?
Best regards

Hello, In the paper, you mentioned that "For experiments with default settings (Ours), which runs at 17 FPS on the Replica dataset." However, I am experiencing a significantly lower runtime of only 3-4 FPS with approximate similar configurations. Is there anything else that needs attention? Looking forward to your reply and guidance.

Hi HengyiWang,

Thanks so much for releasing your code. I'm currently confused about the coordinate system of coslam in dataloader and ray direction. Assuming a ground truth trajectory and the corresponding mesh. The trajectory poses are flipped (only y and z) by the data loader, but the ground truth mesh stays in the original coordinate system. Will It cause any problems in the following evaluation phase? Besides, If I export a ground truth trajectory and mesh from other modeling software like Blende .., is it necessary to convert to openGL in both dataloader and ray direction?

# convert poses to openGL 
c2w[:3, 1] *= -1 # * y-axis flip
c2w[:3, 2] *= -1 # * z-axis flip
c2w[:3, 3] *= self.sc_factor

# convert rays to OpenGL
if type is  'OpenGL':
    dirs = torch.stack([(i - cx)/fx, -(j - cy)/fy, -torch.ones_like(i)], -1)
elif type is 'OpenCV':
    dirs = torch.stack([(i - cx)/fx, (j - cy)/fy, torch.ones_like(i)], -1)

Hi, thanks for your excellent work. I have some questions after reading the paper. Global BA from all rays is cool and it prevent catastrophic forgetting of decoder. However, since the update of recent frame is reduced. It may harm the mapping and location performance when shifting to a newly founded area. Also, the size of bundle rise( you have to adjust more pose), the optimization speed would be slower. Do you have any insight about that? Thanks.

Hi,

Thanks for your great work. Not sure if the algorithm was tested with large view movements.

Since the Replica office0 video has high FPS, I tested the algorithm at different FPS settings.

The algorithm works fine when the FPS was reduced to 1/2, 1/3, 1/5, 1/10 and 1/20 with modifications of parameters. But the algorithm failed to estimate poses at 1/30.

There is still a large view overlapping between frames by visually inspecting the 1/30 FPS data. However, the results are very bad (wrong poses).

In my experiment, I modified the following parameters at 1/30:

• mapping_iters = 500
• mapping_keyframe_every = 1
• mapping-map_every = 1
• mapping_n_pixels = 0.1
• mapping_first_iters = 500
• tracking_iters = 500

Any idea why the algorithm doesn't work for low FPS videos ?
What I observed is that if the pose is incorrect from tracking, it is hard to correct in mapping.

Hello. I am trying to run the code on KITTI-360 dataset. I use Lidar projected images instead of depth maps. I get cuda out of memory error. What should I do?

Hello, does the hash code have to be normalized?

Dr. authors,

I understand that, please correct me if I am wrong, in Co-SLAM, tracking has to wait for mapping to complete first.

Therefore, the total time the system takes to map and track a single frame is defined by the tracking time.

Therefore, the FPS will not take into account the mapping time.

Therefore, it makes sense that tracking time should be larger than mapping time since it should include the mapping time. right?

But, I noticed that in Table 1 in the main paper, for the Replica dataset, the tracking time was less than that of the mapping (58 ms to 99 ms), yet the total FPS was counted based on the tracking time only.

Could you please clear this out for me?

Thank you in advance for your time and effort in answering my question.

Hey there, I was trying to build this extension but it seems not working, I don't know if it is a problem with my VS. Or Co-SLAM can not work on windows? Or can I Ignore this extension?

here is the last line for my error:

error: command 'C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\bin\HostX86\x64\cl.exe' failed with exit code 2

Hi @HengyiWang, thanks for open source this impressive project. I am interested in using your project for our research project, but I noticed that the repository currently lacks a clear license file. A license would provide guidance on how your project can be used, modified, or shared by others.
Would you consider adding a standard open-source license to your repository?

hello, thanks for sharing your code!
As a newbie, I would like to ask if the initial camera pose is copied from GT?
I'm using my own dataset without GT and getting very poor results.

Hello,

Great work, I really appreciate everything about this project.

I was wondering concerning the iphone dataset, I have an RGB-D dataset captured by PolyCam on Iphone with image data in shape of 1024768 and depth data in shape 256192.

I'm trying to use the iphone configuration with this dataset yet for some reason I receive some really bad representation and trajectory, I was wondering what are the parts of the configuration that you recommend working on, mainly I only changed the camera size and the intrinsics.

I tried also to use vis_bound script to generate the corresponding bounding box, but the output is always: TriangleMesh with 0 points and 0 triangles.

Hi, Thank you for this awesome work and generous sharing! I have followed the instruction and attempted to run on the replica room0, but I came across the following RuntimeError:

Noted that I have changed different locations:
self.device = torch.device("cuda:6" if torch.cuda.is_available() else "CPU")
I also tried to directly git clone from the original tinycudann repository but it obviously didn't work...
I would appreciate some suggestions and advice, Thanks a lot!

Hello Dr, I have questions about the position of your pose initialization, when I change the first frame of initialization prediction pose est_pose (0) to the given second frame gt pose, the reconstruction effect is very unsatisfactory, do you know what is the cause? Or does co-slam have to set the prediction est_pose (0) to the gt pose of the first frame to achieve pose estimation for subsequent frames?

Hi!
Thanks for your great work! We intend to evaluate some Tum sequences with your code. We found that there are some floaters in the rendering image and depth(We provide some visualizations about fr2_xyz and fr3_long_office sequence). Can you give me some suggestions or experience about this phenomenon? Do you think Hash collision is a potential cause?
Thanks!

    def freeze_model(self):
        '''
        Freeze the model parameters (Mapping)
        '''
        for param in self.model.embed_fn.parameters():
            param.require_grad = False
        
        for param in self.model.decoder.parameters():
            param.require_grad = False

Here require_grad should be requires_grad, is it necessary to unfreeze after optimizing the pose? Or has it not been frozen, and has been jointly optimized?

Hi

First of all thank you for the wonderful work and for releasing the code-base. I had some queries regarding the implementation of the SDF loss which seems slightly different than that mentioned in paper. It would be great to know the reasoning behind this-

1. Co-SLAM/model/utils.py

   Line 142 in dfb69c8

   fs_loss = compute_loss(predicted_sdf * front_mask, torch.ones_like(predicted_sdf) * front_mask, loss_type) * fs_weight

   
   In this case while computing the free space loss, it seems like the loss is being computed with respect to 1 and not w.r.t the truncation value. Is there a multiplication with the truncation value missing in the equation ?

2. Co-SLAM/model/utils.py

   Line 143 in dfb69c8

   sdf_loss = compute_loss((z_vals + predicted_sdf * truncation) * sdf_mask, target_d * sdf_mask, loss_type) * sdf_weight

   
   In this case while computing the sdf loss, why is there a multiplication with truncation ?

3. While computing the masks for sdf and freespace, is there any particular reason why only the points in the front are considered for free-space loss and not the points in the back ? This seems a bit different then the paper where everything which is not in the truncation region is considered as free space.

Thank you

Best
Manthan

Hi,
Thanks for your amazing job. I met some problem on using the tinycudann.
It seems that tinycudann v1.6 dosen't support CUDA 11.3, so I wonder which version do you use. I may manually install it.

When I was using Replica Dataset Office_0 as the dataset, I found the predicted trajectory map and the ground truth trajectory map exhibit similar shapes but differ significantly in scale, with an approximate discrepancy of 5x.
Could you please explain why this happened? Is this issue happening during drawing using Matplotlib, or is it due to errors in pose estimation during Bundle Adjustment (BA)? In other words, does this plotting with Matplotlib indicate a potential impact on SLAM performance?
Thanks a lot!

Hi, thanks for your excellent work!
I have a question after evaluating scene0000_00 with the code you provided. There are two types results pose_x.png and pose_r_x.png generated by following codes.

It seems that using pose_relative can achieve better ATE RMSE result, while the other result is more similar as the one on paper.
I wonder which result is the one you provided on paper and why you didn't use the other one.
Thank you!

Hi @HengyiWang,

Thank for your excellent work!, It's so impressive.

Any plan to release code for us to evaluate you model or provide a simple code for video real-time inference ?

Look forward to your reply!

Hi!
Thanks for your excellent work! We intend to evaluate some other datasets with the code which you have provided. Therefore, we need to reconstruct the config file following your code example. We found that these parameters, especially the bounding box, learning rate, and loss weight, significantly impact rendering and reconstruction in tracking and mapping thread, even if the GT pose is given. Can you give me some suggestions or experience about the details of parameter configuration for the new dataset such as ETH3D (how to fine-tune the parameter on the new indoor dataset)?
Thanks!

Hi Hengyi, the neural RGBD data sequences can't be downloaded at the moment. Would you happen to know why? (https://kaldir.vc.in.tum.de/ can't be accessed at all)
Great work btw!

Hello, and thank you for this wonderful repository.

I have created this scene in Blender with a very simple object and a camera rotating arround it, I unfortunatly was not able to get good result. I have tried many configurations and different conventions in order to solve this issue.

For a total of 2001 frames, I have created a camera that will be able to output both depth and rgb information and a script that which is created to change the convention from Blender into OpenCV convention, in order to work the same way Replica is ran with Co-SLAM. I my Replica in order to create my own yaml file.
The main difference in the yaml file is :
cam: H: 480 W: 640 fx: 888.8889 fy: 1000.0000 cx: 320.0 cy: 240.0
and for the bound I chose :
mapping: bound: [[-5,5],[-5,5],[-5,5]] marching_cubes_bound: [[-5,5],[-5,5],[-5,5]]
This is the visualisation of the bounding box :

This is the resulting mesh from vis_bound :

Results :

We can conclude that the convention used is correct from the way the ground truth ATE RMSE is matching, the expected ground truth.

Any suggestion where things went wrong ?

After execution
python coslam.py --config './configs/{Dataset}/{scene}.yaml
python coslam_mp.py --config './configs/{Dataset}/{scene}.yaml

First off, huge thanks for all the hard work on this repo! It's been super helpful, and I'm really grateful for what you've put together here.

I'm diving into recording my own RealSense dataset and I'm hoping to get it working with your code. Got a couple of questions I'm scratching my head over, and I'd love your input:

1. About my_room dataset:

• What's the deal with align_mat.txt and trajectory.log in the dataset? What do they actually do?
• Would it be possible for you to share the YAML config file you used for your "my_room" RealSense D435i dataset? It'd be super helpful to see how you set it up. Thanks a bunch!
• If I want to make these files for my own dataset, any tips on how to go about it? Are there any tools or methods you'd recommend?

2. Debugging my Dataset:

• I'm trying to debug some frames to see how the network is rendering the RGB and depth images. Any pointers on which part of the code I should dive into for this kind of info?

Any advice or pointers you can share would be awesome. Thanks a ton in advance!

Hi,

Thanks for your great work. I looked at your code and found the implementation uses a shared Adam optimizer for all poses in global BA.

My question is that if some poses are not selected in a global BA iteration, do they get updated in the optimizer ?

• My expectation is that if the randomly selected rays don't have pose A, pose A shouldn't be updated.
• However, the Adam optimizer may use history gradient to update all parameters even if some parameters have zero gradient.

Do you think this is a valid concern ?
Thanks.

Hello Ph.D, in the get_masks function of Model / Utils.py,
calculate

fs_weight` = 1.0 -num_fs_samples / num_samples
sdf_weight = 1.0 -num_sdf_samples / num_samples

Is it inconsistent with the formula in the paper? Whether it should be:

fs_weight = num_fs_samples / num_samples
sdf_weight = num_sdf_samples / num_samples

Thanks for your great work!I run your code in some dataset,and have some question.

1.About FPS mentioned in your main paper.I run your code in A40 Server,in replica dataset,about 2000 frames.The time

consume about 20 min in your deault set(tracking iter 10,mapping iter 20).How to understand FPS mentioned in your paper?

2.I found in Real world dataset,the mesh result is bad,such as in Scannet and tum_rbd datasets.I found tracking and mapping

iter is not enough,because the psnr is low.Can you give me some advice to improve the performance in real datasets？Add iters

or higher voxel resolution but reduce efficience?

Thanks for your reply!

Hello there. I was about to install the Co-SLAM environment, when I was installing the 'marching_cubes' according to the given README.md (which is cd external/NumpyMarchingCubes; python setup.py install) but I encountered a problem error:
'command 'E:\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe' failed with exit code 2'
I use windows platform and the python version is 3.8.
Could you please explain why? Tks!

Hello,

First and foremost, thank you for the exceptional work you've been doing.

I'm keen to know when you plan to publish tutorials on creating sequences using RealSense. Eagerly awaiting its release.

Best regards,
Lvmingzhe.

Can you tell me the equipment used during the code training phase?

Thank you for sharing your excellent work!
When I read your code, I notice that you predefine the bounding of scenes just like Nice-slam. And for the implementation of your joint representation, you mainly depend the tiny-cuda-nn to get encoding function easily, which is convenient but abstract compared to self-designed encoding function.
Thus my main concern may be whether the encoding function provided by tiny-cuda-nn can be used to encode unbounded scene incrementally, just like Vox-Fusion and H2-Mapping. Because I have noticed that tiny-cuda-nn requires normalized coordinate input, which is impractical for robots in unknown scenes.
I'd appreciate for your excellent work and profound reply!

Dear Authors.

I understand that, please correct me if I am wrong, in Co-SLAM, tracking has to wait for mapping to complete first.

Therefore, the total time the system takes to map and track a single frame is defined by the tracking time.

Therefore, the FPS will not take into account the mapping time.

Therefore, it makes sense that tracking time should be larger than mapping time since it should include the mapping time. right?

But, I noticed that in Table 1 in the main paper, for the Replica dataset, the tracking time was less than that of the mapping (58 ms to 99 ms), yet the total FPS was counted based on the tracking time only.

Could you please clear this out for me?

Thank you in advance for your time and effort in answering my question.

Thank you for sharing your excellent work！
But I have a question about your implementation of hash encoding. In model/encodings.py, you wrote this code to achieve the allocation of multiple resolutions:
per_level_scale = np.exp2(np.log2(desired_resolution / n_levels) / (n_levels - 1))
But I found that the settings for this instant-ngp are not the same, you did not use N_ Min for calculation. May I ask how to understand this code?
I would be very grateful if you could explain this code

Hi, First of all amazing work!
I am trying to do underwater 3D reconstruction , and was wondering if I could use your codebase.
Our setup is a ZED2i camera in an underwater housing. We do opencv checkerboard pattern calibration , but do not get proper tracking with zed sdk's spatial mapping.
I think NERF like approach is essential since it can properly model water which traditional approaches might fail to do.
Please share your thoughts if you find this interesting :)

PS. I think I need to create a new dataset class in https://github.com/HengyiWang/Co-SLAM/blob/main/datasets/dataset.py , can you give an explaination on that?

Hii, I have a question about the implementation of the algorithm.
In particular, for what I saw everything is done offline. You take an RGBD video, then it is passed to the Co-SLAM algorithm and it will give the reconstructed mesh of the scene and the estimated trajectory of the camera. So my question is, the Real-time SLAM that is written in the title refer just of the fact we do mapping and tracking simultaneously or the it is done in real time so we can use it to mapping and localize our robot in an environment?
Am I missing something? The second question is, if everything is offline, what is the purpose of having the trajectory of the camera?
Thank a lot for having shared your code.

Thanks so much for sharing this excellent work. It has excellent reconstruction effect, but I set it up as in the supplementary material of the article
hash_size: 14
smooth_weight: 0.001
Then run the code on office0 and sample the generated results using the same comparison method as nice-slam, and get
accuracy: 9.465963119058479
completion: 1.6472761277591914
completion ratio: 96.2565
Why is accuracy so ridiculously high?

Hello, how to visualize the process of online construction.