princeton-vl / droid-slam Goto Github PK

Hi,
First of all, thank you for making this awesome work public. really really impressive work.
My main interest is to estimate a camera's trajectory in an "online" manner.
From what I saw, in all your evaluation scripts, you use the Umeyama alignment to align and scale the predicted trajectory to the ground truth before calculating the error metrics.
In an online scenario, where you don't have access to the ground truth trajectory, how can you still get poses that accurately represent the real-world poses? meaning, aligned and up to scale.

will appreciate any ideas\thoughts on the matter. thanks.

Hi! Thanks for sharing your remarkable work!
I have compiled successfully following your guides, but it throws errors when i run the demo on ETH3D and EuRoC dataset. The error log is shown as below:

 Traceback (most recent call last):
  File "demo.py", line 119, in <module>
    traj_est = droid.terminate(image_stream(args.imagedir, args.calib, args.stride))
  File "/home/jinyu/Documents/DROID-SLAM/droid_slam/droid.py", line 81, in terminate
    self.backend(7)
  File "/home/jinyu/anaconda3/envs/droidenv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/home/jinyu/Documents/DROID-SLAM/droid_slam/droid_backend.py", line 33, in __call__
    graph.add_proximity_factors(rad=self.backend_radius, 
  File "/home/jinyu/Documents/DROID-SLAM/droid_slam/factor_graph.py", line 368, in add_proximity_factors
    ii, jj = torch.as_tensor(es, device=self.device).unbind(dim=-1)
ValueError: not enough values to unpack (expected 2, got 0)

Can you give some advice on how to solve the problem? Sincerely looking forward to your reply!

While executing the "python setup.py install" command, I receive the following error.

Python Version: 3.7
Cuda Version: 11.1
Pytorch Version: 1.10.0

Can you give some advice on how to solve the problem?

Thanks for your open source code
I am trying to train the model on KITTI. So I want to ask, how DEPTH_SCALE value is decided? currently, for tartan, it's
set to 5

Hi @zachteed @xhangHU
I couldn't use your weights "droid.pth" for training? I faced this error:

Traceback (most recent call last):
  File "train.py", line 189, in <module>
    mp.spawn(train, nprocs=args.gpus, args=(args,))
  File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 150, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/trainer/droidslam/train.py", line 60, in train
    model.load_state_dict(torch.load(args.ckpt))
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1482, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DistributedDataParallel:
        size mismatch for module.update.weight.2.weight: copying a param with shape torch.Size([3, 128, 3, 3]) from checkpoint, the shape in current model is torch.Size([2, 128, 3, 3]).
        size mismatch for module.update.weight.2.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).
        size mismatch for module.update.delta.2.weight: copying a param with shape torch.Size([3, 128, 3, 3]) from checkpoint, the shape in current model is torch.Size([2, 128, 3, 3]).
        size mismatch for module.update.delta.2.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]). 

I am trying to train the model on KITTI
These are the parameters which I am using :

 clip=2.5,  edges=24, fmax=96.0, fmin=8.0, gpus=4, iters=15, lr=5e-05, n_frames=7, noise=False, restart_prob=0.2, scale=False, steps=250000, w1=10.0, w2=0.01, w3=0.05, world_size=4

how to process backend on multi-gpus?

Can I setup the source code in GeForce RTX 2080i?
I tried in windows but it failed, especially for Eigen and Cuda.
Do you have any suggestions?
By the way, what Eigen version have you used?

error: namespace "Eigen" has no member "VectorX"
\DROID-SLAM\src\droid_kernels.cu(1100): error: this declaration has no storage class or type specifier
\DROID-SLAM\src\droid_kernels.cu(1100): error: expected a ";"
\DROID-SLAM\src\droid_kernels.cu(1107): error: namespace "Eigen" has no member "VectorX"
\DROID-SLAM\src\droid_kernels.cu(1107): error: expected a ")"
\DROID-SLAM\src\droid_kernels.cu(1104): error: identifier "b" is undefined
\DROID-SLAM\src\droid_kernels.cu(1108): error: "b" is not a nonstatic data member or base class of class "SparseBlock"
\DROID-SLAM\src\droid_kernels.cu(1148): error: identifier "b" is undefined
\DROID-SLAM\src\droid_kernels.cu(1155): error: identifier "b" is undefined
\DROID-SLAM\src\droid_kernels.cu(1155): error: class "SparseBlock" has no member "b"
error: no instance of constructor "SparseBlock::SparseBlock" matches the argument list
argument types are: (const Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<double, double>, const Eigen::SparseMatrix<double, 0, int>, const Eigen::SparseMatrix<double, 0, int>>, , const int, const int)
\DROID-SLAM\src\droid_kernels.cu(1164): error: identifier "b" is undefined
\DROID-SLAM\src\droid_kernels.cu(1182): error: identifier "b" is undefined
\DROID-SLAM\src\droid_kernels.cu(1182): error: no instance of overloaded function "Eigen::SimplicialLLT<_MatrixType, _UpLo, _Ordering>::solve [with _MatrixType=Eigen::SparseMatrix<double, 0, int>, _UpLo=1, _Ordering=Eigen::AMDOrdering]" matches the argument list
argument types are: ()
object type is: Eigen::SimplicialLLT<Eigen::SparseMatrix<double, 0, int>, 1, Eigen::AMDOrdering>

15 errors detected in the compilation of "/DROID-SLAM/src/droid_kernels.cu".

Thanks a lot.

Hi @zachteed, I have some question in the depth (disparity) reading part:

def build_frame_graph(self, poses, depths, intrinsics, f=16, max_flow=256):
    """ compute optical flow distance between all pairs of frames """
    def read_disp(fn):
        depth = self.__class__.depth_read(fn)[f//2::f, f//2::f]   # FIXME: Why down-sample? What does f mean?
        depth[depth < 0.01] = np.mean(depth)
        return 1.0 / depth

    poses = np.array(poses)
    intrinsics = np.array(intrinsics) / f
    ...

It seems a hyperparameter f is used to down-sample the disparity map / inverse depth map. This f is also used to adjust the intrinsics. I wonder what is the meaning of f here? Thanks!

Hi,

In paper its mentioned, DROID SLAM trained on monocular data generalizes to stereo and as well as to RGBD.
But when I checked the training code specially with tartan data, looks we need depth as well.

Can you please confirm is this understanding correct or am i missing something?

Thanks!

Best, KP!

I find there is a learned eta variable with small values in the BA solver, what is it for? For numerical stability？ For better gradient backward?

Where can I find eigen @ 3d4ba85?

Hi!
Thank you very much for sharing the code of DROID_SLAM.
I've trained 80,000 times on the Tartanair dataset so far, but why hasn't loss_function shown any signs of convergence so far?

System:
Windows 11
Python 3.9
Pytroch 1.10
cuda version 11.3
RTX 3070

lietorch installation works but droid_backend fails with:

droid_kernels.obj : error LNK2001: unresolved external symbol "public: long * __cdecl at::TensorBase::data_ptr<long>(void)const " (??$data_ptr@J@TensorBase@at@@QEBAPEAJXZ)
build\lib.win-amd64-3.9\droid_backends.cp39-win_amd64.pyd : fatal error LNK1120: 1 unresolved externals
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\link.exe' failed with exit code 1120

When I run setup.py to install droid_backend and lietroch I run into the a lot of warnings e.g.

C:/Users/hanna/Documents/GitHub/DROIDSLAM/venv/lib/site-packages/torch/include\c10/util/Optional.h(432): warning C4624: "c10::trivially_copyable_optimization_optional_base<T>": The destrcutor was deleted implicitly or defined as deleted
        with
        [
            T=std::vector<at::Tensor,std::allocator<at::Tensor>>
        ]

In the DROID-SLAM paper , you use different terms for bundle adjustment.
Such as ,
Local bundle adjustment
global bundle adjustment
dense bundle adjustment
full bundle adjustment
Motion-only bundle adjustment
Would share some links or would you explain it please for clear understanding, why you called full, dense, etc.?

File "/home/itadmin/DROID-SLAM/demo.py", line 100, in
droid = Droid(args)
File "/home/itadmin/DROID-SLAM/droid_slam/droid.py", line 37, in init
from visualization import droid_visualization
File "/home/itadmin/DROID-SLAM/droid_slam/visualization.py", line 8, in
import open3d as o3d
ModuleNotFoundError: No module named 'open3d'

kidly help me out

Hi, thank you for opening this great project!
When I try to install this project, I meet some compilation problem. It is probably related to conda environment, I find that there are also some other issues caused by env difference. So, I suggest you to create yaml by using conda env export > environment.yml command, so we can get the exact conda env.

Thanks for releasing this excellent work!
One small thing drew my attention and unsure if this is on purpose, which is the xi[45] at this line.

Hi, when i run python setup.py install, the compile error happened: TypeError: expected string or bytes-like object

Python version: 3.8
setuptools version: 50.3.0

Please give some advice if convenient, thanks.

Hi, thanks for the wonderful work.

I got some questions

• why use correlation_kernels.cu for corr sampling? Is it faster?
• what corr_index_forward_kernel function in correlation_kernels.cu is doing? Sorry I am a little confused.
• Is that possible to rewrite the sample corr using torch (or put it in a network format) like what RAFT did?

Thanks

Hi,

Thanks for excellent work.

1. In paper its mentioned "If the camera returns to a previously mapped region, we add long range connections in the graph to perform loop closure." Can you please tell me which portion of the code implement this?

2. Is there a ablation studies which you folks did on improvement (drift reduction) with loop closure?

Best,
KP

Hi @zachteed, Amazing work (and track of works) :), thanks a lot for open-sourcing it!

I'm curious about some aspects regarding the keyframes:

1. I see that at Sec. 3.4 of the paper, the frontend is in charge of selecting them. However, I believe that how this selection is done is not present within the Frontend explanation.

2. On the other hand, at the end of Section 3.4, it is mentioned that non-keyframe frames perform motion-only bundle adjustment, whereas the front-end performs local bundle-adjustment. Does this mean that unless the incoming new frame is selected as a keyframe, it won't be optimized in the frontend / backend?

I'd highly appreciate if you could shed some light on this,

Hi, thanks for the wonderful job!
However, I have met the error when I run the demo:
python demo.py --imagedir=data/abandonedfactory --calib=calib/tartan.txt --stride=2

The error happens after the first image is shown which is:

1it [00:07, 7.25s/it]
Traceback (most recent call last):
File "demo.py", line 102, in
droid.track(t, image, intrinsics=intrinsics)
File "droid_slam/droid.py", line 66, in track
self.filterx.track(tstamp, image, depth, intrinsics)
File "/home/.conda/envs/py37/lib/python3.7/site-packages/torch/autocast_mode.py", line 198, in decorate_autocast
return func(*args, **kwargs)
File "/home/.conda/envs/py37/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, kwargs)
File "droid_slam/motion_filter.py", line 69, in track
corr = CorrBlock(self.fmap[None,[0]], gmap[None,[0]])(coords0)
File "droid_slam/modules/corr.py", line 47, in call
corr = CorrSampler.apply(self.corr_pyramid[i], coords/2i, self.radius)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Traceback (most recent call last):
File "", line 1, in
File "/home/.conda/envs/py37/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "/home/.conda/envs/py37/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/home/.conda/envs/py37/lib/python3.7/multiprocessing/synchronize.py", line 110, in setstate
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory

Could you help me figure it out?

error: could not convert ‘torch::kFanIn’ from ‘const torch::enumtype::kFanIn’ to ‘torch::nn::init::FanModeType {aka c10::variant<torch::enumtype::kFanIn, torch::enumtype::kFanOut>}’
init::kaiming_uniform_(weight, /a=/std::sqrt(5)); // NOLINT(cppcoreguidelines-avoid-magic-numbers)
~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:66:23: error: could not convert ‘torch::kLeakyReLU’ from ‘const torch::enumtype::kLeakyReLU’ to ‘torch::nn::init::NonlinearityType {aka c10::variant<torch::enumtype::kLinear, torch::enumtype::kConv1D, torch::enumtype::kConv2D, torch::enumtype::kConv3D, torch::enumtype::kConvTranspose1D, torch::enumtype::kConvTranspose2D, torch::enumtype::kConvTranspose3D, torch::enumtype::kSigmoid, torch::enumtype::kTanh, torch::enumtype::kReLU, torch::enumtype::kLeakyReLU>}’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h: In instantiation of ‘void torch::nn::ConvNdImpl<D, Derived>::reset_parameters() [with long unsigned int D = 2ul; Derived = torch::nn::ConvTranspose2dImpl]’:
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:62:17: required from ‘void torch::nn::ConvNdImpl<D, Derived>::reset() [with long unsigned int D = 2ul; Derived = torch::nn::ConvTranspose2dImpl]’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/optim/sgd.h:48:48: required from here
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:66:23: error: could not convert ‘torch::kFanIn’ from ‘const torch::enumtype::kFanIn’ to ‘torch::nn::init::FanModeType {aka c10::variant<torch::enumtype::kFanIn, torch::enumtype::kFanOut>}’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:66:23: error: could not convert ‘torch::kLeakyReLU’ from ‘const torch::enumtype::kLeakyReLU’ to ‘torch::nn::init::NonlinearityType {aka c10::variant<torch::enumtype::kLinear, torch::enumtype::kConv1D, torch::enumtype::kConv2D, torch::enumtype::kConv3D, torch::enumtype::kConvTranspose1D, torch::enumtype::kConvTranspose2D, torch::enumtype::kConvTranspose3D, torch::enumtype::kSigmoid, torch::enumtype::kTanh, torch::enumtype::kReLU, torch::enumtype::kLeakyReLU>}’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h: In instantiation of ‘void torch::nn::ConvNdImpl<D, Derived>::reset_parameters() [with long unsigned int D = 1ul; Derived = torch::nn::ConvTranspose1dImpl]’:
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:62:17: required from ‘void torch::nn::ConvNdImpl<D, Derived>::reset() [with long unsigned int D = 1ul; Derived = torch::nn::ConvTranspose1dImpl]’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/optim/sgd.h:48:48: required from here
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:66:23: error: could not convert ‘torch::kFanIn’ from ‘const torch::enumtype::kFanIn’ to ‘torch::nn::init::FanModeType {aka c10::variant<torch::enumtype::kFanIn, torch::enumtype::kFanOut>}’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:66:23: error: could not convert ‘torch::kLeakyReLU’ from ‘const torch::enumtype::kLeakyReLU’ to ‘torch::nn::init::NonlinearityType {aka c10::variant<torch::enumtype::kLinear, torch::enumtype::kConv1D, torch::enumtype::kConv2D, torch::enumtype::kConv3D, torch::enumtype::kConvTranspose1D, torch::enumtype::kConvTranspose2D, torch::enumtype::kConvTranspose3D, torch::enumtype::kSigmoid, torch::enumtype::kTanh, torch::enumtype::kReLU, torch::enumtype::kLeakyReLU>}’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h: In instantiation of ‘void torch::nn::ConvNdImpl<D, Derived>::reset_parameters() [with long unsigned int D = 3ul; Derived = torch::nn::Conv3dImpl]’:
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:62:17: required from ‘void torch::nn::ConvNdImpl<D, Derived>::reset() [with long unsigned int D = 3ul; Derived = torch::nn::Conv3dImpl]’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/optim/sgd.h:48:48: required from here
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:66:23: error: could not convert ‘torch::kFanIn’ from ‘const torch::enumtype::kFanIn’ to ‘torch::nn::init::FanModeType {aka c10::variant<torch::enumtype::kFanIn, torch::enumtype::kFanOut>}’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:66:23: error: could not convert ‘torch::kLeakyReLU’ from ‘const torch::enumtype::kLeakyReLU’ to ‘torch::nn::init::NonlinearityType {aka c10::variant<torch::enumtype::kLinear, torch::enumtype::kConv1D, torch::enumtype::kConv2D, torch::enumtype::kConv3D, torch::enumtype::kConvTranspose1D, torch::enumtype::kConvTranspose2D, torch::enumtype::kConvTranspose3D, torch::enumtype::kSigmoid, torch::enumtype::kTanh, torch::enumtype::kReLU, torch::enumtype::kLeakyReLU>}’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h: In instantiation of ‘void torch::nn::ConvNdImpl<D, Derived>::reset_parameters() [with long unsigned int D = 2ul; Derived = torch::nn::Conv2dImpl]’:
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:62:17: required from ‘void torch::nn::ConvNdImpl<D, Derived>::reset() [with long unsigned int D = 2ul; Derived = torch::nn::Conv2dImpl]’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/optim/sgd.h:48:48: required from here
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:66:23: error: could not convert ‘torch::kFanIn’ from ‘const torch::enumtype::kFanIn’ to ‘torch::nn::init::FanModeType {aka c10::variant<torch::enumtype::kFanIn, torch::enumtype::kFanOut>}’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:66:23: error: could not convert ‘torch::kLeakyReLU’ from ‘const torch::enumtype::kLeakyReLU’ to ‘torch::nn::init::NonlinearityType {aka c10::variant<torch::enumtype::kLinear, torch::enumtype::kConv1D, torch::enumtype::kConv2D, torch::enumtype::kConv3D, torch::enumtype::kConvTranspose1D, torch::enumtype::kConvTranspose2D, torch::enumtype::kConvTranspose3D, torch::enumtype::kSigmoid, torch::enumtype::kTanh, torch::enumtype::kReLU, torch::enumtype::kLeakyReLU>}’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h: In instantiation of ‘void torch::nn::ConvNdImpl<D, Derived>::reset_parameters() [with long unsigned int D = 1ul; Derived = torch::nn::Conv1dImpl]’:
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:62:17: required from ‘void torch::nn::ConvNdImpl<D, Derived>::reset() [with long unsigned int D = 1ul; Derived = torch::nn::Conv1dImpl]’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/optim/sgd.h:48:48: required from here
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:66:23: error: could not convert ‘torch::kFanIn’ from ‘const torch::enumtype::kFanIn’ to ‘torch::nn::init::FanModeType {aka c10::variant<torch::enumtype::kFanIn, torch::enumtype::kFanOut>}’
/home/sofa/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/conv.h:66:23: error: could not convert ‘torch::kLeakyReLU’ from ‘const torch::enumtype::kLeakyReLU’ to ‘torch::nn::init::NonlinearityType {aka c10::variant<torch::enumtype::kLinear, torch::enumtype::kConv1D, torch::enumtype::kConv2D, torch::enumtype::kConv3D, torch::enumtype::kConvTranspose1D, torch::enumtype::kConvTranspose2D, torch::enumtype::kConvTranspose3D, torch::enumtype::kSigmoid, torch::enumtype::kTanh, torch::enumtype::kReLU, torch::enumtype::kLeakyReLU>}’

I want to inference on multiple GPUs. However, none of the evaluation scripts seem to support multi-GPU.
What should I do to fix this? Please let me know if there are any code examples in this repository that can help me.

Thanks for your great work! When I run the script "./tools/download_sample_data.sh".
In line 10 to line 12

gdown https://drive.google.com/uc?id=0B-ePgl6HF260NzQySklGdXZyQzA
unzip Barn.zip
rm Barn.zip

The url "https://drive.google.com/uc?id=0B-ePgl6HF260NzQySklGdXZyQzA" does not appear to exist. I would appreciate it that anyone can help me!

Is there a standard way to get the camera intrinsic parameters given just the images?

I would like to begin by saying that this is indeed an exceptional effort. I am impressed by the robustness and stability of DROID-SLAM in particular. I am currently trying to implement DROID-SLAM on a Jetson Xavier NX. I understand that a GPU with at least 11GB of memory is needed for inference, but I am not currently interested in real-time performance. The Jetson Xavier NX board that I am using currently is running with Python 3.6, with CUDA 10.2 and Pytorch 1.10, and I am getting the following error:

Hi, thank you for sharing your great work!

I would like to output not just estimated pose but also estimated depth like demo.py which you provided.
But I could not know much about how to change the code.

Could you please provide the steps and/or information to change the code?

Thank you,

The projection function proj() in droid_slam/geom/projective_ops.py (L44) has a threshold for Z. Similar thresholding for Z is also applied in projective_transform() to generate a valid map. I am confused about the the way of thresholding and its aim:

1. In projective_transform() and proj(), it seems the threshold is used to exclude points too close to camera. However, Z is the third coordinate in the 3D homogenous coordinates, which is not the actual depth. The actual depth should be Z/D. Thus, why does it threshold Z instead of Z/D?
2. projective_transform() and proj() have different thresholds for Z: MIN_DEPTH and 0.5*MIN_DEPTH respectively. Is there any reason?
3. I am curious why we should exclude points too close to camera. Is it because if the points are too close, the division might be inaccurate and lead to large error? However, the threshold is not very small (MIN_DEPTH=0.2), which seems too loose for the division stability.

Thank you for your great work, but i cannot install 'droid_backends' on cudatoolkit=10.1，is it must be 11.1 according to environment.yaml？

Hi, thank you for your great work! Recently when I read the data loading part of Droid-SLAM, I found the pose is parsed as:

poses = np.loadtxt(osp.join(scene, 'pose_left.txt'), delimiter=' ')
poses = poses[:, [1, 2, 0, 4, 5, 3, 6]]   

From the TartanAir tools, the line of the pose data file has the format tx ty tz qx qy qz qw, which uses a NED frame. In your implementation it seems you convert XYZ to YZX coordinates. I wonder if there is any reason behind it? Thanks!

0it [00:00, ?it/s]./checkpoints/bla_250000.pth
/root/anaconda3/envs/droidenv5/lib/python3.9/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1639180487213/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
51it [00:06, 7.47it/s]
################################
Traceback (most recent call last):
File "/root/docker2/droid/2new/DROID-SLAM/demo.py", line 117, in
traj_est = droid.terminate(image_stream(args.imagedir, args.calib, args.stride))
File "/root/docker2/droid/2new/DROID-SLAM/droid_slam/droid.py", line 81, in terminate
self.backend(7)
File "/root/anaconda3/envs/droidenv5/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "/root/docker2/droid/2new/DROID-SLAM/droid_slam/droid_backend.py", line 33, in call
graph.add_proximity_factors(rad=self.backend_radius,
File "/root/docker2/droid/2new/DROID-SLAM/droid_slam/factor_graph.py", line 368, in add_proximity_factors
ii, jj = torch.as_tensor(es, device=self.device).unbind(dim=-1)
ValueError: not enough values to unpack (expected 2, got 0)
Traceback (most recent call last):
File "", line 1, in
File "/root/anaconda3/envs/droidenv5/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/root/anaconda3/envs/droidenv5/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/root/anaconda3/envs/droidenv5/lib/python3.9/multiprocessing/synchronize.py", line 110, in setstate
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
[W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

Thanks for your excellent work!
I have a little confusion about the calculation of jacob of depth below:

I would like to know where in the code the last four digits of the quaternion are converted into rotation matrices, because I am trying to train with other datasets, using rotation matrices instead of xyzw quaternions。
thank you！

Hey there!

I'm trying to use DROIDSLAM with RGBD input.

I noticed that the track function in droid.py accepts depth as an input, but no info on the correct format, or if something more must be done.

This is my new image_stream function:
`def image_stream(imagedir, depthdir, calib, stride):
calib = np.loadtxt(calib, delimiter=" ")
fx, fy, cx, cy = calib[:4]

K = np.eye(3)
K[0,0] = fx
K[0,2] = cx
K[1,1] = fy
K[1,2] = cy

image_list = sorted(os.listdir(imagedir))[::stride]
depth_list = sorted(os.listdir(depthdir))[::stride]

for t, (imfile,dfile) in enumerate(zip(image_list,depth_list)):
    image = cv2.imread(os.path.join(imagedir, imfile))
    depth = cv2.imread(os.path.join(depthdir, dfile),-1)

    if len(calib) > 4:
        image = cv2.undistort(image, K, calib[4:])

    h0, w0, _ = image.shape
    h1 = int(h0 * np.sqrt((384 * 512) / (h0 * w0)))
    w1 = int(w0 * np.sqrt((384 * 512) / (h0 * w0)))

    image = cv2.resize(image, (w1, h1))
    image = image[:h1-h1%8, :w1-w1%8]
    image = torch.as_tensor(image).permute(2, 0, 1)

    depth = cv2.resize(depth, (w1, h1))
    depth = depth[:h1-h1%8, :w1-w1%8]
    depth = torch.as_tensor(depth)

    intrinsics = torch.as_tensor([fx, fy, cx, cy])
    intrinsics[0::2] *= (w1 / w0)
    intrinsics[1::2] *= (h1 / h0)

    yield t, image[None], depth[None], intrinsics`

This is my current error:

RuntimeError: expand(torch.FloatTensor{[0, 41, 584]}, size=[41, 73]): the number of sizes provided (2) must be greater or equal to the number of dimensions in the tensor (3)

Any help on this matter would be greatly appreciated, thanks for this amazing software.

Traceback (most recent call last):
File "demo.py", line 15, in
from droid import Droid
File "/home/vip/DROID-SLAM/droid_slam/droid.py", line 5, in
from droid_net import DroidNet
File "/home/vip/DROID-SLAM/droid_slam/droid_net.py", line 8, in
from modules.corr import CorrBlock
File "/home/vip/DROID-SLAM/droid_slam/modules/corr.py", line 4, in
import droid_backends
ModuleNotFoundError: No module named 'droid_backends'

The code is missing this module,How to solve it!?
Thanks!

Hi,

There is redundant calculation of cnet features in MotionFilter:__context_encoder at line no 35. which is not used.

Thanks!

Best, KP

Hi,

Thanks for uploading the amazing work. I tested the pretrained model on KITTI dataset to see how it works on outdoor env. It gave an APE around 40. I assume it's becuase the data distributon of KITTI is extremly different from datasets the model has trained on, and therefore DROID-SLAM gave very bad performance compared to ORB-SLAM3. Appearantly, ORB-SLAM3 does not have any assumption(prior) on how data distributes.

I'm wondering if you have tested on such dataset and how DROID-SLAM performs?

Thanks a lot!

Thanks for your great work. I have downloaded the model and I have faced an issue in running demos. when I run the first command "python demo.py --imagedir=data/abandonedfactory --calib=calib/tartan.txt --stride=2" I get an error "FileNotFoundError: Error: Could not find /home/cdh-h91/anaconda3/lib/python3.9/site-packages/lietorch/lib/liblietorch.so" I could not find liblietorch.so anywhere, I have installed lietorch... Any help would be appreciated!

in demo.py, there two lines, where you use 384 and 512:

 h1 = int(h0 * np.sqrt((384 * 512) / (h0 * w0)))
 w1 = int(w0 * np.sqrt((384 * 512) / (h0 * w0)))

is it fixed values? what will be for higher resolution images?
I mean how can I choose those values to resize the images?

Thanks in advanced

The evaluation scripts provide some useful tools for evaluate the trajectory quality based on evo.

It seems that if I didn't have a groundtruth file, I will not be able to get the trajectory file with time stamps.

Can you give some advices?

Requested to kindly provide the least or exact version required for following
Python
Open3d
Pytorch

Hello, thank you for sharing your great work!

I'd like to create a dense 3D point cloud of the scene similar to the one in DeepV2D.

Could you help give me the steps and/or information I need to alter the code.

Thank you

i saw 'args.stereo = False' in demo.py,

Hi,thanks for your code.I'm very interesting in it.But when I run the setup.py,I have a problem.
my driver:Nvidia 2080Ti
cuda:10.2
pytorch:1.10.0

running install
/home/u/anaconda3/envs/droidenv/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
setuptools.SetuptoolsDeprecationWarning,
/home/u/anaconda3/envs/droidenv/lib/python3.7/site-packages/setuptools/command/easy_install.py:159: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
EasyInstallDeprecationWarning,
running bdist_egg
running egg_info
writing droid_backends.egg-info/PKG-INFO
writing dependency_links to droid_backends.egg-info/dependency_links.txt
writing top-level names to droid_backends.egg-info/top_level.txt
/home/u/anaconda3/envs/droidenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py:381: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
reading manifest file 'droid_backends.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'droid_backends.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'droid_backends' extension
gcc -pthread -B /home/u/anaconda3/envs/droidenv/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/u/Documents/RS/thirdparty/eigen -I/home/u/anaconda3/envs/droidenv/lib/python3.7/site-packages/torch/include -I/home/u/anaconda3/envs/droidenv/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/u/anaconda3/envs/droidenv/lib/python3.7/site-packages/torch/include/TH -I/home/u/anaconda3/envs/droidenv/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda-10.2/include -I/home/u/anaconda3/envs/droidenv/include/python3.7m -c src/droid.cpp -o build/temp.linux-x86_64-3.7/src/droid.o -O3 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -DTORCH_EXTENSION_NAME=droid_backends -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/usr/local/cuda-10.2/bin/nvcc -I/home/u/Documents/RS/thirdparty/eigen -I/home/u/anaconda3/envs/droidenv/lib/python3.7/site-packages/torch/include -I/home/u/anaconda3/envs/droidenv/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/u/anaconda3/envs/droidenv/lib/python3.7/site-packages/torch/include/TH -I/home/u/anaconda3/envs/droidenv/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda-10.2/include -I/home/u/anaconda3/envs/droidenv/include/python3.7m -c src/droid_kernels.cu -o build/temp.linux-x86_64-3.7/src/droid_kernels.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -O3 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=droid_backends -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
nvcc fatal : Unsupported gpu architecture 'compute_80'
error: command '/usr/local/cuda-10.2/bin/nvcc' failed with exit status 1

How can I sovle it?Is it because the video card don't have enough computing power?

src/droid_kernels.cu:15:10: fatal error: Eigen/Sparse: No such file or directory
#include <Eigen/Sparse>
^~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/local/cuda-11/bin/nvcc' failed with exit code 1

when i run python setup.py install, i got some problem, can you help me~

Since I am running on the server and cannot visually see the results, I want to know how to save the results, can you help me, thank you