should we use our own sensors' (monocular+imu) picture?

I am working with OpenVins and I have noticed something that is unclear for me, in particular in "ov_msckf/src/state/Propagator.cpp" at lines 327:328 reported below:

Qc.block(6,6,3,3) = _noises.sigma_wb_2/dt*Eigen::Matrix<double,3,3>::Identity(); Qc.block(9,9,3,3) = _noises.sigma_ab_2/dt*Eigen::Matrix<double,3,3>::Identity();

based on what reported in "Indirect Kalman Filter for 3D Attitude Estimation Technical Report
Number 2005-002, Rev. 57 March 2005" an on my knowledge I would like to say that the division "/" should be substituted by a multiplication "*" for the biases as follows:

Qc.block(6,6,3,3) = _noises.sigma_wb_2*dt*Eigen::Matrix<double,3,3>::Identity(); Qc.block(9,9,3,3) = _noises.sigma_ab_2*dt*Eigen::Matrix<double,3,3>::Identity();

Thanks.

VioManager spells initialized wrong on this line:

Hi Patrick,

Thanks for sharing the codes!

I read and understand the derivation about the error-state propagation.

But I have a question about the difference between your derivation and the one described in Joan Sola's "Quaternion kinematics for the error-state Kalman filter". In Joan's paper, the error-state kinematics is derived by combining the kinematics of true and norminal state, and then the error state propagation is obtained through certain integration method. This should also be correct, but the result (using Euler integration as a sample, picture below) is different from yours (the major difference is the missing of Jr_so3 in Joan's derivation).
I'm wondering why this happens? THANKS!

Hi,
I have two questions about the tracking part and really hope you could help.

(1) in the part of "check maximum baseline" in FeatureIntializer.cpp. Is there any reference about the way we calculate the base_line as
double base_line = ((Q.block(0,1,3,2)).transpose() * p_CiinA).norm();

(2)In the part of KLT based stereo tracking, only features that are successfully tracked in both cameras are good features.
If we do not do feature matching between left and right cameras and only do KLT individually on each camera. with the extrinsics of two cameras, features that cannot be mathched between two cameras, but can be tracking on one of the cameras will also be used to update the state. Do you have any idea about the performance of this two methods?

Thanks,

So I have created a system that uses the non-ROS OpenVINS and sends IMU/Camera data from the dataset files similar to the ros example subscribing to the ROS bag. I seem to be having some timing issues that are resulting the following to be printed to stderr:
Propagator::select_imu_readings(): There are not enough measurements to propagate with 0 sec missing Propagator::select_imu_readings(): IMU-CAMERA time offset is likely messed up, check time offset value!!! /tmp/workspace/catkin_ws_ov/src/open_vins/ov_msckf/src/state/Propagator.cpp on line 207

This is causing the VIO to drift a lot. I'm a little confused as to what causes this issue to occur so it is hard for me to debug my system. If you guys could give insight as to why an issue like this would happen that would help me a lot.

Hi,

In Simulator::get_next_cam in src/sim/Simulator.cpp, this part modifies the feature ids

        // Append the map size so all cameras have unique features in them (but the same map)
        // Only do this if we are not enforcing stereo constraints between all our cameras
        for (size_t f=0; f<uvs.size() && !use_stereo; f++){
            uvs.at(f).first += i*featmap.size();
        }

in feed_measurement_simulation in /src/track/TrackSIM.cpp the ids are

        // Update our feature database, with theses new observations
        // NOTE: we add the "currid" since we need to offset the simulator
        // NOTE: ids by the number of aruoc tags we have specified as tracking
        for(const auto &feat : feats.at(i)) {

            // Get our id value
            size_t id = feat.first+currid;

it seems that each feature measurement in one frame has a unique feature id, so I am wondering how does the tracker know that the measurements across different frames are from the same feature?

This would be useful for applications that require some sort of pose to begin their behavior.

Why to use equalizeHist? As we known, this may break down the brightness constancy assumption in Optical Flow Algorithm, and as a result, KLT may fail in some cases.

A trivial question puzzled me.
Lots of comparison of two double type variables in open_vins, like:
if (timefeat == timestamp) { //timefeat and timestamp are double
has_timestamp = true;
break;
}
However I didn't find the definenation of operator “==”.
And in some std::map usage, double is the key. I didn't find the hashfunction of < or == either.
Maybe I lost important details, could you please tell me where the implementation is.

https://github.com/rpng/open_vins/blob/master/ov_core/src/feat/FeatureInitializer.cpp#L224
Hi, authors. Shouldn't this be p_AinCi.noalias() = -R_AtoCi*p_CiinA;?

In measurement update, QR decomposition using Givens rotations is not cheap, and H must be multiplied by Q. The K matrix could be calculated using sequential measurement update, for R is always diagonal. In this method matrix inversion is not required and no additonal process is needed.

In this video, https://www.youtube.com/watch?v=MCzTF9ye2zw&t=30s,
ZV-UP is used when the vehicle is static, can you share some references or open the feature in this repo.

[ 68%] Built target ov_core_lib
[ 73%] �[32m�[1mLinking CXX executable /home/ivy/workspace/catkin_ws_ov/devel/.private/ov_core/lib/ov_core/test_tracking�[0m
[ 78%] �[32m�[1mLinking CXX executable /home/ivy/workspace/catkin_ws_ov/devel/.private/ov_core/lib/ov_core/test_repeat�[0m
[ 84%] �[32m�[1mLinking CXX executable /home/ivy/workspace/catkin_ws_ov/devel/.private/ov_core/lib/ov_core/test_simulator�[0m
/home/ivy/workspace/catkin_ws_ov/devel/.private/ov_core/lib/libov_core_lib.so: undefined reference to `cv::aruco::detectMarkers(cv::_InputArray const&, cv::Ptr<cv::aruco::Dictionary> const&, cv::_OutputArray const&, cv::_OutputArray const&, cv::Ptr<cv::aruco::DetectorParameters> const&, cv::_OutputArray const&, cv::_InputArray const&, cv::_InputArray const&)'
collect2: error: ld returned 1 exit status
/home/ivy/workspace/catkin_ws_ov/devel/.private/ov_core/lib/libov_core_lib.so: undefined reference to `cv::aruco::detectMarkers(cv::_InputArray const&, cv::Ptr<cv::aruco::Dictionary> const&, cv::_OutputArray const&, cv::_OutputArray const&, cv::Ptr<cv::aruco::DetectorParameters> const&, cv::_OutputArray const&, cv::_InputArray const&, cv::_InputArray const&)'
CMakeFiles/test_repeat.dir/build.make:219: recipe for target '/home/ivy/workspace/catkin_ws_ov/devel/.private/ov_core/lib/ov_core/test_repeat' failed
make[2]: *** [/home/ivy/workspace/catkin_ws_ov/devel/.private/ov_core/lib/ov_core/test_repeat] Error 1
collect2: error: ld returned 1 exit status
CMakeFiles/Makefile2:2418: recipe for target 'CMakeFiles/test_repeat.dir/all' failed
make[1]: *** [CMakeFiles/test_repeat.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
CMakeFiles/test_simulator.dir/build.make:219: recipe for target '/home/ivy/workspace/catkin_ws_ov/devel/.private/ov_core/lib/ov_core/test_simulator' failed
make[2]: *** [/home/ivy/workspace/catkin_ws_ov/devel/.private/ov_core/lib/ov_core/test_simulator] Error 1
CMakeFiles/Makefile2:2381: recipe for target 'CMakeFiles/test_simulator.dir/all' failed
make[1]: *** [CMakeFiles/test_simulator.dir/all] Error 2
/home/ivy/workspace/catkin_ws_ov/devel/.private/ov_core/lib/libov_core_lib.so: undefined reference to `cv::aruco::detectMarkers(cv::_InputArray const&, cv::Ptr<cv::aruco::Dictionary> const&, cv::_OutputArray const&, cv::_OutputArray const&, cv::Ptr<cv::aruco::DetectorParameters> const&, cv::_OutputArray const&, cv::_InputArray const&, cv::_InputArray const&)'
collect2: error: ld returned 1 exit status
CMakeFiles/test_tracking.dir/build.make:219: recipe for target '/home/ivy/workspace/catkin_ws_ov/devel/.private/ov_core/lib/ov_core/test_tracking' failed
make[2]: *** [/home/ivy/workspace/catkin_ws_ov/devel/.private/ov_core/lib/ov_core/test_tracking] Error 1
CMakeFiles/Makefile2:899: recipe for target 'CMakeFiles/test_tracking.dir/all' failed
make[1]: *** [CMakeFiles/test_tracking.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2

I'm trying to build open vins, but it fails. How can I fix it?

Question in the title.

For more info, I am trying to run open_vins on a Jetson AGX which ships with ubuntu 18.04 so I can't install ros-kinetic unless I build the whole thing from source.

Using the following launch file, pgeneva_serial_eth.launch, with OpenVINS and running the V1_02_medium dataset the poses drift to infinity. Has anybody else ran into this issue?

I am trying to get the current estimated pose while open_vins SLAM is running. Would the VioManager::get_state() be the function I want to use? I basically just want the final translation and rotation that is estimated by the system.

Hi, I am currently running the simulation by the command roslaunch ov_msckf pgeneva_sim.launch with the original provided launch file. However, I am not able to run the simulation properly. It shows that none of the features are used in MSCKF update. Any advice? Thanks!

Hi,
As the default noise parameter _noises is in the continuous case, for the bias driving noise, should we use _noises.sigma_wb_2*dt instead of _noises.sigma_wb_2/dt to convert to discrete according to (130)?

// Note that we need to convert our continuous time noises to discrete
// Equations (129) amd (130) of Trawny tech report
Eigen::Matrix<double,12,12> Qc = Eigen::Matrix<double,12,12>::Zero();
Qc.block(0,0,3,3) = _noises.sigma_w_2/dtEigen::Matrix<double,3,3>::Identity();
Qc.block(3,3,3,3) = _noises.sigma_a_2/dtEigen::Matrix<double,3,3>::Identity();
Qc.block(6,6,3,3) = _noises.sigma_wb_2/dtEigen::Matrix<double,3,3>::Identity();
Qc.block(9,9,3,3) = _noises.sigma_ab_2/dtEigen::Matrix<double,3,3>::Identity();

Hi, if I understand correctly, there's no gating test (eq. 6 in this paper) in your code. The only outlier rejection is the ransac used in tracking, is that correct? I am wondering whether there's a reason why there's no gating test used in this codebase?

Hi, when loop through current SLAM features and grab them for update, is that possible "feats_slam" already has "feat2" from "feats_maxtracks" ? If so, "feat2" will be double used ?

VioManager.cpp (line 599-607 )
if(feat2 != nullptr) feats_slam.push_back(feat2);

I get an error when building, at lines 48 and 59 of ov_core/src/track/TrackAruco.h . I have OpenCV 3.2. Which version of OpenCV are you using?

When I use my own datasets,the terminal appears following error:
[ERROR] [1575476563.620017963]: cv_bridge exception: [8UC1] is not a color format. but [mono8] is. The conversion does not make sense.
I don't know how to solve it,could you help me,thank you!

Thank you very much for prompt reply for issue #31.
I appreciate your help!!

I have one more question: Is it possible to read the V1_01_easy.bag in separate terminal using rosplay for example. rosbag play V1_01_easy.bag.bag --pause -s 10.

to achieve this, which file should be modified? an example would be really helpful.
Looking forward to hear from you. Thank you very much.

Does OpenVINS use the default ROS coordinate system? Also if I am using OpenVINS with the EUROC dataset are the values going to be in the same frame as the vicon ball on the drone?

Hi,
When will you add schmidt ekf (As your paper SEVIS proposed) to this system?

added support to ready stereo feed, and IMU , I am using the Zed mini from stereoLabs.

I added the corresponding offline calibration parameters and noise parameter of IMU and tried modifying the "_sigma_px" parameters. But it seems not to help much

After it starts the launch file, the inertial frame moves towards infinity specially on the z axis.
Do you have any idea why this is happening?
I was wondering whether if there is something on the KTL that is causing this? also I ran it with the KTL set to false (discrete mode), and it broke with an exception.

any ideas?

I see from your documentation that you are using camera/IMU but don't see info about hardware. What are you using and calibrating for your tests?

https://docs.openvins.com/gs-calibration.html

Thanks for the info!

I am doing work with OpenVINS that requires it to be used without ROS. I have been able to hack around the build process to use OpenVINS in this way but I won't be able to pull updates from the upstream repo without a lot of manual work. I was wondering if you guys have considered building OpenVINS in a way that even the core libraries don't rely on ROS. Much of the core libraries are built in that way from what I have seen but they still #include ros/ros.h for the logging abilities. I think if OpenVINS were to have a standalone build to use as a library it would make it more widely used in a variety of different areas.

re. kalibr distortion, make sure my understanding is correct (w/ pinhole)?

• if fisheye, using equi
• if not, using radtan

Hi,
In Propagator.cpp, when do_fej, why do you use line280 instead of line281 (and similarly 283/284, 288/289, 294/295)? I think the line 281 is the correct one according to derivative of FEJ methods.

Thanks : )

Hi, thanks for the code! Do you have plans to incorporate g2o or Ceres for pose graph optimization into this?

Hi, thanks for opening this project. It runs well in my self-collected dataset !

But I found a small visualization problem when showing the camera calibration on RViz. Perhaps the codes in https://github.com/rpng/open_vins/blob/master/ov_msckf/src/core/RosVisualizer.cpp #Line261 & 268 should be modified since the quaternion representation used in your code is not the same as in tf::Quaternion?

See this also:

Hi, when do you plan to release these two features?

1. Leverage this sliding-window batch method to perform SFM initialization of all methods.
2. Support for arbitrary Schmidt'ing of state elements allowing for modeling of their prior uncertainties but without optimizing their values online.

Kostia

Hi,

Thank you for open-sourcing such an amazing tool.

I hoped I could get some advice. I needed to explicitely get the noise/bias corrected estimates of the IMU angular velocity and linear acceleration. These corrected measurements are used in the Propagation step as described here. My question is what are the correct values I should be looking to get and what is the proper way to extract them into ROS topics? I had two ideas and would appreciate your comments.

1. Create a new member publisher in RosVisualizer class and use it to publish the gyroscope and accelerometer biases in the method publish_state by getting the bias values using calls like state->imu()->bias_g(). This, however, would require that I somehow subtract this value from the correct imu measurement, which would need some sort of synchronization. I would prefer to avoid this.

2. Make the local variables w_hat and a_hat in lines 299 and 300 a member of Propagator class, and obtain them from RosVisualizer through its VioManager* app member. I might need to create a method get_propagator in VioManager and get_w_hat, get_a_hat in Propagator. Finally use a call like _app -> get_propagator() -> get_w_hat() from RosVisualizer. If this was sensible, should I extract w_hat, a_hat or w_hat2, a_hat2?

This diagram shows, in red, the previous ideas:

Finally, I made some changes to publish biases. I got the following results using the Euroc easy rosbag. The field linear_acceleration is in deed acceleration_bias and angular_velocity is in deed gyroscope_bias. I was wondering if the estimated gyroscope biases are what is to be expected, since are very different from accelerometer biases and very small.

A zoomed-in view:

Thank you!

Hi, authors. Hope I don't get the code wrong, but it seems that the jacobian matrix with respect to time offset is not calculated in https://github.com/rpng/open_vins/blob/master/ov_msckf/src/update/UpdaterHelper.cpp?

During initialization the FEJ value should also be set:
https://github.com/rpng/open_vins/blob/master/ov_msckf/src/core/VioManager.cpp#L280
https://github.com/rpng/open_vins/blob/master/ov_msckf/src/core/VioManager.h#L115

Thank you.

Thank you very much for the open source software.
Is it possible to upload and launch file for Monocular vision?
Or
How we can modify the existing launch file to make it function like Monocular camera?
Thank you very much in advance and for your suggestions.

As the title says, I was wondering if the current version of OpenVINS uses the SEVIS algorithm mentioned in this paper: https://arxiv.org/pdf/1903.08636.pdf

Hi,

What are the specific camera calibration values needed for test_webcam script on line 94? Is it in the format of "intrinsics" then distortion_coeffs?

I have set up a system where a EUROC dataset reader mimics the timing of the sensors in the dataset and passes the data to VioManager similar to a ros subscriber system. VioManager is setup with the params in the pgeneva_ros_eth.launch file. When I am running the V1_01_easy (Vicon Room Easy) it takes about 5 seconds before there is enough IMU excitation to initialize the MSCKF and then the poses seem fairly accurate. This makes sense to me since the drone is sitting still for the first 4.7 seconds of the dataset. How accurate is the IMU_threshold of 1.5 for the EUROC dataset and in a real system would you have to manually tune this excitation to get accurate poses?

The function to compute the length of the trajectory is wrong:

hi,

It seem that lots of features are featsup_MSCKF, but only few of them will be the good_features_msckf. Most of the features will be rejected by "ct_meas < 3"
When using descriptor and only doing msckf, only about 1/5 features will be used. Most of the "feats_lost" have the length less than 3. is this nomal?

You said set the first measurement to be the anchor frame,but in the FUNCTION single_triangulationfeat->anchor_clone_timestamp = feat->timestamps.at(feat->anchor_cam_id).back(); you set the newest time, Am I not understanding correctly?

besides in the file of FeatureInitializerOptions.h/// **Minimum** distance to accept triangulated features double max_dist = 40;

Many msckf VIO implementations on github don't contain loop closure so they can't be used as a true SLAM implementation. I was wondering if open VINS contained loop closure.