我不知道如何录制覆盖图片所有位置的点云，是这种原因导致了错误吗？
或许您知道如何不改变雷达位置录制覆盖图片所有位置的点云吗

Lidar: Livox Mid-100
Camera: Flir BlackFly
Rosbag: https://cloud.tsinghua.edu.cn/f/09ceccd8e1c647f59650/?dl=1
Camera_config: https://cloud.tsinghua.edu.cn/f/2687d771b0ea421ea30d/?dl=1

The calibration result seems promising after rough calibration, by collapsed after fine-tune optimization.

Can mid 100 or 40 use this algorithm for automatic calibration?

在void Calibration::colorCloud函数里有以下代码：
for (size_t i = 0; i < pts_2d.size(); i++) {
if (pts_2d[i].x >= 0 && pts_2d[i].x <= image_cols && pts_2d[i].y >= 0 &&
pts_2d[i].y <= image_rows) {
cv::Scalar color = rgb_img.atcv::Vec3b(pts_2d[i]);
if (color[0] == 0 && color[1] == 0 && color[2] == 0) {
continue;
}
其中cv::Scalar color = rgb_img.atcv::Vec3b(pts_2d[i]);是否应改为cv::Scalar color = rgb_img.atcv::Vec3b(pts_2d[i].y,pts_2d[i].x),
opencv 访问数据看起来应该是这种方式

I have been searching for a targetless automatic lidar-camera calibration package and I came across your repository, however the info section on README says that Velodyne support is to be added later. I was wondering why is it not possible to calibrate Velodyne and RealSense using this package. Theoretically, I can just modify the corresponding topic names before calibration and it should be able to treat the pointcloud as a sensor_msgs/PointCloud2 object independently (from the hardware). Why is that not possible? I am willing to try it myself in a few days but wanted to consult you before that.

Thank you for your work!

当我执行roslaunch livox_camera_calib calib_indoor.launch后，终端提示[ INFO] [1625557327.839229229]: Sucessfully load Point Cloud and Image
[ INFO] [1625557327.840642338]: Building Voxel
[ INFO] [1625557347.866017169]: Extracting Lidar Edge
[ INFO] [1625557352.606176671]: Load all data!
[ INFO] [1625557352.678317787]: Finish prepare!
但是等了好久还是没有变化，还是显示initial extrinsic的图片，不知道这是为什么

你好，我用的是速腾128线机械LiDAR，想采用此方案进行和相机的联合标定，这样需要的rosbag大概录制多久的？我理解livox是非重复扫描的可能需要录制的rosbag长一点。而机械式LiDAR是重复扫描的，那我这机械LiDAR 录制10帧是不是就可以了？

Thanks for your impressive work. It works on my setup Livox mid70 + Realsense L515 when use_rough_calib is True with the default initial extrinsic matrix. However, I still have some questions.

1. Most edges match well. But the edges have some offset at the boundary of the field of view. Is it due to the distortion of the camera? I have used the dist_coeffs from the Realsense API. How do you retrieve the dist_coeffs? Maybe the given coeffs from Realsense are not accurate.

2. Is there any benefit to setting use_rough_calib as False with a better initial extrinsic matrix? I have done several trials. When I use the default initial extrinsic matrix, the final calibration just has obvious offsets. when I use a better initial extrinsic matrix, the calibration drifts out of the image in the optimization stage, i.e no points in the image anymore.

Thanks for your help in advance.

What camera is used in this code?

Is there a ros driver for hikrobot MV-CE060-10UC?

Thanks

Thank you for your works, it's a great idea to solve these kind of problems.
I've run your newest code with the your indoor example , but it doesn't work.

I'm pretty sure the config parameters are set to be correct. Could you please to check?

Hello,

I tried with non rectified images and rectified images. Calibration output is much better with non rectified images. Is the code is expecting distorted images? Are you running rectification in the pipeline anywhere?

Regards

Hi, thank you for your contribution to the society.
When I run your code, I met a problem:

[ INFO] [1626940489.756662588]: Loading the rosbag /home/catkin_ws/src/livox_camera_calib/data/merge.bag
[ INFO] [1626940490.902931052]: Sucessfully load Point Cloud and Image
[ INFO] [1626940490.903897494]: Building Voxel
[ INFO] [1626940508.755300261]: Extracting Lidar Edge
[ INFO] [1626940526.485055384]: Load all data!
[ INFO] [1626940526.904265229]: Finish prepare!
[pcl::KdTreeFLANN::setInputCloud] Cannot create a KDTree with an empty input cloud!

Do you know in which function this error happens? So I can check my own data.
Thank you for your time.

include/lidar_camera_calib.hpp line 514
fill_img.at(y, x) = (up_depth + right_depth) / 2;
should be
fill_img.at(y, x) = (up_depth + down_depth) / 2;

在运行multi_calib是出现这种错误，请问是什么原因？是图像边缘特征太少吗？

非常感谢您的分享，有几个问题想请教下您：

1. 下图这个公式可以解释下吗，代表什么含义？
2. 代码里构建的残差公式是依据的下图这个公式，但是比这个公式多乘了一个V.transpose()，可以解释下吗？
3. 方便的话，能否把这篇论文发我看看。地址: [email protected]

再次感谢，期待您的回复

We were very impressed with the results seen running your sample datasets, but once we tried using our own data, the extrinsics calibration failed in 99% of tested scenarios. In every case, your tool miscalculates scene edges and shifts the image projection, in some cases, by as much as 50-60% off original estimates.

For example, here is the projection of an ultra-HiRes image onto a cloud generated with the Livox Avia:

And here is the same cloud / image pair filtered through your calibration tool:

We provided the same exact intrinsics and initial extrinsics that were used to create the first projection, hoping that your tool would properly recognize scene edges and perfect / optimize the calibration. Instead, it identifies edges far away from their origin and attempts to shift the projection to match them. This results in a completely inaccurate calibration.

Here is the planner cloud:

And the residuals seem to show that the lines are properly detected:

Nonetheless, the optimization shifts everything, producing the result I have posted above. If there are any settings you could recommend that might improve our result, please let me know. I presume that the reason your sample datasets work better is that they are based on scenes with very clean, broad lines, such as those found in artificial structures. We even tried indoor scenes with Aruco / checkerboard plates placed in various locations, and your tool does not recognize any of the lines in these plates. It completely ignores the plates and, as with the example above, "hunts" for edges far from the original extrinsic origin.

Perhaps this tool is only applicable in industrial / city-centric settings where large buildings or artificial structures are present. We will attempt to re-test in a "cleaner" environment, but the fact that this tool cannot work with even standard Aruco / checkerboard patterns to properly orient itself makes it mostly unusable for the majority of contexts we would require.

为什么要计算
matrix[1][4] = a1 * x1 + b1 * y1 + c1 * z1;

calc(matrix, point);
calc中的计算原理是什么？

以及后续分别计算6次得到的point各代表的含义是什么？

你好，请教一个问题，标定的参数中，旋转部分比较准确，但平移部分相差了大概10几厘米，请问是场景选的太远了吗？由于平移部分不准，远处的点对齐的比较好，距离5米以内的点效果比较差

demo数据的链接打不开，能否给个百度网盘的链接

Your work is great and inspiring.
I want to put this code together with my SLAM program. Do you think it is feasible?
If possible, I would like to know the specific interface
thank you

Hi,
Because of lower version of PCL(1.17), i prepared a docker image(ros:kinetic-ros-core-xenial), with all dependencies. Also placed the respective files on path.
While running i am getting the following error-

[ INFO] [1629189514.066569998]: Loading the rosbag /opt/catkin_ws/src/livox_camera_calib/data/indoor1.bag
process[rviz-3]: started with pid [381]
process 381: D-Bus library appears to be incorrectly set up; failed to read machine uuid: UUID file '/etc/machine-id' should contain a hex string of length 32, not length 0, with no other text
See the manual page for dbus-uuidgen to correct this issue.
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
Could not initialize OpenGL for RasterGLSurface, reverting to RasterSurface.
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast

[rviz-3] process has died [pid 381, exit code -11, cmd /opt/ros/kinetic/lib/rviz/rviz -d /opt/catkin_ws/src/livox_camera_calib/rviz_cfg/calib.rviz __name:=rviz __log:=/root/.ros/log/81017cba-ff36-11eb-bf90-2c56dc38a7a2/rviz-3.log].
log file: /root/.ros/log/81017cba-ff36-11eb-bf90-2c56dc38a7a2/rviz-3*.log*

[ INFO] [1629189515.516145283]: Sucessfully load Point Cloud and Image
[ INFO] [1629189515.516546482]: Building Voxel
[ INFO] [1629189539.422697927]: Extracting Lidar Edge
[ERROR] [1629189544.520860218]: Can not load image from /opt/catkin_ws/src/livox_camera_calib/data/indoor1.bag
[lidar_camera_calib-2] process has died [pid 374, exit code 255, cmd /opt/catkin_ws/devel/lib/livox_camera_calib/lidar_camera_calib /opt/catkin_ws/src/livox_camera_calib/config/camera.yaml /opt/catkin_ws/src/livox_camera_calib/config/config_outdoor.yaml /opt/catkin_ws/src/livox_camera_calib/data/indoor1.bag /opt/catkin_ws/src/livox_camera_calib/result __name:=lidar_camera_calib __log:=/root/.ros/log/81017cba-ff36-11eb-bf90-2c56dc38a7a2/lidar_camera_calib-2.log].
log file: /root/.ros/log/81017cba-ff36-11eb-bf90-2c56dc38a7a2/lidar_camera_calib-2*.log*

I am not sure, why process has been died and telling indoor1.bag is not found but it has already started with the same.

Any suggestions ?
Thanks.

您好，首先感谢您的工作让Liar-Cam标定更便捷！因最近也在做这方面的工作，所以试了一下您的方法，遇到一个运行问题，如下图：

程序总是在这个位置直接报错，也没有其他提示，可能是什么原因呢？

您好，我正在进行一项研究
请问是否可以分享下 livox+D435setting下点云 对齐到D435双目图像上的一些数据？
万分感谢！
[email protected]
wechat 16600286930
Best！
Xuanyi

maybe the start x y are all 0,not 2

您好，我在运行下载的indoor和outdoor的bag都没有问题，但是运行自己的indoor数据时出现报错，我把相机的内参和image的topic都改了，路径也改过了，出现以下报错：

在rviz中的画面是这样的：

请问这是什么问题呢？
我用的ubuntu18.04 melodic和pcl1.8

您好，请问这个如果要用自己的相机和雷达的数据如何替换呢，录制图片和点云的那个bag包，没有看懂，我刚开始学这个，所以可能问的您觉得有些简单

感谢您的工作。
我准备录取horizon + realsense的数据，我想把他们都录取成一段简短的bag包，然后转换成相应的pcd和图片，因为帧率不同，如何确认他们是同一帧呢。

我下载了源码并成功编译安装, 数据集使用readme中链接提供的;运行室内1和室外1, 都没有得到好的对齐结果;
根据初值, 二者的匹配效果正常; rough calibration阶段把结果拉偏了, 优化的结果更差; 其中室内数据的匹配结果如下:
运行命令为 roslaunch livox_camera_calib calib_indoor.launch

Dear author, thanks for your excellent work, but I am puzzled by the vpnp cost function, I did not understand why we need the line direction to compute the matrix V, and the physical meaning of this formula VRV.transpose

Look forward to your reply

启动程序的时候[lidar_camera_calib-2]进程被杀死，想请问下这是什么原因导致的呢，有什么解决方法嘛？
我的平台是jeston TX2, ubuntu18.04, Ros melodic, PCL 1.8.1

您好，感谢你的出色工作，有一个问题我想请教一下：
从论文中看到你是用点云中提取的直线投影到图像中，然后和图像中提取的直线进行拟合。在代码中看到你是用canny来提取直线的，但是canny计算的是边缘而不是直线，那么为什么不用类似houghline这种方法来提取图像的直线呢?

What are the run commands etc..

First ,thanks for your work!
the sensors mid70+fisheye camera(pinhole+equi),i dont know what the blue screen is and why it is round.

Hi there,

Cool package, hoping to get this to work! I'm using an os1-64. When I launch the calib the Rough Optimization starts and is pretty good. However, when it moves onto the Optimization step, the point cloud is aligned very poorly. It seems when Optimization starts the cloud is rotated by some amount, maybe 90 degrees.

Here are a couple photos. Please note that my camera images do not fit on my monitor so I can only show what I can see:

Rough Optimization step:

Optimization Step:

Any thoughts or ideas would be appreciated. Thank you

How to record the image msg to rosbag

direction_theta_min_ and direction_theta_max_　have no init value.they are all 0.

T xd = xo * distortion + (distorT[2] * xo * yo + distorT[2] * xo * yo) + distorT[3] * (r2 + xo * xo + xo * xo);
      
      
T yd = yo * distortion + distorT[2] * xo * yo + distorT[2] * xo * yo + distorT[2] * (r2 + yo * yo + yo * yo);

should be ?

T xd = xo * distortion + (distorT[2] * xo * yo + distorT[2] * xo * yo) + distorT[3] * (r2 + xo * xo + xo * xo);
      
      
T yd = yo * distortion + distorT[3] * xo * yo + distorT[3] * xo * yo + distorT[2] * (r2 + yo * yo + yo * yo);

I have setup an environment with Ubuntu 18.04 and ROS Melodic. All dependencies have been installed, however, using your sample ROS bag files, I am not able to get past the Lidar edge extraction phase. Here is the terminal output:

Camera Matrix: 
[1364.45, 0, 958.327;
 0, 1366.46, 535.074;
 0, 0, 1]
Distortion Coeffs: 
[0.0958277;
 -0.198233;
 -0.000147133;
 -0.000430056;
 0]
Init extrinsic: 
[0, -1, 0, 0;
 0, 0, -1, 0;
 1, 0, 0, 0;
 0, 0, 0, 1]
[ INFO] [1617231323.709287053]: Loading the rosbag /home/visionarymind/Documents/scans/livoxcalib/outdoor1.bag
[ INFO] [1617231328.493049027]: Sucessfully load Point Cloud and Image
[ INFO] [1617231328.493564736]: Building Voxel
[ INFO] [1617231357.841487217]: Extracting Lidar Edge
[lidar_camera_calib-2] process has died [pid 2586, exit code -11, cmd /home/visionarymind/vision_ws/devel/lib/livox_camera_calib/lidar_camera_calib /home/visionarymind/vision_ws/src/livox_camera_calib/config/camera.yaml /home/visionarymind/vision_ws/src/livox_camera_calib/config/config_outdoor.yaml /home/visionarymind/Documents/scans/livoxcalib/outdoor1.bag /home/visionarymind/vision_ws/src/livox_camera_calib/result __name:=lidar_camera_calib __log:=/home/visionarymind/.ros/log/2c3b8fe2-9274-11eb-9545-000c29df74c2/lidar_camera_calib-2.log].
log file: /home/visionarymind/.ros/log/2c3b8fe2-9274-11eb-9545-000c29df74c2/lidar_camera_calib-2*.log

Would you have any advice on how to correct this problem?

livox好像不可以直接录制pcd,如果从rosbag里读取每一帧的pcd则点云密度很低，不知道您是怎么处理的，是使用CloudCompare类的软件merge之后保存的吗？

English Version
Hello
I have some question about the calibration problem.
I set my vlp-16 and D345i camera as your configuration and I calibrate the extrinsic parameter. However, the value of extrinsic parameter is much different. I think that the problem is the camera intrinsic parameter. I want to ask you how to calibrate your camera intrinsic parameter (I use Matlab)? Or, maybe I deploy myself lab environment is not good?
PS: The attached image which show the color between the pointcloud and image are not matched.

Thank you, bless you have the good day.

中文版本
你好
我想請教一下問題
就是我將光達(vlp-16)與相機(realsense)像你那樣擺設
然後開始進行校正,但我校正出來的值差很多
我推測有可能是相機內參問題
因此想請問你相機內參是如何校正的呢？
目前我是使用matlab
亦或是我的實驗環境擺設有問題呢？

PS:底下可以很明顯看到我的點雲上image的顏色沒對齊

感謝您, 祝您有個美好的一天

English version

Thank you for the open source. This work is very perfect.
However, I have the some question about the code.
First, what's the vpnp. I don't know the Physical mean. Could you provide me reference?
Second, why's the camera frame map into the pixel frame(image frame), you use the (u0-cx)/ fx. According to my knowledge, I would use (u0)/fx - cx.
Thank you very much.

中文版
感謝您提供的開源程式碼,這個程式碼非常的棒
但是我有一些疑問是有關於code的部份

1. 請問什麼是vpnp,我不太清楚其物理意義,請問能麻煩您提供我關於這個參考資料嘛？
2. 在lidar_camera_calib.cpp裡面中,您透過相機內參投影到pixel frame,您使用的是(u0-cx)/ fx,您是先減掉像素中心在除以焦距長. 但照我知識中,應該是(u0)/fx - cx,先除以焦距長,在減掉像素的中心,請問我哪裡搞錯了嘛？

我在自己的数据中运行程序出现节点错误，但并没有报出具体的错误。

所以我在不同地方添加输出消息，定位到出错的位置（lidar_camera_calib/loadCalibConfig）。

运行消息显示，在fSettings初始化时就出现了错误。我们之前通过相类似的读取方式，是可以读取标定文件的。我也在网上查找相关问题，并没找到有效的答案。
不知道您是否能给出建设性的意见？感谢您的回复。

Hi
I have checked your git repository regarding lidar camera calibration. I am a research intern at my Home university NUST Pakistan. I have an os1 64 spinning lidar available at my lab. I need to calibrate the camera and lidar. Now kindly tell me the precise step I need to follow to get that calibration parameter. I don't know how to use Fast LIO for spinning lidar calibration. kindly outline the steps to follow to get extrinsic parameters with your package using os1 64 lidar. I would very thankful to you for this favor.I have bag file recorded with topic pointcloud2 and hikvision camera image. kindly help me in this regard

换边缘信息更丰富的场景也是这样，因为初始偏差过大会选择就近的边缘拟合没法出正确的结果

#19
您好，我遇到了同样的问题，已经把Data.custom_msg: 1改好了，并确认了路径

感觉点云已经加载进去了但还是会报错

您好， 请问你们使用这个方法最后标定出来的外参误差是多少？我在论文里没找到写这部分的内容

Hello,

Thanks for sharing such amazing work. I am able to calibrate livox avia and rgb camera really well. I am trying modify fast lio 2 (https://github.com/hku-mars/FAST_LIO) for rgb point cloud. But the getting good results only in first 2 3 frames. After that mapping is not good. I saw your example video even you are using fast lio for SLAM and then colouring it. Can you please throw some light for the same?

Thanks a lot

Thank you for continuing to develop this project. It is extremely useful. Currently, this project does not support fisheye calibrations with OpenCV, so I rewrote the code to accommodate such a calibration using this camera model. Rough optimization works almost perfectly, generating a very close calibration:

Unfortunately, when the algorithm enters the optimization stage, it gradually shrinks the point cloud before finalizing the calibration. Here is a snapshot:

I wished to share a larger area of the screen, but OpenCV resizing is not working (nor is imwrite functional). If you would like for me to share our dataset, please let me know where to upload it. In the meantime, if you have any advice on how to prevent this "shrinking" from happening, please let me know.

I have seen this behavior before when using distortion coefficients and was able to clear it up by setting them to [0,0,0,0]. In this case, they are required to properly conform the points to the fisheye image. I have also tried undistorting the images, and the same shrinking behavior is seen.