chobitsfan / apriltag_plnd Goto Github PK

Hi @chobitsfan,

I'm the current maintainer of RosettaDrone and we are looking forward to contribute to an opensource visual based precision landing project.

I reviewed your implementation and understood everything (code is clean).

Your implementation has this pros:

• Uses Apriltag
• Computes a landing point (offset) independent of the marker centers.
• Sends out the data to compute the yaw

Vision landing's track_targets has this pros:

• gstreamer support...but probably slower then just working with the latest decoded frame
• Uses an input video queue to reduce latency (receives new frames in background while doing the image processing at the same time in other thread)...but I believe the V4L implementation does something similar under the hood (?). It's important to always have the next frame decoded in memory ready to be processed as soon as the previous detection process finishes. Unnecessary frames will be dropped (wasted resources), but the overall latency is optimized.
• outputs AR images/video to debug.
• threshold filter based on historical detections (tells if we can trust the detection). Does the Apriltag library output wrong detections? If so, maybe we can just trust all detections (?) and we don't need to filter again. Anyway, the filters should be applied using the results of multiple markers.
• filter wrong pose estimations, for example, when a marker is detected with negative z value, ie. supposedly being behind the camera.
• documentation (how to calibrate the camera for correct pose estimation)

I believe both projects should be merged somehow so a maintainer community can be built around.
Of course, they are completely different implementations, but the final goal is exactly the same.

Before proposing a project merge strategy, I would like to see the code of the "frontend" where you process the marker data sent via IPC from the backend "capture.c".

You are probably doing there stuff like filtering out detection errors, maybe generating AR images for debugging and other stuff that is done in the equivalent vision landing python script.

About the scope:

We are also interested in doing some extrapolation to:

• Compute the absolute target position to deal with latencies: we need to track the drone position when the image was captured and add the displacement of the drone since this timestamp.
• Correct the absolute target position considering the last computed absolute positions: since the target could be moving, we could do a linear extrapolation based on the last two positions or a b-spline extrapolation considering tree or more points.
• Also a PID could be used to smooth the drone landing motion

Anyway, I believe the merged project should just focus on returning the position of the target relative to the camera, including the error filters required to compute a robust and consistent landing target position based on all markers and maybe also considering previous computations.

The rest I mentioned above could be implemented in the flight controller.

BTW, do you know if this has been implemented?

For RosettaDrone, we will need a independent implementation (apart from the FC), so it would be ideal to use a shared library for this (a second layer separated from the flight controller and from the target tracking/capture layer).

Did you implement video output generation for debugging on your "front end"?
If not, I could start implementing this.
This code should probably go in the backend, since we have the input image there.

Are you using any camera simulator?
If not, I can provide a simple Python based OpenGL simulator and provide instructions.