cyclonedds (0.10.4 - ros2 humble) occasionally gets SIGSEGV about cyclonedds HOT 7 CLOSED

Strange, I would certainly not have expected it to crash there. I can't recall ever having seen it call ddsrt_recvmsg with "bad" addresses, and the memory should also definitely not be freed before that thread shuts down.

Is there a simple way to reproduce this that you know of?

from cyclonedds.

I could not reproduce with a simple code, but my code for an integration test on Gazebo often gets SIGSEGV at the exit.

https://github.com/CMU-cabot/cabot-navigation/blob/main/cabot_navigation2/test/run_test.py

from cyclonedds.

I could not reproduce with a simple code, but my code for an integration test on Gazebo often gets SIGSEGV at the exit.

https://github.com/CMU-cabot/cabot-navigation/blob/main/cabot_navigation2/test/run_test.py

I'll give it a try, with a bit of luck I can run that test locally and reproduce it that way.

from cyclonedds.

Hello @eboasson

I made some scripts to run the integration test. Please read README in the zip file.
I can run it on Ubuntu20.04, 32GB RAM, 12 core machine.
The code will run several docker containers and require a display (maybe works with virtual display)

cyclone-dds-bug-1937.zip

from cyclonedds.

Hi @daisukes, that's really nice! 🙏

I tried yesterday by following the information in the cabot README, but to no avail: gazebo on aarch64 Ubuntu 22.04 doesn't even exist. Even if it had, it would anyway have had to run in qemu and no doubt run out of memory, disk space or CPU cycles. Now I know my old Intel box here at home also won't do.

That means the first real chance comes on Monday.

from cyclonedds.

Hi @daisukes, we (@PatrickM-ZS really) tried it out and can easily reproduce it, so that's great.

What we see is that the Python exit handling ends up closing the RMW implementation too early, so in the main thread of the program. It looks like Cyclone DDS is still active when the exit handlers call librmw_implementation.so, which dlcloses librmw_cyclonedds_cpp.so and results in the unmapping (via munmap) of the memory for that library and the Cyclone library (libddsc.so). At that point, the memory containing the code that the thread that crashed is executing (and some of the data it uses) has been yanked from the address space and it crashes.

I'm pretty much certain this is not a bug in Cyclone or the Cyclone RMW layer but something in the Python (test) scripting that causes the ROS 2 nodes and other entities to not be cleaned up before this exit handlers kick in. I don't know Python well enough to know how you could get into that state, and I also haven't tried to find out yet. Perhaps you have an idea what to look for?

Note that it could be fairly trivially "solved" in the Cyclone RMW layer if can run some code just prior to unmapping the library: dds_delete(DDS_CYCLONEDDS_HANDLE) will immediately delete all entities, stop all threads and deinitialize the Cyclone library. However, I think that goes directly against the philosophy behind rclcpp, rcl and rmw, which seems to be that the application must delete all the entities it created from the leaf entities to the nodes/contexts.

from cyclonedds.

@eboasson

Thank you very much for your support!
I put rclpy's cleanup before exiting the test code, and it looks stable.

from cyclonedds.