test_communication tests take a long time and provide unclear feedback on error about system_tests HOT 3 OPEN

I fully agree assertion messages can (and should) be made clearer, simply not obscuring subscriber output, where the actual test is. But I don't think:

If failures exist, each test case terminates in a short (<1s) time and reports a relevant failure message (something like "10 messages were sent but 0 were received").

is desirable, let alone achievable (in the short to mid-term). Particularly making tests faster.

Cross-vendor pub sub tests consist of a publisher process sending a fixed sequence of known messages in cycles and a subscriber process asserting on those messages. In that context, launch testing infrastructure plays the same role than ROS 1 rostest, and nothing more. Namely, a process launcher with timeouts and return codes checks, plus test output aggregation (where IMHO launch_testing is particularly bad at).

To assert on the number of sent messages vs. received messages implies there's a synchronization mechanism between launcher, publisher and subscriber processes in place, or otherwise latencies in process creation, process scheduling, DDS participant creation, DDS participant discovery, to name a few, will render the test flakey and unpredictable. A synchronization mechanism that does not currently exist in the framework. We can later discuss whether that should be introduced or not, but it's certainly unlikely to land in the short term.

We could achieve something like what you describe by simplifying the test down to whether a message was received or not, but that would then be a different test.

from system_tests.

1. There is a synchronization mechanism implied; all messages must be received within a 10s window from launch for the test to be considered passing. That window seems unreasonably long and could be made much shorter. Considering all test cases I can see pass in < 2 seconds, it seems that window can be reduced.

2. Yes, it's totally reasonable to have some communication from the listener back to the test. Right now, that communication is an exit status. It could be a file, shared memory, a stream, or any sort of IPC channel.

from system_tests.

There is a synchronization mechanism implied; all messages must be received within a 10s window from launch for the test to be considered passing.

Processes start of execution is synchronized, but that's far from enough. That window is a worst case scenario, though AFAIK it's true that no exhaustive search has been conducted to find a lower bound for it.

That window seems unreasonably long and could be made much shorter. Considering all test cases I can see pass in < 2 seconds, it seems that window can be reduced.

Which tests? With which RMW implementation? On which platform? Under what CPU load?

Yes, it's totally reasonable to have some communication from the listener back to the test.

The listener is the test. So I'd refrain from rolling out an IPC just to get the launcher on this test to do the assertion.

Thinking about this again, we could explore having more timeouts in the listener e.g. timeout to first message arrival, timeout to final message arrival, instead of a single, global one, though I'd not dare to guess how small these can be made to get the test passing for all (RMW, OS) combinations. You're more than welcome to contribute an attempt.

from system_tests.