Provide an option in the test verb to return an error code on test failure about colcon-core HOT 12 CLOSED

I guess there is the help, but it would be nice if it was in a tutorial too.

colcon test-result --help
usage: colcon test-result [-h] [--test-result-base TEST_RESULT_BASE] [--all]
                          [--verbose]

Collect the jUnit results generated when testing a set of packages.

optional arguments:
  -h, --help            show this help message and exit
  --test-result-base TEST_RESULT_BASE
                        The base path for all test results (default: build)
  --all                 Show all test result file (even without errors /
                        failures)
  --verbose             Show additional information for errors / failures

from colcon-core.

Now besides

colcon test --packages-select turtle_robot
colcon test-result
echo $?  # Will print 1 if test failed

this also works 😃

colcon test --packages-select turtle_robot --return-code-on-test-failure
echo $?  # Will print 1 if test failed

from colcon-core.

Hi @dirk-thomas, have you ever considered return a non-zero code when colcon build fails? For now the return code is always 0.
I made an automated tools that can build the system and monitor the time, or failure. I scanned the ./log/latest/ folder for output but that's not elegant at all. So adding a return code would be helpful.

from colcon-core.

This seems to be a separate question as this feature request is specific to the test verb.

AFAIK the build verb already returns non-zero in case of build failure. For example, this jobs has a build error: https://ci.ros2.org/job/ci_windows/5173/console
And returns 1:

'env.bat C:\Python37\Scripts\colcon.EXE build --base-paths "src" --build-base "build" --install-base "install" --merge-install --event-handlers console_cohesion+ console_package_list+ --cmake-args -DBUILD_TESTING=ON --no-warn-unused-cli -DINSTALL_EXAMPLES=OFF -DSECURITY=ON -DBUILD_SHARED_LIBS=OFF --packages-up-to rcutils' exited with return code '1'

Can you provide an example of failing build with a zero return code ?

from colcon-core.

Yes. When I deliberately delete a header file in cpp source code of mypkg, building fails. The output is like:

Aborted  <<< rosidl_typesupport_introspection_c
Aborted  <<< rosidl_generator_cpp
Aborted  <<< rcutils

Summary: 82 packages finished [17.0s]
  1 package failed: mypkg
  3 packages aborted: rcutils rosidl_generator_cpp rosidl_typesupport_introspection_c
  1 package had stderr output: mypkg
  110 packages not processed

But the return code I get from $? is still 0.

$ echo $?
0

from colcon-core.

I tried removing a header in rcutils and got a non-zero return code.

--- stderr: rcutils
/tmp/test_colcon/src/ros2/rcutils/src/filesystem.c:19:32: fatal error: rcutils/filesystem.h: No such file or directory
compilation terminated.
make[2]: *** [CMakeFiles/rcutils.dir/src/filesystem.c.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/rcutils.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [all] Error 2
---
Failed   <<< rcutils	[ Exited with code 2 ]

Summary: 0 packages finished [7.01s]
  1 package failed: rcutils
  1 package had stderr output: rcutils

$ echo $?
2

Can you please open a separate issue with a ready to run, reproducible example?

from colcon-core.

from colcon-core.

I have this same question / issue. I'm running colcon test in Travis and it always returns 0, even when the test fails. Is there an easy output that can be grepped? Otherwise a non-zero return code would help.

from colcon-core.

@mkhansen-intel To check if any tests failed you can use colcon test-results after running the tests which will give you a summary of the results and an error return code if any tests failed.

from colcon-core.

To check if any tests failed you can use colcon test-results after running the tests ...

That will only work if the result files have been generated correctly. An option on the test verb working in all the cases is still desired.

from colcon-core.

Yes I didn't suggest that it would resolve this issue, for the full result I would suggest that there's two semantically valuable information, one is all tests successfully ran and generated results, and the 2nd is that all tests passed. The test verb is the only one that can know if the tests successfully ran.

But as a short term workaround the second results is likely going to be easier than trying to automatically grep through the whole. It will not catch every potential error but will capture most errors.

from colcon-core.

OK, that seems to work. I don't see any documentation on that option anywhere though. You might want to add that.

from colcon-core.