ros2 / system_tests Goto Github PK

License: Apache License 2.0

CMake 14.09% C++ 75.54% Python 10.37%

system_tests's Introduction

System Tests

New system tests may be added when new functionality needs to be tested using the complete ROS 2 core. Before contributing new system tests please discuss the tests you would like to add with the ROS 2 team. If you're implementing new features that may benefit from additional system tests you can ask the reviewers of your pull requests whether system tests are desired. If you would like to add system tests for existing features please open an issue explaining the proposed tests and rationale so it can be reviewed before you start work.

When implementing additional system tests on you could start by using existing tests as a template.

In the test_rclcpp package there is a test of a node with a single service. This test includes a server executable and a client executable. The server executable is a "plain" ROS 2 node implementing a service and the client executable contains GTEST macros testing functionality. These are included in the build from this stanza of CMakeLists.txt and added to this macro of tests to be run for each rmw implementation.

system_tests's People

Contributors

Stargazers

Watchers

system_tests's Issues

OpenSplice with Connext tests fail on OS X

This results in 24 failing tests. It seems to be related to discovery so it affects all of the pairs (connext publishing with opensplice subscribing and vise versa). I can also reproduce this locally by running this:

% ./install/bin/listener__rmw_connext_cpp

And then this in a separate terminal:

% OSPL_VERBOSITY=2 OSPL_INFOFILE="<stdout>" OSPL_ERRORFILE="<stderr>" ./install/bin/talker__rmw_opensplice_cpp
========================================================================================
Report      : WARNING
Date        : Tue Jun 09 17:56:42 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters

Node        : farl
Process     : talker__rmw_opensplice_cpp <17916>
Thread      : dq.builtins 10fa2d000
Internals   : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897802.113387000
========================================================================================
Report      : WARNING
Date        : Tue Jun 09 17:56:42 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters

Node        : farl
Process     : talker__rmw_opensplice_cpp <17916>
Thread      : dq.builtins 10fa2d000
Internals   : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897802.113441000
========================================================================================
Report      : WARNING
Date        : Tue Jun 09 17:56:43 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters

Node        : farl
Process     : talker__rmw_opensplice_cpp <17916>
Thread      : dq.builtins 10fa2d000
Internals   : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897803.117932000
========================================================================================
Report      : WARNING
Date        : Tue Jun 09 17:56:43 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters

Node        : farl
Process     : talker__rmw_opensplice_cpp <17916>
Thread      : dq.builtins 10fa2d000
Internals   : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897803.118007000
Publishing: 'Hello World: 1'
Publishing: 'Hello World: 2'
========================================================================================
Report      : WARNING
Date        : Tue Jun 09 17:56:44 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters

Node        : farl
Process     : talker__rmw_opensplice_cpp <17916>
Thread      : dq.builtins 10fa2d000
Internals   : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897804.121762000
========================================================================================
Report      : WARNING
Date        : Tue Jun 09 17:56:44 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters

Node        : farl
Process     : talker__rmw_opensplice_cpp <17916>
Thread      : dq.builtins 10fa2d000
Internals   : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897804.121831000
Publishing: 'Hello World: 3'
Publishing: 'Hello World: 4'
========================================================================================
Report      : WARNING
Date        : Tue Jun 09 17:56:45 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters

Node        : farl
Process     : talker__rmw_opensplice_cpp <17916>
Thread      : dq.builtins 10fa2d000
Internals   : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897805.126194000
========================================================================================
Report      : WARNING
Date        : Tue Jun 09 17:56:45 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters

Node        : farl
Process     : talker__rmw_opensplice_cpp <17916>
Thread      : dq.builtins 10fa2d000
Internals   : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897805.126253000
Publishing: 'Hello World: 5'
Publishing: 'Hello World: 6'
========================================================================================
Report      : WARNING
Date        : Tue Jun 09 17:56:46 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters

Node        : farl
Process     : talker__rmw_opensplice_cpp <17916>
Thread      : dq.builtins 10fa2d000
Internals   : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897806.127074000
========================================================================================
Report      : WARNING
Date        : Tue Jun 09 17:56:46 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters

Node        : farl
Process     : talker__rmw_opensplice_cpp <17916>
Thread      : dq.builtins 10fa2d000
Internals   : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897806.127108000
Publishing: 'Hello World: 7'
Publishing: 'Hello World: 8'
========================================================================================
Report      : WARNING
Date        : Tue Jun 09 17:56:47 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters

Node        : farl
Process     : talker__rmw_opensplice_cpp <17916>
Thread      : dq.builtins 10fa2d000
Internals   : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897807.129180000
========================================================================================
Report      : WARNING
Date        : Tue Jun 09 17:56:47 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters

Node        : farl
Process     : talker__rmw_opensplice_cpp <17916>
Thread      : dq.builtins 10fa2d000
Internals   : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897807.129228000
Publishing: 'Hello World: 9'
Publishing: 'Hello World: 10'
========================================================================================
Report      : WARNING
Date        : Tue Jun 09 17:56:48 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters

Node        : farl
Process     : talker__rmw_opensplice_cpp <17916>
Thread      : dq.builtins 10fa2d000
Internals   : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897808.132293000
========================================================================================
Report      : WARNING
Date        : Tue Jun 09 17:56:48 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters

Node        : farl
Process     : talker__rmw_opensplice_cpp <17916>
Thread      : dq.builtins 10fa2d000
Internals   : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897808.132369000
Publishing: 'Hello World: 11'
Publishing: 'Hello World: 12'
Publishing: 'Hello World: 13'
Publishing: 'Hello World: 14'
Publishing: 'Hello World: 15'
Publishing: 'Hello World: 16'
Publishing: 'Hello World: 17'
Publishing: 'Hello World: 18'
Publishing: 'Hello World: 19'
Publishing: 'Hello World: 20'
^Csignal_handler(2)

Eventually the warnings settle out, but they never exchange messages. This seems new to me, at one point communication between the two was working on OS X.

cppcheck timing out on test_rclcpp

Bug report

Required Info:

Operating System:
- Ubuntu 20.04 ARM64, macOS AMD64
Installation type:
- source
Version or commit hash:
- master
DDS implementation:
- n/a
Client library (if applicable):
- n/a

Steps to reproduce issue

Run test_rclcpp tests on ARM64 or macOS and you have a chance of the test timing out rather than completing. Here's an example build where I reached into ament_cppcheck and bumped the timeout up to 240. The test completed in 131s.

As seen on CI

Expected behavior

cppcheck runs to completion

Actual behavior

cppcheck run times out before completion generating no test result output.

Additional information

Test fails with extra request

Example:

http://ci.ros2.org/view/nightly/job/nightly_linux_debug/105/testReport/(root)/test_requester_replier_cpp__Primitives__rmw_opensplice_cpp_Debug/test_requester_replier/

I can reproduce this problem locally. test_replier receives the 2 expected requests, which look like this:

$5 = (test_communication::srv::Primitives_Request_<std::allocator<void> > &) @0x6949d0: {bool_value = false, byte_value = 0 '\000', 
  char_value = 0 '\000', float32_value = 0, float64_value = 0, int8_value = 0 '\000', uint8_value = 0 '\000', int16_value = 0, 
  uint16_value = 0, int32_value = 0, uint32_value = 0, int64_value = 0, uint64_value = 0, string_value = "request"}

$6 = (test_communication::srv::Primitives_Request_<std::allocator<void> > &) @0x6949d0: {bool_value = false, byte_value = 0 '\000', 
  char_value = 0 '\000', float32_value = 0, float64_value = 0, int8_value = 0 '\000', uint8_value = 0 '\000', int16_value = 0, 
  uint16_value = 0, int32_value = 0, uint32_value = 0, int64_value = 0, uint64_value = 0, string_value = "request"}

and then receives a third, unexpected one, which looks bogus:

$8 = (test_communication::srv::Primitives_Request_<std::allocator<void> > &) @0x68eb70: {bool_value = false, byte_value = 214 '\326', 
  char_value = -1 '\377', float32_value = 4.59163468e-41, float64_value = 6.953355807310109e-310, int8_value = 112 'p', 
  uint8_value = 214 '\326', int16_value = -1, uint16_value = 32767, int32_value = -10608, uint32_value = 32767, 
  int64_value = 140737488344768, uint64_value = 6969568, string_value = ""}

Failing rclpy-to-rclcpp WString pub/sub over Connext on Windows

Bug report

Required Info:

Operating System:
- Windows 10
Installation type:
- From source
Version or commit hash:
- Current master at f65f852
DDS implementation:
- RTI Connext
Client library (if applicable):
- rclcpp & rclpy

Steps to reproduce issue

Run rclpy-to-rclcpp pub/sub tests using Connext on Windows:

colcon test --event-handlers=console_direct+ --packages-select test_communication --ctest-args " -R" " ^test_publisher_subscriber__rclpy__rclcpp__rmw_connext_cpp"

Expected behavior

All tests pass.

Actual behavior

Subscription to a WString publisher fails with:

18: [test_subscriber-24] r
18: INFO:test_subscriber-24-stderr:[test_subscriber-24] r
18: [test_subscriber-24] eceived message does not match any expected message
18: INFO:test_subscriber-24-stderr:[test_subscriber-24] eceived message does not match any expected message
18: INFO:test_publisher-23-stdout:[test_publisher-23]
18: [ERROR] [test_subscriber-24]: process has died [pid 3156, exit code 3221226505, cmd 'C:/Users/Administrator/nuclearsandwich_ws/build/test_communication/Release/test_subscriber_cpp.exe WStrings /test_time_21_23_20'].
18: ERROR:test_subscriber-24:process has died [pid 3156, exit code 3221226505, cmd 'C:/Users/Administrator/nuclearsandwich_ws/build/test_communication/Release/test_subscriber_cpp.exe WStrings /test_time_21_23_20'].

Additional information

This issue does not manifest itself on rclcpp-to-rclcpp nor rclpy-to-rclpy nor rclcpp-to-rclpy tests.

The process terminates almost silently. Looking at the rclcpp subscriber implementation, this suggests that a crash occurs during rclcpp::shutdown.

[test_security] Secure communication tests are unstable causing CI to fail sometimes

Bug report

Required Info:

Operating System:
- Ubuntu Focal
Installation type:
- source
Version or commit hash:
- master
DDS implementation:
- Fast-RTPS (static and dynamic)
Client library (if applicable):
- rclcpp

Since ros2/sros2#208 the end-to-end tests of sros2 are enabled on the CI of this repo (daily master branch test + PR testing).
While the code is located at https://github.com/ros2/system_tests/tree/master/test_security I'm reporting it here because that's the repo watched by he sros2 maintainer and the test failures are happening on the CI of this repo.

Some tests time out making CI fail, this seems unrelated to the changes being tested, some examples of this:

Daily build: https://github.com/ros2/sros2/runs/710709221?check_suite_focus=true

41 - test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_dynamic_cpp__secure_comm_0 (Timeout)

PR builds:
- https://github.com/ros2/sros2/runs/722938561

19 - test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_1 (Timeout)

https://github.com/ros2/sros2/pull/220/checks?check_run_id=723326846

30 - test_secure_publisher_subscriber__Empty__rmw_fastrtps_dynamic_cpp__secure_comm_0 (Timeout)

https://github.com/ros2/sros2/pull/220/checks?check_run_id=723328796

30 - test_secure_publisher_subscriber__Empty__rmw_fastrtps_dynamic_cpp__secure_comm_0 (Timeout)

Looking at the ROS2 CI from ci.ros2.org over the last week it seems to fail very often but only on linux and only for fastrtps. It goes unnoticed because tests are ran multiple times:

Linux:

May 30, 2020

  33 - test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_3 (Timeout)
  43 - test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_2 (Timeout)

May 29, 2020

  41 - test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_0 (Timeout)

May 28, 2020

  43 - test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_2 (Timeout)

May 27, 2020
✔️
May 26, 2020

  43 - test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_2 (Timeout)

May 24, 2020 (25 may did not run)
✔️

MacOS:

All Connext tests fail (for a different reason) but all Fast-RTPS tests pass ✔️

Windows

All tests pass ✔️

update test for avoid_ros_namespace_conventions QoS option to use no_demangle

Leftover from: ros2/ros2#361

Again, like ros2/rcl#146, I'm not sure how to test this exactly.

But if we can figure out how to test this in a portable way, the test here can probably be removed in favor of the test in ros2/rcl#146.

[test_communication] CMake variable target_suffix doesn't appear to be set anywhere

The cmake code for creating tests in test_communication references a variable target_suffix in several places, but it doesn't appear to be set anywhere.

One place is

system_tests/test_communication/CMakeLists.txt

Line 75 in 6028313

"${target}${target_suffix}__${TEST_MESSAGE_TYPE}"

The earliest reference to target_suffix I found is 03ff4fe, and it doesn't appear to be set in that commit, though then it was a macro instead of a function so it's possible target_suffix was supposed to be set in the scope the macro was called in. @dirk-thomas do you remember what this variable is supposed to do? Can it be deleted?

Add tests for Parameters

Similar to the tests for pub/sub and services.

Tests fail to detect ungraceful ending of action client

Bugreport

Required Info:

Operating System:
- Ubuntu Focal
Installation type:
- source
Version or commit hash:
- master
DDS implementation:
- FastRTPS
Client library (if applicable):
- rclpy

Steps to reproduce issue

colcon test --packages-select test_communication

Expected behavior

If the action server or action client is terminated ungracefully, tests should fail.

Actual behavior

Tests pass despite log output that indicates cleanup failed:
ros2/rclpy#524

17: Test command: /usr/bin/python3 "-u" "/tmp/ws/install_isolated/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/tmp/ws/test_results/test_communication/test_action_client_server__rclpy__rmw_fastrtps_cpp.xunit.xml" "--package-name" "test_communication" "--output-file" "/tmp/ws/build_isolated/test_communication/launch_test/CHANGEME.txt" "--append-env" "LD_LIBRARY_PATH=/tmp/ws/build_isolated/test_communication" "--command" "/usr/bin/python3" "-m" "launch_testing.launch_test" "/tmp/ws/build_isolated/test_communication/test_action_client_server__rclpy__rmw_fastrtps_cpp_.py" "--junit-xml=/tmp/ws/test_results/test_communication/test_action_client_server__rclpy__rmw_fastrtps_cpp.xunit.xml" "--package-name=test_communication"
17: Test timeout computed to be: 60
17: -- run_test.py: extra environment variables to append:
17:  - LD_LIBRARY_PATH+=/tmp/ws/build_isolated/test_communication
17: -- run_test.py: invoking following command in '/tmp/ws/build_isolated/test_communication':
17:  - /usr/bin/python3 -m launch_testing.launch_test /tmp/ws/build_isolated/test_communication/test_action_client_server__rclpy__rmw_fastrtps_cpp_.py --junit-xml=/tmp/ws/test_results/test_communication/test_action_client_server__rclpy__rmw_fastrtps_cpp.xunit.xml --package-name=test_communication
17: 
17: Starting test run test_communication.test_action_client_server__rclpy__rmw_fastrtps_cpp_.launch_tests[Fibonacci]
17: [INFO] [launch]: All log files can be found below /home/buildfarm/.ros/log/2020-03-12-04-03-39-582329-e523ecceac2a-5255
17: [INFO] [launch]: Default logging verbosity is set to INFO
17: test_client_finishes_in_a_finite_amount_of_time[Fibonacci] (test_communication.TestActionClientServer)
17: Test that the second executable terminates after a finite amount of time. ... [INFO] [test_action_server-1]: process started with pid [5258]
17: [INFO] [test_action_client-2]: process started with pid [5260]
17: [test_action_client-2] Sending goal for test number 0
17: [test_action_client-2] Exception ignored in: <function ActionClient.__del__ at 0x7f2115f12700>
17: [test_action_client-2] Traceback (most recent call last):
17: [test_action_client-2]   File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/action/client.py", line 596, in __del__
17: [test_action_client-2]   File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/action/client.py", line 589, in destroy
17: [test_action_client-2]   File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/handle.py", line 124, in __enter__
17: [test_action_client-2]   File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/handle.py", line 106, in _get_capsule
17: [test_action_client-2] rclpy.handle.InvalidHandle: Tried to use a handle that has been destroyed.
17: [INFO] [test_action_client-2]: process has finished cleanly [pid 5260]
17: ok
17: 
17: ----------------------------------------------------------------------
17: Ran 1 test in 1.187s
17: 
17: OK
17: [INFO] [test_action_server-1]: sending signal 'SIGINT' to process[test_action_server-1]
17: [test_action_server-1] Exception ignored in: <function ActionServer.__del__ at 0x7fce84bc3040>
17: [test_action_server-1] Traceback (most recent call last):
17: [test_action_server-1]   File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/action/server.py", line 625, in __del__
17: [test_action_server-1]   File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/action/server.py", line 618, in destroy
17: [test_action_server-1]   File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/handle.py", line 124, in __enter__
17: [test_action_server-1]   File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/handle.py", line 106, in _get_capsule
17: [test_action_server-1] rclpy.handle.InvalidHandle: Tried to use a handle that has been destroyed.
17: [INFO] [test_action_server-1]: process has finished cleanly [pid 5258]
17: test_processes_finished_gracefully[Fibonacci] (test_communication.TestActionClientServerAfterShutdown)
17: Test that both executables finished gracefully. ... ok
17: 
17: ----------------------------------------------------------------------
17: Ran 1 test in 0.000s
17: 
17: OK

Services test is flaky

Starting from a failure in a nightly job (http://ci.ros2.org/view/nightly/job/nightly_osx/147/consoleFull#console-section-170 - search for test_services_cpp__rmw_opensplice_cpp ..........................***Timeout), I can locally reproduce an apparent deadlock in the services test like so:

while true; do nosetests3 -s ./build/test_rclcpp/test_services_cpp__rmw_opensplice_cpp.py; done

Eventually, it'll stop. Then if I attach gdb to each process, the server has this backtrace:

#0  0x0000000000416127 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7ffeb6060268, 
    __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:545
#1  0x0000000000415fc6 in std::__shared_ptr<rclcpp::node::Node, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7ffeb6060260, 
    __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:781
#2  0x0000000000415fe0 in std::shared_ptr<rclcpp::node::Node>::~shared_ptr (this=0x7ffeb6060260, __in_chrg=<optimized out>)
    at /usr/include/c++/4.8/bits/shared_ptr.h:93
#3  0x00007ff01409578a in rclcpp::executor::Executor::get_next_timer(std::shared_ptr<rclcpp::executor::AnyExecutable>) ()
   from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#4  0x00007ff014095c25 in rclcpp::executor::Executor::get_next_ready_executable() ()
   from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#5  0x00007ff014095eda in rclcpp::executor::Executor::get_next_executable(std::chrono::duration<long, std::ratio<1l, 1000000000l> >)
    () from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#6  0x00007ff01409dd5b in rclcpp::executors::single_threaded_executor::SingleThreadedExecutor::spin() ()
   from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#7  0x00007ff01409928c in rclcpp::spin(std::shared_ptr<rclcpp::node::Node>) ()
   from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#8  0x0000000000415d8d in main (argc=1, argv=0x7ffeb60605e8)
    at /home/gerkey/ros2_ws/src/ros2/system_tests/test_rclcpp/test/test_services_server.cpp:50

Not surprising; it's waiting in spin().

The client's backtrace is:

#0  0x0000000000489fce in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::~_Sp_counted_base (this=0xff0d00, 
    __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:114
#1  0x000000000049311f in std::_Sp_counted_ptr<test_rclcpp::srv::AddTwoInts_Response_<std::allocator<void> >*, (__gnu_cxx::_Lock_policy)2>::~_Sp_counted_ptr (this=0xff0d00, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:281
#2  0x000000000049314e in std::_Sp_counted_ptr<test_rclcpp::srv::AddTwoInts_Response_<std::allocator<void> >*, (__gnu_cxx::_Lock_policy)2>::~_Sp_counted_ptr (this=0xff0d00, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:281
#3  0x00000000004931ed in std::_Sp_counted_ptr<test_rclcpp::srv::AddTwoInts_Response_<std::allocator<void> >*, (__gnu_cxx::_Lock_policy)2>::_M_destroy (this=0xff0d00) at /usr/include/c++/4.8/bits/shared_ptr_base.h:294
#4  0x0000000000487520 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0xff0d00)
    at /usr/include/c++/4.8/bits/shared_ptr_base.h:161
#5  0x00000000004860d3 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7ffd77070218, 
    __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:546
#6  0x00007f65a1fc0780 in std::__shared_ptr<void, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() ()
   from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#7  0x00007f65a1fc079a in std::shared_ptr<void>::~shared_ptr() ()
   from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#8  0x00007f65a1fc28f6 in rclcpp::executor::Executor::execute_client(std::shared_ptr<rclcpp::client::ClientBase>) ()
   from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#9  0x00007f65a1fc21c4 in rclcpp::executor::Executor::execute_any_executable(std::shared_ptr<rclcpp::executor::AnyExecutable>) ()
   from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#10 0x00007f65a1fc1d52 in rclcpp::executor::Executor::spin_once(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) ()
   from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#11 0x0000000000489ee0 in rclcpp::executor::Executor::spin_until_future_complete<std::pair<std::shared_ptr<test_rclcpp::srv::AddTwoInts_Request_<std::allocator<void> > >, std::shared_ptr<test_rclcpp::srv::AddTwoInts_Response_<std::allocator<void> > > >, std::ratio<1l, 1000l> > (this=0x7ffd77070400, future=..., timeout=...) at /home/gerkey/ros2_ws/install/include/rclcpp/executor.hpp:167
#12 0x0000000000488d30 in rclcpp::executors::spin_node_until_future_complete<std::pair<std::shared_ptr<test_rclcpp::srv::AddTwoInts_Request_<std::allocator<void> > >, std::shared_ptr<test_rclcpp::srv::AddTwoInts_Response_<std::allocator<void> > > >, std::ratio<1l, 1000l> > (executor=..., node_ptr=..., future=..., timeout=...) at /home/gerkey/ros2_ws/install/include/rclcpp/executors.hpp:67
#13 0x0000000000487440 in rclcpp::spin_until_future_complete<std::pair<std::shared_ptr<test_rclcpp::srv::AddTwoInts_Request_<std::allocator<void> > >, std::shared_ptr<test_rclcpp::srv::AddTwoInts_Response_<std::allocator<void> > > >, std::ratio<1l, 1000l> > (
    node_ptr=..., future=..., timeout=...) at /home/gerkey/ros2_ws/install/include/rclcpp/executors.hpp:81
#14 0x00000000004841d1 in test_services_client_test_return_request_Test::TestBody (this=0xf88e40)
    at /home/gerkey/ros2_ws/src/ros2/system_tests/test_rclcpp/test/test_services_client.cpp:76
#15 0x00000000004aeddc in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0xf88e40, 
    method=&virtual testing::Test::TestBody(), location=0x4bb49b "the test body") at /usr/src/gtest/src/gtest.cc:2090
#16 0x00000000004aa3cc in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0xf88e40, 
    method=&virtual testing::Test::TestBody(), location=0x4bb49b "the test body") at /usr/src/gtest/src/gtest.cc:2126
#17 0x0000000000497e83 in testing::Test::Run (this=0xf88e40) at /usr/src/gtest/src/gtest.cc:2162
#18 0x0000000000498588 in testing::TestInfo::Run (this=0xf88830) at /usr/src/gtest/src/gtest.cc:2338
#19 0x0000000000498ae4 in testing::TestCase::Run (this=0xf885d0) at /usr/src/gtest/src/gtest.cc:2445
#20 0x000000000049d58a in testing::internal::UnitTestImpl::RunAllTests (this=0xf882d0) at /usr/src/gtest/src/gtest.cc:4243
#21 0x00000000004afd1d in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (
    object=0xf882d0, 
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x49d31c <testing::internal::UnitTestImpl::RunAllTests()>, location=0x4bbfd8 "auxiliary test code (environments or event listeners)") at /usr/src/gtest/src/gtest.cc:2090
#22 0x00000000004ab2d6 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (
    object=0xf882d0, 
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x49d31c <testing::internal::UnitTestImpl::RunAllTests()>, location=0x4bbfd8 "auxiliary test code (environments or event listeners)") at /usr/src/gtest/src/gtest.cc:2126
#23 0x000000000049c4f5 in testing::UnitTest::Run (this=0x6dfc00 <testing::UnitTest::GetInstance()::instance>)
    at /usr/src/gtest/src/gtest.cc:3880
#24 0x00000000004842f4 in main (argc=2, argv=0x7ffd77070938)
    at /home/gerkey/ros2_ws/src/ros2/system_tests/test_rclcpp/test/test_services_client.cpp:83

It's waiting in spin_until_future_complete() for the test_return_request test (https://github.com/ros2/system_tests/blob/master/test_rclcpp/test/test_services_client.cpp#L70 - line numbers are different in my backtrace because I'm adding print statements).

From added print statements, I can see that the server gets the callback handle_add_two_ints_noreqid() invoked once with args (1,2) and gets the callback handle_add_two_ints_reqid() invoked twice, both times with args (4,5). So all three service requests are getting to the server. But the third response doesn't make its way back to the client.

Add bounded array messages to communication tests

Move tests closer to the source code

Feature request

Feature description

There are a bunch of tests that can be moved closer to the repository where the source code is.

e.g. all tests in tests_rclcpp can be moved to rclcpp.
See ros2/rclcpp#931 (comment) discussion.

Probably, the sames apply to test_rclpy.

tests for services

Since services are a requirement for parameters I would suggest to implement tests for services before #14.

two clients with the same settings behave differently for fastrtps

In ros2/rmw_fastrtps#48 (test_client_scope_cpp failures), the first client received the response but the second client did not. It can be fixed by making a change to the QoS reliability settings of the second client.

The second client in that demo should have the same behaviour as the first. I have made a regression test that currently fails for fastrtps: 0b024f5

it can be fixed by setting

rmw_qos_profile.reliability = RMW_QOS_POLICY_RELIABILITY_SYSTEM_DEFAULT;

for the second client.

should I open a PR even though the test is failing? or is this ticket sufficient?

intra process test is flaky

http://ci.ros2.org/job/ros2_batch_ci_linux/297/testReport/junit/(root)/test_intra_process_within_one_node/nominal_usage/

It has failed a few times on the CI, but locally I cannot reproduce it. I think we can just add more re-try loops, but first I want to understand why two is not enough.

building test_communication with other RMWs

Bug report

Required Info:

Operating System:
- Ubuntu 18.04
Installation type:
- source
Version or commit hash:
- 68c495 (master)
DDS implementation:
- apparently only FAST-RTPS
Client library (if applicable):
- N/A

Steps to reproduce issue

How does this work with colcon? I tried building with RMW_IMPLEMENTATION (for something other than rmw_fastrtps_cpp but that doesn't seem to affect this line which is how I assume I run test_communication against other RMWs.

More precisely:

$ colcon list
...
rmw_cyclonedds_cpp

$ RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
$ colcon build --symlink-install --packages-up-to test_communication

Expected behavior

$ grep -i cyclone log/latest_build/test_communication/*
(cyclone matches in logs)

Actual behavior

no matches

Additional information

My assumption is that log matches would indicate cyclone tests were built as indicated by the line in the CMakeLists.txt

New nightly test failures in test_communication

There are new test failures in the linux nightly build:

First failures: https://ci.ros2.org/view/nightly/job/nightly_linux_debug/1456/ (Feb 11, 2020)

Failures are:

test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Arrays]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BasicTypes]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BoundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Builtins]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Constants]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Defaults]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Empty]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[MultiNested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Nested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Strings]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[UnboundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[WStrings]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Arrays]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BasicTypes]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BoundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Builtins]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Constants]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Defaults]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Empty]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[MultiNested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Nested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Strings]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[UnboundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[WStrings]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Arrays]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BasicTypes]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BoundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Builtins]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Constants]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Defaults]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Empty]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[MultiNested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Nested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Strings]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[UnboundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[WStrings]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Arrays]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BasicTypes]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BoundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Builtins]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Constants]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Defaults]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Empty]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[MultiNested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Nested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Strings]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[UnboundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[WStrings]

Failure message:

Traceback (most recent call last):
  File "/home/jenkins-agent/workspace/nightly_linux_debug/ws/build/test_communication/test_publisher_subscriber__rclpy__rmw_connext_cpp__rmw_fastrtps_cpp_Debug.py", line 66, in test_subscriber_terminates_in_a_finite_amount_of_time
    proc_info.assertWaitForShutdown(process=subscriber_process, timeout=10)
  File "/home/jenkins-agent/workspace/nightly_linux_debug/ws/install/launch_testing/lib/python3.6/site-packages/launch_testing/proc_info_handler.py", line 144, in assertWaitForShutdown
    assert success, "Timed out waiting for process '{}' to finish".format(process)
AssertionError: Timed out waiting for process '<launch.actions.execute_process.ExecuteProcess object at 0x7f16457296a0>' to finish

[test_security] test access control policies

This is tracking the parts of test_security not implemented in #230
#230 tests authentication and encryption in various scenarios but doesnt test access control.

Tests for access control should:

have a yaml file (same format as the sros2 example) defining all the access control policies used by the tests
use the ros2 security tool to generate keys, certificates, governance and permission files for these tests
test successful communication between nodes respecting the predefined access control policies
test access denial for nodes violating the permissions defined in the yaml policies files

Security "not connecting" tests are flaky

http://ci.ros2.org/job/nightly_osx_repeated/863

projectroot.test_secure_publisher_subscriber__DynamicArrayNested__rmw_connext_cpp__secure_not_connecting_0
projectroot.test_secure_publisher_subscriber__DynamicArrayNested__rmw_connext_cpp__secure_not_connecting_1
projectroot.test_secure_publisher_subscriber__Empty__rmw_connext_cpp__secure_not_connecting_1
projectroot.test_secure_publisher_subscriber__DynamicArrayNested__rmw_fastrtps_cpp__secure_not_connecting_1

They are receiving connections when they're not supposed to. We know that some nodes can take a while to shutdown properly so perhaps it's cross-talk from re-running the tests multiple times. I'll add a namespace to the nodes and see if it fixes it.

Move test_cli_remapping into test_cli

Feature request

Two packages test command line arguments passed to a node: test_cli tests parameters (ex: __params:=/path/to/initialvalues.yaml) while test_cli_remapping tests remap rules (ex: foo:=/bar/foo). The remapping test should be moved into the test_cli package.

Implementation considerations

Bonus points would be to reuse code between the two packages.

Move messages and fixtures from test_communication into another repo

#190 arranges for installation of messages and helpful Python code that's now being used elsewhere (ros2/cli_tools#5). But it would be better to move that stuff into a different repo (#190 (comment)).

possible test-result masking

While working on ros2/rclcpp#236, I ran this build: http://ci.ros2.org/job/ci_osx/1232/testReport/, which shows some fastrtps test failures. Looking at the console log (http://ci.ros2.org/job/ci_osx/1232/consoleFull#console-section-200), among much other output, I noticed this section:

23:52:30 test 13
23:52:30       Start 13: test_parameter_server_cpp__rmw_connext_cpp
23:52:30 
23:52:30 13: Test command: /Users/osrf/jenkins/workspace/ci_osx/venv/bin/python3 "-u" "/Users/osrf/jenkins/workspace/ci_osx/ws/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/Users/osrf/jenkins/workspace/ci_osx/ws/build/test_rclcpp/test_results/test_rclcpp/test_parameter_server_cpp__rmw_connext_cpp.xunit.xml" "--output-file" "/Users/osrf/jenkins/workspace/ci_osx/ws/build/test_rclcpp/ament_cmake_nose/test_parameter_server_cpp__rmw_connext_cpp.txt" "--env" "RCL_ASSERT_RMW_ID_MATCHES=rmw_connext_cpp" "--command" "/Users/osrf/jenkins/workspace/ci_osx/venv/bin/python3" "/Users/osrf/jenkins/workspace/ci_osx/venv/bin/nosetests" "/Users/osrf/jenkins/workspace/ci_osx/ws/build/test_rclcpp/test_parameter_server_cpp__rmw_connext_cpp_.py" "--with-xunit" "--xunit-file=/Users/osrf/jenkins/workspace/ci_osx/ws/build/test_rclcpp/test_results/test_rclcpp/test_parameter_server_cpp__rmw_connext_cpp.xunit.xml" "--xunit-testsuite-name=test_rclcpp.nosetests"
23:52:30 13: Test timeout computed to be: 15
23:52:30 13: -- run_test.py: extra environment variables:
23:52:30 13:  - RCL_ASSERT_RMW_ID_MATCHES=rmw_connext_cpp
23:52:30 13: -- run_test.py: invoking following command in '/Users/osrf/jenkins/workspace/ci_osx/ws/src/ros2/system_tests/test_rclcpp':
23:52:30 13:  - /Users/osrf/jenkins/workspace/ci_osx/venv/bin/python3 /Users/osrf/jenkins/workspace/ci_osx/venv/bin/nosetests /Users/osrf/jenkins/workspace/ci_osx/ws/build/test_rclcpp/test_parameter_server_cpp__rmw_connext_cpp_.py --with-xunit --xunit-file=/Users/osrf/jenkins/workspace/ci_osx/ws/build/test_rclcpp/test_results/test_rclcpp/test_parameter_server_cpp__rmw_connext_cpp.xunit.xml --xunit-testsuite-name=test_rclcpp.nosetests
23:52:45 13/35 Test #13: test_parameter_server_cpp__rmw_connext_cpp .................***Timeout  15.09 sec

That sure looks to me like the connext variant of the parameter_server test failed. But there's no report of any connext failures in Jenkins. I wonder whether the fastrtps and connext test variants are masking each other in the report in one direction or the other. In this case, the fastrtps variant of the parameter_server test also timed out and it was reported as a failure.

Benchmark for different ROS2 clients

Feature request

Feature description

We are developing rclnodejs, a Node.js client for ROS2. We want to add some Performance and Power (PnP) tests for the current rclcpp, rclpy and rclnodejs (may be more) clients, some cases have been designed here. But I don't think it is proper to put these codes in the repo of rclnodejs, does the community have any plan to hold the benchmark test on a specific repo? any suggestions? Thanks!

Intraprocess multithreaded test is flaky

(Not sure whether this is the same problem as #39.)

Below is the console output from a failure seen in a nightly build on OSX, running the multithreaded test with connext.

Looks like this is the sequence of events (https://github.com/ros2/system_tests/blob/master/test_rclcpp/test/test_multithreaded.cpp#L78-L110):

Publish one message, spin, and expect it to have been received by all subscribers. Instead, nobody has yet received it (failure on line 84).
Spin more and expect that all subscribers will still have received only that first message. Instead, by this time, 1 subscriber has received it (failure on line 88).
Reset the counter, publish 5 more messages, and expect that each subscriber will have received exactly 5 messages. Instead, both those 5 new messages and the 3 pending message from earlier are received (failure on line 110).

I'm not sure of the correct fix here. Increase the heuristic delay at the beginning (https://github.com/ros2/system_tests/blob/master/test_rclcpp/test/test_multithreaded.cpp#L72)? Spin more before checking the initial delivery (https://github.com/ros2/system_tests/blob/master/test_rclcpp/test/test_multithreaded.cpp#L83)? Something else?

[ RUN      ] test_multithreaded__.multi_consumer_intra_process
RTI Data Distribution Service Evaluation License issued to OSRF (OSRF01) [email protected] For non-production use only.
Expires on 08-jun-2016 See www.rti.com for more information.
/Users/osrf/jenkins/workspace/nightly_osx/ws/src/ros2/system_tests/test_rclcpp/test/test_multithreaded.cpp:84: Failure
Value of: subscriptions.size()
  Actual: 4
Expected: counter.load()
Which is: 0
callback()    1 with message data 1
/Users/osrf/jenkins/workspace/nightly_osx/ws/src/ros2/system_tests/test_rclcpp/test/test_multithreaded.cpp:88: Failure
Value of: subscriptions.size()
  Actual: 4
Expected: counter.load()
Which is: 1
callback()    1 with message data 1
callback()    2 with message data 1
callback()    3 with message data 1
callback()    4 with message data 1
callback()    5 with message data 1
callback()    6 with message data 1
callback()    7 with message data 1
callback()    8 with message data 2
callback()    9 with message data 2
callback()   10 with message data 2
callback()   11 with message data 2
callback()   12 with message data 3
callback()   13 with message data 3
callback()   14 with message data 3
callback()   15 with message data 3
callback()   16 with message data 4
callback()   17 with message data 4
callback()   18 with message data 4
callback()   19 with message data 4
callback()   20 with message data 5
callback()   21 with message data 5
callback()   22 with message data 5
callback()   23 with message data 5
/Users/osrf/jenkins/workspace/nightly_osx/ws/src/ros2/system_tests/test_rclcpp/test/test_multithreaded.cpp:110: Failure
Value of: 5 * subscriptions.size()
  Actual: 20
Expected: counter.load()
Which is: 23

FastRTPS/Connext Action tests are suspiciously short, and probably don't work

Bug report

Required Info:

Operating System:
- Ubuntu Bionic
Installation type:
- source
Version or commit hash:
- https://gist.github.com/sloretz/6e5d7324b115da574d79c7e785d8968e
DDS implementation:
- Fast-RTPS + RTI Connext
Client library (if applicable):
- clcpp

Steps to reproduce issue

Run the action test using rclcpp with fastrtps and connext

colcon test --event-handlers console_direct+ --packages-select test_communication --ctest-args -R test_action_client_server__Fibonacci__rclcpp__rmw_fastrtps_cpp__rmw_connext_cpp

Run the executables the above test should be running on their own

RMW_IMPLEMENTATION=rmw_fastrtps_cpp ./build/test_communication/test_action_client_cpp Fibonacci /foo/bar

RMW_IMPLEMENTATION=rmw_connext_cpp ./build/test_communication/test_action_server_cpp Fibonacci /foo/bar

Expected behavior

The test would pass, taking about a second or so, and outputting info about goals and feedback
Run independently the tests should output information about goals and feedback

Actual behavior

The test passes in 0.1 seconds with no output
Run independently the client says it sent a goal, but the server does not print a message saying it received it.

Additional information

Same behavior if the rmw implementations are flipped so it's a connext client with fastrtps server. If both are fastrtps or both are connext it works as expected. The service/client and subscriber/publisher tests appear to work fine across fastrtps and connext.

authentication plugin tests

Feature request

Add tests for `node_creation` success

test_security includes tests that target the authentication plugin. I see that it's making the node_creation fail when security env. variables are not set or are not set correctly. Perhaps we should also add the successful cases where a node is authentication and able to be created.

Enable cross-vendor tests for unknown RMW implementations

Feature request

Feature description

Some private or yet-to-be released RMW implementations may intend to interoperate with other RMW implementations. For example, DDS based implementations probably want to communicate with the other DDS implemenations. This is a ticket to enable them to use tests in this repo to determine whether that actually works.

See discussion on #428

Implementation considerations

One option is to by default create tests for all RMW implementations and blacklist the ones which are known not to work. This enables accessing the cross-vendor tests by building this repo in a workspace with all RMW implementations they want to test compatibility with. It has a drawback in that this repo contains more than cross-vendor tests, so someone wanting to test an rmw implementation that is not expected to communicate with other vendors, such as iceoryx, would build and run many tests that are expected to fail.

Another option might be to create a package containing tools for generating a cross-vendor tests. This separates the test suite from the build of an individual package. test_communication could generate the cross-vendor test suite for the RMW implementations we use, and other RMW implementers could use the same tools to generate tests for the combinations they expect to work.

take service server request fails but test succeeds

I noticed these anomalous results: https://ci.ros2.org/job/ci_windows/10042/consoleText
The weird part is [test_replier-1] [ERROR] [1586854796.352196400] [rclcpp]: executor taking a service server request from service '/test_time_08_59_55/test/service/Arrays' unexpectedly failed: error not set

It seems like maybe the test should be reporting failure, not success.

19: Starting test run test_communication.test_requester_replier__rclpy__rclcpp__rmw_connext_cpp_Release.launch_tests[Arrays]
19: [INFO] [launch]: All log files can be found below C:\Users\ContainerAdministrator\.ros/log\2020-04-14-08-59-55-155571-ee5129ec6efc-12504
19: [INFO] [launch]: Default logging verbosity is set to INFO
19: test_requester_finishes_in_a_finite_amount_of_time[Arrays] (test_communication.TestRequesterReplier)
19: Test that the second executable terminates after a finite amount of time. ... [INFO] [test_replier-1]: process started with pid [9268]
19: [INFO] [test_requester-2]: process started with pid [14304]
19: [test_replier-1] [ERROR] [1586854796.352196400] [rclcpp]: executor taking a service server request from service '/test_time_08_59_55/test/service/Arrays' unexpectedly failed: error not set
19: [INFO] [test_requester-2]: process has finished cleanly [pid 14304]
19: [WARNING] [test_replier-1]: 'SIGINT' sent to process[test_replier-1] not supported on Windows, escalating to 'SIGTERM'
19: [INFO] [test_replier-1]: sending signal 'SIGTERM' to process[test_replier-1]
19: [ERROR] [test_replier-1]: process has died [pid 9268, exit code 1, cmd 'C:/ci/ws/build/test_communication/Release/test_replier_cpp.exe Arrays /test_time_08_59_55'].
19: ok

remove disabled tests when wait_for_service support is added to fastrtps

See: ros2/ros2#215 (comment)

Links:

[test_communication] memory leak issues in test_messages_c.cpp

A non-heap pointer is placed on the free list, likely causing a crash later.
In verify_message<test_msgs__msg__StaticArrayPrimitives>(T1 &, unsigned long): Free of an address-of expression, which can never be heap allocated (CWE-590)

Input expected_msg is struct:

ros2/system_tests/test_communication/test/test_messages_c.cpp:496
495 test_msgs__msg__StaticArrayPrimitives expected_msg;
496: get_message(&expected_msg, msg_num);

install/test_msgs/include/test_msgs/msg/static_array_primitives__struct.h:21
21 typedef struct test_msgs__msg__StaticArrayPrimitives

trace functions the msg be freed

build/test_msgs/rosidl_generator_c/test_msgs/msg/static_array_primitives__functions.c:
86 test_msgs__msg__StaticArrayPrimitives__destroy(test_msgs__msg__StaticArrayPrimitives * msg)
87 {
88 if (msg) {
89 test_msgs__msg__StaticArrayPrimitives__fini(msg);
90 }
91 free(msg);
92 }

Incorrect values may be read from freed memory, causing incorrect computations.
In get_message<test_msgs__msg__StaticArrayPrimitives>(T1 *, unsigned long): Reads target of a freed pointer (CWE-416)

ros2/system_tests/test_communication/test/test_messages_c.cpp:447
443 void get_message(test_msgs__msg__StaticArrayPrimitives * msg, size_t msg_num)
444 {
// may freed in test_msgs__msg__StaticArrayPrimitives__destroy when init
445 test_msgs__msg__StaticArrayPrimitives__init(msg)
446 if (msg_num == 0)
447 msg->bool_values[0] = false;

Figure out how to test QoS parameters

There are 2 branches currently in this repository that start adding QoS tests:

These should probably be combined, but then we run into a combinatorial problem; we would test all combinations of RMW implementations x the rclcpp or rclpy x all QoS parameters, which would lead to a very long test. What we should probably do is combine the above, and then choose some subset of the tests that make sense and run those.

[test_security] Fails compilation on second try

Bug report

Required Info:

Operating System:
- Windows 10
Installation type:
- from source
Version or commit hash:
- b9bebaf
DDS implementation:
- N/A
Client library (if applicable):
- N/A

Steps to reproduce issue

Build test_security twice:

colcon build --merge-install --packages-select test_security
colcon build --merge-install --packages-select test_security

Expected behavior

Compiles twice without error.

Actual behavior

The second time, we see the following error:

--- stderr: test_security
CMake Error at CMakeLists.txt:341 (message):
  Failed to generate security artifacts: Traceback (most recent call last):

    File "C:\Users\jacob\dev\ws\ros\foxy\install\Scripts\ros2-script.py", line 11, in <module>
      load_entry_point('ros2cli==0.9.3', 'console_scripts', 'ros2')()
    File "C:\Users\jacob\dev\ws\ros\foxy\install\Lib\site-packages\ros2cli\cli.py", line 67, in main
      rc = extension.main(parser=parser, args=args)
    File "C:\Users\jacob\dev\ws\ros\foxy\install\Lib\site-packages\sros2\command\security.py", line 36, in main
      return extension.main(args=args)
    File "C:\Users\jacob\dev\ws\ros\foxy\install\Lib\site-packages\sros2\verb\generate_artifacts.py", line 47, in main
      success = _artifact_generation.generate_artifacts(
    File "C:\Users\jacob\dev\ws\ros\foxy\install\Lib\site-packages\sros2\api\_artifact_generation.py", line 31, in generate_artifacts
      if not _key.create_key(keystore_path, identity):
    File "C:\Users\jacob\dev\ws\ros\foxy\install\Lib\site-packages\sros2\api\_key.py", line 86, in create_key
      _permission.create_permission_file(permissions_path, _utilities.domain_id(), policy_element)
    File "C:\Users\jacob\dev\ws\ros\foxy\install\Lib\site-packages\sros2\api\_permission.py", line 65, in create_permission_file
      cert_content = _utilities.load_cert(cert_path)
    File "C:\Users\jacob\dev\ws\ros\foxy\install\Lib\site-packages\sros2\api\_utilities.py", line 131, in load_cert
      return x509.load_pem_x509_certificate(
    File "c:\python38\lib\site-packages\cryptography\x509\base.py", line 52, in load_pem_x509_certificate
      return backend.load_pem_x509_certificate(data)
    File "c:\python38\lib\site-packages\cryptography\hazmat\backends\openssl\backend.py", line 1222, in load_pem_x509_certificate
      raise ValueError(

  ValueError: Unable to load certificate.  See
  https://cryptography.io/en/latest/faq/#why-can-t-i-import-my-pem-file for
  more details.

Additional information

If I remove the build directory for test_security, then I can compile again.

Bad WStrings encoding when publishing over Connext on Windows

Bug report

Required Info:

Operating System:
- Windows 10
Installation type:
- From source
Version or commit hash:
- Current master at f65f852
DDS implementation:
- RTI Connext
Client library (if applicable):
- rclcpp & rclpy

Steps to reproduce issue

Run rclcpp-to-rclpy pub/sub tests using Connext on Windows:

colcon test --event-handlers=console_direct+ --packages-select test_communication --ctest-args " -R" " ^test_publisher_subscriber__rclcpp__rclpy__rmw_connext_cpp"

Expected behavior

All tests pass.

Actual behavior

All test pass but it can be clearly seen that the subscriber fails to match incoming WString messages with the following traceback:

21: [test_subscriber-24]     raise RuntimeError('received unexpected message %r' % msg)
21: INFO:test_subscriber-24-stderr:[test_subscriber-24]     raise RuntimeError('received unexpected message %r' % msg)
21: --- Logging error ---
21: Traceback (most recent call last):
21:   File "C:\Python37\lib\logging\__init__.py", line 1037, in emit
21:     stream.write(msg + self.terminator)
21:   File "C:\Python37\lib\encodings\cp1252.py", line 19, in encode
21:     return codecs.charmap_encode(input,self.errors,encoding_table)[0]
21: UnicodeEncodeError: 'charmap' codec can't encode character '\u4e09' in position 136: character maps to <undefined>
21: Call stack:
21:   File "C:\Python37\lib\runpy.py", line 193, in _run_module_as_main
21:     "__main__", mod_spec)
21:   File "C:\Python37\lib\runpy.py", line 85, in _run_code
21:     exec(code, run_globals)
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\launch_test.py", line 165, in <module>
21:     main()
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\launch_test.py", line 142, in main
21:     results = runner.run()
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\test_runner.py", line 231, in run
21:     results[run] = worker.run()
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\test_runner.py", line 143, in run
21:     self._launch_service.run()  # This will block until the test thread stops it
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\launch_service.py", line 350, in run
21:     self.__loop_from_run_thread.run_until_complete(run_loop_task)
21:   File "C:\Python37\lib\asyncio\base_events.py", line 571, in run_until_complete
21:     self.run_forever()
21:   File "C:\Python37\lib\asyncio\base_events.py", line 539, in run_forever
21:     self._run_once()
21:   File "C:\Python37\lib\asyncio\base_events.py", line 1775, in _run_once
21:     handle._run()
21:   File "C:\Python37\lib\asyncio\events.py", line 88, in _run
21:     self._context.run(self._callback, *self._args)
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\launch_service.py", line 177, in _process_one_event
21:     await self.__process_event(next_event)
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\launch_service.py", line 186, in __process_event
21:     entities = event_handler.handle(event, self.__context)
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\event_handlers\on_process_io.py", line 75, in handle
21:     return self.__on_stderr(event)
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\actions\execute_process.py", line 327, in __on_process_stderr
21:     self.__output_format.format(line=line, this=self)
21: Message: "[test_subscriber-24] RuntimeError: received unexpected message test_msgs.msg.WStrings(wstring_value='', array_of_wstrings=['1', 'two', '\u4e09'], bounded_sequence_of_wstrings=['one', '\u4e8c'], unbounded_sequence_of_wstrings=['.', '..', '...', '\u56db'])"
21: Arguments: ()
21: --- Logging error ---
21: Traceback (most recent call last):
21:   File "C:\Python37\lib\logging\__init__.py", line 1037, in emit
21:     stream.write(msg + self.terminator)
21:   File "C:\Python37\lib\encodings\cp1252.py", line 19, in encode
21:     return codecs.charmap_encode(input,self.errors,encoding_table)[0]
21: UnicodeEncodeError: 'charmap' codec can't encode character '\u4e09' in position 155: character maps to <undefined>
21: Call stack:
21:   File "C:\Python37\lib\runpy.py", line 193, in _run_module_as_main
21:     "__main__", mod_spec)
21:   File "C:\Python37\lib\runpy.py", line 85, in _run_code
21:     exec(code, run_globals)
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\launch_test.py", line 165, in <module>
21:     main()
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\launch_test.py", line 142, in main
21:     results = runner.run()
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\test_runner.py", line 231, in run
21:     results[run] = worker.run()
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\test_runner.py", line 143, in run
21:     self._launch_service.run()  # This will block until the test thread stops it
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\launch_service.py", line 350, in run
21:     self.__loop_from_run_thread.run_until_complete(run_loop_task)
21:   File "C:\Python37\lib\asyncio\base_events.py", line 571, in run_until_complete
21:     self.run_forever()
21:   File "C:\Python37\lib\asyncio\base_events.py", line 539, in run_forever
21:     self._run_once()
21:   File "C:\Python37\lib\asyncio\base_events.py", line 1775, in _run_once
21:     handle._run()
21:   File "C:\Python37\lib\asyncio\events.py", line 88, in _run
21:     self._context.run(self._callback, *self._args)
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\launch_service.py", line 177, in _process_one_event
21:     await self.__process_event(next_event)
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\launch_service.py", line 186, in __process_event
21:     entities = event_handler.handle(event, self.__context)
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\event_handlers\on_process_io.py", line 75, in handle
21:     return self.__on_stderr(event)
21:   File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\actions\execute_process.py", line 327, in __on_process_stderr
21:     self.__output_format.format(line=line, this=self)
21: Message: "[test_subscriber-24] RuntimeError: received unexpected message test_msgs.msg.WStrings(wstring_value='', array_of_wstrings=['1', 'two', '\u4e09'], bounded_sequence_of_wstrings=['one', '\u4e8c'], unbounded_sequence_of_wstrings=['.', '..', '...', '\u56db'])"
21: Arguments: ()
21: INFO:test_subscriber-24-stderr:[test_subscriber-24] RuntimeError: received unexpected message test_msgs.msg.WStrings(wstring_value='', array_of_wstrings=['1', 'two', '\u4e09'], bounded_sequence_of_wstrings=['one', '\u4e8c'], unbounded_sequence_of_wstrings=['.', '..', '...', '\u56db'])
21: [ERROR] [test_subscriber-24]: process has died [pid 5580, exit code 1, cmd 'C:\Python37\python.exe C:/Users/Administrator/nuclearsandwich_ws/src/ros2/system_tests/test_communication/test/subscriber_py.py WStrings /test_time_20_56_09'].

Additional information

This issue does not manifest itself on rclcpp-to-rclcpp nor rclpy-to-rclpy tests.

The issue is twofold, as not only an encoding/decoding step is wrong but the test shouldn't be passing. The latter is due to an inappropriately constructed assertion.

Add rclpy action communication tests

Feature request

Feature description

Now that Python Actions are available we can add tests across RMW implementations and between C++ and Python client libraries.

Lines 256 to 259 in e812374

 # TODO(wjwwood): actions do not support python 

 if(client_library1 STREQUAL "rclpy" OR client_library2 STREQUAL "rclpy") 

 set(SKIP_TEST "SKIP_TEST") 

 endif()

system_tests/test_communication/CMakeLists.txt

Lines 318 to 320 in e812374

 set(TEST_PUBLISHER_EXECUTABLE "${CMAKE_CURRENT_SOURCE_DIR}/test/publisher_py.py") 

 set(TEST_REQUESTER_EXECUTABLE "${CMAKE_CURRENT_SOURCE_DIR}/test/requester_py.py") 

 # TODO(wjwwood): actions do not support python

system_tests/test_communication/CMakeLists.txt

Lines 328 to 330 in e812374

 set(TEST_SUBSCRIBER_EXECUTABLE "${CMAKE_CURRENT_SOURCE_DIR}/test/subscriber_py.py") 

 set(TEST_REPLIER_EXECUTABLE "${CMAKE_CURRENT_SOURCE_DIR}/test/replier_py.py") 

 # TODO(wjwwood): actions do not support python

Metaticket ros2/ros2#583

Non-primitive field types for services

Hi @dirk-thomas,

This is Anita from Erle Robotics. Recently, under the supervision of @vmayoral I've been trying to port ROS 1 packages to ROS 2. However, working with services I get errors with non-primitive field types.
I followed these steps:

· Build the package with primitives. (In this case the package builds perfectly)

· Build the package with both primitives and non-primitives. (I get an AttributeError: 'ServiceSpecification' object has no attribute 'base_type' error)

· Again, build the package just with primitives. (I still get errors)

Here is the link of the package: https://github.com/erlerobot/ros2_spider_msgs

I'm following the Migration guide from ROS 1, am I missing something?

Thank you,

Anita

OpenSplice Logging

Currently on OS X several of these tests fail:

http://54.183.26.131:8080/job/ros2_batch_ci_osx/149/testReport/

I've been trying to track down the failures with OpenSplice, but all of the opensplice processes simultaneously dump their logs to the same file:

http://54.183.26.131:8080/job/ros2_batch_ci_osx/ws/workspace/src/ros2/system_tests/test_communication/ospl-info.log

This makes it hard to debug, and it is being generated into the source space, neither of which are desirable. After doing just a little bit of research it looks like we can use OSPL_ERRORFILE and OSPL_INFOFILE to direct the location of these files. The other option is to run each executable in its own folder to prevent them from using the same file. See:

http://forums.opensplice.org/index.php?/topic/1974-log-file-management/?p=2850

The above also hints that we can build a library that gives us even more control over OpenSplice's logging and maybe the long term solution to embedding the OpenSplice logs into the ROS 2 logging system.

Intermittently failing requester/replier test

The test_requester_replier test is flaky on master.

e.g. on a recent test branch, it passed on osx but not Linux:
http://ci.ros2.org/job/ros2_batch_ci_linux/403/testReport/junit/(root)/test_requester_replier_cpp__empty__rmw_connext_dynamic_cpp/test_requester_replier/
http://ci.ros2.org/job/ros2_batch_ci_osx/300/

access control plugin tests

Feature request

Tests for Access Control Plugin

In test_security, tests for access control plugin are missing. I am wondering if there is a plan to add that? I have some tests that target the access control security plugin, but my tests generate policy file at run time, and clean them up before tests are finished.

If you don't already have a plan for access control plugin, could you let me know if you'd prefer to have tests generate policy file at runtime or commit all security-related files? I'd like to contribute.

thanks.

how to run tests

Sorry for basic question, how to run tests?
I'm using beta1 release and ament test ~/ros2_ws/src/ros2/system_tests/test_communication/ outputs many failures.


38: [test_subscriber] exception in subscriber:
38: [test_subscriber] Traceback (most recent call last):
38: [test_subscriber]   File "/home/user/ros2_ws/src/ros2/system_tests/test_communication/test/subscriber_py.py", line 88, in <module>
38: [test_subscriber]     listener(message_name=args.message_name)
38: [test_subscriber]   File "/home/user/ros2_ws/src/ros2/system_tests/test_communication/test/subscriber_py.py", line 69, in listener
38: [test_subscriber]     qos_profile_default)
38: [test_subscriber]   File "/home/user/ros2_ws/install/lib/python3.5/site-packages/rclpy/node.py", line 62, in create_subscription
38: [test_subscriber]     raise NoTypeSupportImportedException
38: [test_subscriber] rclpy.exceptions.NoTypeSupportImportedException: no type_support imported
38: (test_subscriber) rc 1
38: () tear down
38: (test_publisher) signal SIGINT
38: [test_publisher] signal_handler(2)
38: [test_publisher] published for 0.156372 seconds
38: (test_publisher) rc 0







The following tests FAILED:
         13 - test_publisher_subscriber__BoundedArrayNested__rclpy__rmw_fastrtps_cpp (Failed)
         14 - test_publisher_subscriber__BoundedArrayPrimitives__rclpy__rmw_fastrtps_cpp (Failed)
         15 - test_publisher_subscriber__Builtins__rclpy__rmw_fastrtps_cpp (Failed)
         16 - test_publisher_subscriber__DynamicArrayNested__rclpy__rmw_fastrtps_cpp (Failed)
         17 - test_publisher_subscriber__DynamicArrayPrimitives__rclpy__rmw_fastrtps_cpp (Failed)
         18 - test_publisher_subscriber__Empty__rclpy__rmw_fastrtps_cpp (Failed)
         19 - test_publisher_subscriber__Nested__rclpy__rmw_fastrtps_cpp (Failed)
         20 - test_publisher_subscriber__Primitives__rclpy__rmw_fastrtps_cpp (Failed)
         21 - test_publisher_subscriber__StaticArrayNested__rclpy__rmw_fastrtps_cpp (Failed)
         22 - test_publisher_subscriber__StaticArrayPrimitives__rclpy__rmw_fastrtps_cpp (Failed)
         23 - test_requester_replier__Empty__rclpy__rmw_fastrtps_cpp (Failed)
         24 - test_requester_replier__Primitives__rclpy__rmw_fastrtps_cpp (Failed)
         25 - test_publisher_subscriber__BoundedArrayNested__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
         26 - test_publisher_subscriber__BoundedArrayPrimitives__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
         27 - test_publisher_subscriber__Builtins__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
         28 - test_publisher_subscriber__DynamicArrayNested__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
         29 - test_publisher_subscriber__DynamicArrayPrimitives__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
         30 - test_publisher_subscriber__Empty__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
         31 - test_publisher_subscriber__Nested__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
         32 - test_publisher_subscriber__Primitives__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
         33 - test_publisher_subscriber__StaticArrayNested__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
         34 - test_publisher_subscriber__StaticArrayPrimitives__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
         35 - test_requester_replier__Empty__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
         36 - test_requester_replier__Primitives__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
         37 - test_publisher_subscriber__BoundedArrayNested__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
         38 - test_publisher_subscriber__BoundedArrayPrimitives__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
         39 - test_publisher_subscriber__Builtins__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
         40 - test_publisher_subscriber__DynamicArrayNested__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
         41 - test_publisher_subscriber__DynamicArrayPrimitives__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
         42 - test_publisher_subscriber__Empty__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
         43 - test_publisher_subscriber__Nested__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
         44 - test_publisher_subscriber__Primitives__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
         45 - test_publisher_subscriber__StaticArrayNested__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
         46 - test_publisher_subscriber__StaticArrayPrimitives__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
         47 - test_requester_replier__Empty__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
         48 - test_requester_replier__Primitives__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)

test_cli.test_params_yaml flaky (link to real tracking issue)

Failures of this test on CI is a known issue and is being tracked at ros2/build_farmer#166

Users who don't already know about this would think to look in this package, where the test is from.

[test_rclcpp] Add local and async set_parameters_atomically tests

Follow-up of #277.

#277 introduces a minimal set of tests to cover the use of set_parameters_atomically on a SyncParametersClient.
This set of tests should be extended to cover AsyncParametersClient as well as setting local parameters via the Client interface

[test_communication] Refactor test macros to generate cross rmw and cross rcl tests

Refactor the multi_target macro to generate tests for the different configurations:

~~same rcl same rmw (will replace custom_test)~~
same rcl but different rmw (what it currently does but only for rclcpp)
same rmw but different rcl
different rcl and different rmw

To do so, need to fix python char and test template to match current cpp tests

[test_communication] Refactor test_messages_c.cpp to be closer to test_publisher.cpp and test_subscriber.cpp

Feature request

Feature description

test_messages_c.cpp defines its own fixtures and is a monolothic >1000 line file. We should refactor it to use create executables similar to the c++ test_publisher/test_subscriber and move the fixtures to the test_msgs package

timer test may be flaky

Between two builds of OS X this test failed:

http://ci.ros2.org/job/ros2_batch_ci_osx/194/testReport/junit/(root)/test_time__rmw_connext_dynamic_cpp/timer_during_wait/

The previous build passed that one test.

http://ci.ros2.org/job/ros2_batch_ci_osx/193/

There were small changes which could have affected this, but I think they're unrelated since this only affects one implementation and the changes should have affected both static and dynamic connext if they were the root cause. I don't have time to dig into it right now so I created this issue.

test_communication puts stack trace and its failure in different test cases

Bug report

System info:
https://ci.ros2.org/view/nightly/job/nightly_win_deb/1589/

Steps to reproduce issue

run CI https://ci.ros2.org/view/nightly/job/nightly_win_deb/1589/testReport/(root)/projectroot/test_publisher_subscriber__rclpy__rmw_cyclonedds_cpp/

Expected behavior

The crash fails the test case associated with it.

Actual behavior

The useful stack trace appears in the passing test case "test_subscriber_terminates_in_a_finite_amount_of_time[WStrings]"

139: test_subscriber_terminates_in_a_finite_amount_of_time[WStrings] (test_communication.TestPublisherSubscriber)
139: Test that the subscriber terminates after a finite amount of time. ... [INFO] [test_publisher-23]: process started with pid [5128]
139: [INFO] [test_subscriber-24]: process started with pid [1732]
139: [test_subscriber-24] exception in subscriber:
139: [test_subscriber-24] Traceback (most recent call last):
139: [test_subscriber-24]   File "C:/ci/ws/src/ros2/system_tests/test_communication/test/subscriber_py.py", line 81, in <module>
139: [test_subscriber-24]     listener(message_name=args.message_name, namespace=args.namespace)
139: [test_subscriber-24]   File "C:/ci/ws/src/ros2/system_tests/test_communication/test/subscriber_py.py", line 65, in listener
139: [test_subscriber-24]     rclpy.spin_once(node)
139: [test_subscriber-24]   File "C:\ci\ws\install\Lib\site-packages\rclpy\__init__.py", line 171, in spin_once
139: [test_subscriber-24]     executor.spin_once(timeout_sec=timeout_sec)
139: [test_subscriber-24]   File "C:\ci\ws\install\Lib\site-packages\rclpy\executors.py", line 685, in spin_once
139: [test_subscriber-24]     raise handler.exception()
139: [test_subscriber-24]   File "C:\ci\ws\install\Lib\site-packages\rclpy\task.py", line 239, in __call__
139: [test_subscriber-24]     self._handler.send(None)
139: [test_subscriber-24]   File "C:\ci\ws\install\Lib\site-packages\rclpy\executors.py", line 397, in handler
139: [test_subscriber-24]     arg = take_from_wait_list(entity)
139: [test_subscriber-24]   File "C:\ci\ws\install\Lib\site-packages\rclpy\executors.py", line 325, in _take_subscription
139: [test_subscriber-24]     msg = _rclpy.rclpy_take(capsule, sub.msg_type, sub.raw)
139: [test_subscriber-24] UnicodeDecodeError: 'utf-16-le' codec can't decode bytes in position 0-1: illegal encoding
139: [ERROR] [test_subscriber-24]: process has died [pid 1732, exit code 1, cmd 'C:\Python37\python_d.exe C:/ci/ws/src/ros2/system_tests/test_communication/test/subscriber_py.py WStrings /test_time_11_49_47'].
139: [WARNING] [test_publisher-23]: 'SIGINT' sent to process[test_publisher-23] not supported on Windows, escalating to 'SIGTERM'
139: [INFO] [test_publisher-23]: sending signal 'SIGTERM' to process[test_publisher-23]
139: [ERROR] [test_publisher-23]: process has died [pid 5128, exit code 1, cmd 'C:\Python37\python_d.exe C:/ci/ws/src/ros2/system_tests/test_communication/test/publisher_py.py WStrings /test_time_11_49_47'].
139: ok

A useless stack trace appears in the failed test case:

139: test_processes_finished_gracefully[WStrings] (test_communication.TestPublisherSubscriberAfterShutdown)
139: Test that both executables finished gracefully. ... FAIL
139: 
139: ======================================================================
139: FAIL: test_processes_finished_gracefully[WStrings] (test_communication.TestPublisherSubscriberAfterShutdown)
139: Test that both executables finished gracefully.
139: ----------------------------------------------------------------------
139: Traceback (most recent call last):
139:   File "C:\ci\ws\build\test_communication\test_publisher_subscriber__rclpy__rmw_fastrtps_cpp_Debug.py", line 77, in test_processes_finished_gracefully
139:     'test_subscriber'
139:   File "C:\ci\ws\install\Lib\site-packages\launch_testing\asserts\assert_exit_codes.py", line 64, in assertExitCodes
139:     info.returncode
139: AssertionError: Proc test_subscriber-24 exited with code 1

Additional information

Unfortunately, searching these logs for the string "[ERROR]" is made somewhat less useful since even a passing test cases send a SIGTERM to the subprocess, causing it to log an error.

Please add a README explaining how to add new tests

Feature request

Add a README with some explanation of how to add new system tests

Feature description

I would like to be able to add a system test if I make an addition to rclcpp, but not completely sure how to do that. Please add basic instructions.

Implementation considerations

N/A

test_communication tests take a long time and provide unclear feedback on error

Bug report

Required Info:

Operating System:
- All
Installation type:
- All
Version or commit hash:
- 68c495a
DDS implementation:
- All
Client library (if applicable):
- All

Steps to reproduce issue

colcon test --packages-select test_communication

Expected behavior

If failures exist, each test case terminates in a short (<1s) time and reports a relevant failure message (something like "10 messages were sent but 0 were received").

Actual behavior

On failure, the test takes a long time (10s) and the message only reports "timed out waiting for ... to finish". This makes it sound like the receiving process deadlocked. Additionally, the assertion uses a confusing string representation of the launch action object, where the subscriber executable name and arguments would be more appropriate.

https://ci.ros2.org/user/rotu/my-views/view/Extra%20RMW/job/nightly_linux-aarch64_extra_rmw_release/670/testReport/test_communication/TestPublisherSubscriber/test_subscriber_terminates_in_a_finite_amount_of_time_Arrays_/

Traceback (most recent call last):
  File "/home/jenkins-agent/workspace/nightly_linux-aarch64_extra_rmw_release/ws/build/test_communication/test_publisher_subscriber__rclpy__rclcpp__rmw_fastrtps_dynamic_cpp__rmw_cyclonedds_cpp_Release.py", line 66, in test_subscriber_terminates_in_a_finite_amount_of_time
    proc_info.assertWaitForShutdown(process=subscriber_process, timeout=10)
  File "/home/jenkins-agent/workspace/nightly_linux-aarch64_extra_rmw_release/ws/install/launch_testing/lib/python3.6/site-packages/launch_testing/proc_info_handler.py", line 144, in assertWaitForShutdown
    assert success, "Timed out waiting for process '{}' to finish".format(process)
AssertionError: Timed out waiting for process '<launch.actions.execute_process.ExecuteProcess object at 0xffff7e693e80>' to finish

New failing test: test_client_scope_cpp__rmw_connext_cpp

Starting with build: http://ci.ros2.org/view/nightly/job/nightly_linux_repeated/696/ , the test known as test_client_scope_cpp__rmw_connext_cpp started failing. If you search for 'test_client_scope_cpp' in the console output, you'll see that this test started timing out.

test_parameter_server_cpp times out

Bug report

Required Info:

Operating System:
- All platforms
Installation type:
- source
Version or commit hash:
- master
DDS implementation:
- Fast-RTPS (branch: 1.10.x, commit: eProsima/Fast-DDS@09e07c8)
Client library (if applicable):
- rclcpp

Steps to reproduce issue

Here is an example failure:

https://ci.ros2.org/view/nightly/job/nightly_osx_repeated/1918/testReport/junit/(root)/projectroot/test_parameter_server_cpp__rmw_fastrtps_cpp/

Note, the "failed to set parameter" messages are expected output of the test.

I'm able to reproduce locally on Linux (note the test is flakey, but happens more often than not):

colcon test --packages-select test_rclcpp --ctest-args -R "test_parameter_server_cpp__rmw_fastrtps_cpp" --event-handlers console_direct+

Expected behavior

Tests pass.

Actual behavior

Tests fail (timeout)

Additional information

With #419 checked out, we can see that the different tests hang at different points in the execution.

With additional print statements, I've been able to determine that the timeouts occur due to the parameter client waiting forever for a service response (GetParameters.srv, SetParameters.srv, or ListParameters.srv). It looks to me that the service request always makes it to the service server (which succesfully makes the rcl call to send the response), but the response sometimes never reaches the client.

So far, I've only been focused on Fast-RTPS, which is the RMW we see failing on the nightlies.

Also, I've only seen the *_sync tests fail. I have not witnessed a test with an AsyncParametersClient fail to get a response.

random failure with connext and opensplice

Between two Linux builds which were expected to be the same this test failed:

http://ci.ros2.org/job/ros2_batch_ci_linux/274/testReport/junit/(root)/test_requester_replier_cpp__primitives__rmw_connext_dynamic_cpp__rmw_connext_cpp/test_requester_replier/

The previous build passed: http://ci.ros2.org/job/ros2_batch_ci_linux/273/

Some of the output seems to indicate a similar failure (maybe not a common cause) to this: https://community.rti.com/kb/osx510

	# TODO(wjwwood): actions do not support python
	if(client_library1 STREQUAL "rclpy" OR client_library2 STREQUAL "rclpy")
	set(SKIP_TEST "SKIP_TEST")
	endif()

	set(TEST_PUBLISHER_EXECUTABLE "${CMAKE_CURRENT_SOURCE_DIR}/test/publisher_py.py")
	set(TEST_REQUESTER_EXECUTABLE "${CMAKE_CURRENT_SOURCE_DIR}/test/requester_py.py")
	# TODO(wjwwood): actions do not support python

	set(TEST_SUBSCRIBER_EXECUTABLE "${CMAKE_CURRENT_SOURCE_DIR}/test/subscriber_py.py")
	set(TEST_REPLIER_EXECUTABLE "${CMAKE_CURRENT_SOURCE_DIR}/test/replier_py.py")
	# TODO(wjwwood): actions do not support python

ros2 / system_tests Goto Github PK

system_tests's Introduction

System Tests

system_tests's People

Contributors

Stargazers

Watchers

Forkers

system_tests's Issues

Bug report

Steps to reproduce issue

Expected behavior

Actual behavior

Additional information

Bug report

Steps to reproduce issue

Expected behavior

Actual behavior

Additional information

Bug report

Linux:

MacOS:

Windows

Bugreport

Steps to reproduce issue

Expected behavior

Actual behavior

Feature request

Feature description

Bug report

Steps to reproduce issue

Expected behavior

Actual behavior

Additional information

Feature request

Implementation considerations

Feature request

Feature description

Bug report

Steps to reproduce issue

Expected behavior

Actual behavior

Additional information

Feature request

Add tests for node_creation success

Feature request

Feature description

Implementation considerations

Bug report

Steps to reproduce issue

Expected behavior

Actual behavior

Additional information

Bug report

Steps to reproduce issue

Expected behavior

Actual behavior

Additional information

Feature request

Feature description

Feature request

Tests for Access Control Plugin

Feature request

Feature description

Bug report

Steps to reproduce issue

Expected behavior

Actual behavior

Additional information

Feature request

Feature description

Implementation considerations

Bug report

Steps to reproduce issue

Expected behavior

Actual behavior

Bug report

Steps to reproduce issue

Expected behavior

Actual behavior

Add tests for `node_creation` success