ros2 / system_tests Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
We are developing rclnodejs, a Node.js client for ROS2. We want to add some Performance and Power (PnP) tests for the current rclcpp, rclpy and rclnodejs (may be more) clients, some cases have been designed here. But I don't think it is proper to put these codes in the repo of rclnodejs, does the community have any plan to hold the benchmark test on a specific repo? any suggestions? Thanks!
The cmake code for creating tests in test_communication
references a variable target_suffix
in several places, but it doesn't appear to be set anywhere.
One place is
The earliest reference to target_suffix
I found is 03ff4fe, and it doesn't appear to be set in that commit, though then it was a macro instead of a function so it's possible target_suffix
was supposed to be set in the scope the macro was called in. @dirk-thomas do you remember what this variable is supposed to do? Can it be deleted?
Between two Linux builds which were expected to be the same this test failed:
The previous build passed: http://ci.ros2.org/job/ros2_batch_ci_linux/273/
Some of the output seems to indicate a similar failure (maybe not a common cause) to this: https://community.rti.com/kb/osx510
Leftover from: ros2/ros2#361
Again, like ros2/rcl#146, I'm not sure how to test this exactly.
But if we can figure out how to test this in a portable way, the test here can probably be removed in favor of the test in ros2/rcl#146.
The test_requester_replier test is flaky on master.
e.g. on a recent test branch, it passed on osx but not Linux:
http://ci.ros2.org/job/ros2_batch_ci_linux/403/testReport/junit/(root)/test_requester_replier_cpp__empty__rmw_connext_dynamic_cpp/test_requester_replier/
http://ci.ros2.org/job/ros2_batch_ci_osx/300/
ros2/system_tests/test_communication/test/test_messages_c.cpp:496
495 test_msgs__msg__StaticArrayPrimitives expected_msg;
496: get_message(&expected_msg, msg_num);
install/test_msgs/include/test_msgs/msg/static_array_primitives__struct.h:21
21 typedef struct test_msgs__msg__StaticArrayPrimitives
build/test_msgs/rosidl_generator_c/test_msgs/msg/static_array_primitives__functions.c:
86 test_msgs__msg__StaticArrayPrimitives__destroy(test_msgs__msg__StaticArrayPrimitives * msg)
87 {
88 if (msg) {
89 test_msgs__msg__StaticArrayPrimitives__fini(msg);
90 }
91 free(msg);
92 }
ros2/system_tests/test_communication/test/test_messages_c.cpp:447
443 void get_message(test_msgs__msg__StaticArrayPrimitives * msg, size_t msg_num)
444 {
// may freed in test_msgs__msg__StaticArrayPrimitives__destroy when init
445 test_msgs__msg__StaticArrayPrimitives__init(msg)
446 if (msg_num == 0)
447 msg->bool_values[0] = false;
Since services are a requirement for parameters I would suggest to implement tests for services before #14.
This results in 24 failing tests. It seems to be related to discovery so it affects all of the pairs (connext publishing with opensplice subscribing and vise versa). I can also reproduce this locally by running this:
% ./install/bin/listener__rmw_connext_cpp
And then this in a separate terminal:
% OSPL_VERBOSITY=2 OSPL_INFOFILE="<stdout>" OSPL_ERRORFILE="<stderr>" ./install/bin/talker__rmw_opensplice_cpp
========================================================================================
Report : WARNING
Date : Tue Jun 09 17:56:42 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters
Node : farl
Process : talker__rmw_opensplice_cpp <17916>
Thread : dq.builtins 10fa2d000
Internals : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897802.113387000
========================================================================================
Report : WARNING
Date : Tue Jun 09 17:56:42 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters
Node : farl
Process : talker__rmw_opensplice_cpp <17916>
Thread : dq.builtins 10fa2d000
Internals : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897802.113441000
========================================================================================
Report : WARNING
Date : Tue Jun 09 17:56:43 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters
Node : farl
Process : talker__rmw_opensplice_cpp <17916>
Thread : dq.builtins 10fa2d000
Internals : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897803.117932000
========================================================================================
Report : WARNING
Date : Tue Jun 09 17:56:43 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters
Node : farl
Process : talker__rmw_opensplice_cpp <17916>
Thread : dq.builtins 10fa2d000
Internals : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897803.118007000
Publishing: 'Hello World: 1'
Publishing: 'Hello World: 2'
========================================================================================
Report : WARNING
Date : Tue Jun 09 17:56:44 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters
Node : farl
Process : talker__rmw_opensplice_cpp <17916>
Thread : dq.builtins 10fa2d000
Internals : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897804.121762000
========================================================================================
Report : WARNING
Date : Tue Jun 09 17:56:44 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters
Node : farl
Process : talker__rmw_opensplice_cpp <17916>
Thread : dq.builtins 10fa2d000
Internals : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897804.121831000
Publishing: 'Hello World: 3'
Publishing: 'Hello World: 4'
========================================================================================
Report : WARNING
Date : Tue Jun 09 17:56:45 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters
Node : farl
Process : talker__rmw_opensplice_cpp <17916>
Thread : dq.builtins 10fa2d000
Internals : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897805.126194000
========================================================================================
Report : WARNING
Date : Tue Jun 09 17:56:45 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters
Node : farl
Process : talker__rmw_opensplice_cpp <17916>
Thread : dq.builtins 10fa2d000
Internals : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897805.126253000
Publishing: 'Hello World: 5'
Publishing: 'Hello World: 6'
========================================================================================
Report : WARNING
Date : Tue Jun 09 17:56:46 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters
Node : farl
Process : talker__rmw_opensplice_cpp <17916>
Thread : dq.builtins 10fa2d000
Internals : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897806.127074000
========================================================================================
Report : WARNING
Date : Tue Jun 09 17:56:46 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters
Node : farl
Process : talker__rmw_opensplice_cpp <17916>
Thread : dq.builtins 10fa2d000
Internals : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897806.127108000
Publishing: 'Hello World: 7'
Publishing: 'Hello World: 8'
========================================================================================
Report : WARNING
Date : Tue Jun 09 17:56:47 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters
Node : farl
Process : talker__rmw_opensplice_cpp <17916>
Thread : dq.builtins 10fa2d000
Internals : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897807.129180000
========================================================================================
Report : WARNING
Date : Tue Jun 09 17:56:47 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters
Node : farl
Process : talker__rmw_opensplice_cpp <17916>
Thread : dq.builtins 10fa2d000
Internals : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897807.129228000
Publishing: 'Hello World: 9'
Publishing: 'Hello World: 10'
========================================================================================
Report : WARNING
Date : Tue Jun 09 17:56:48 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters
Node : farl
Process : talker__rmw_opensplice_cpp <17916>
Thread : dq.builtins 10fa2d000
Internals : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897808.132293000
========================================================================================
Report : WARNING
Date : Tue Jun 09 17:56:48 PDT 2015
Description : SPDP (vendor 1.1): invalid qos/parameters
Node : farl
Process : talker__rmw_opensplice_cpp <17916>
Thread : dq.builtins 10fa2d000
Internals : V6.4.140407OSS///ddsi2/q_ddsi_discovery.c/683/0/1433897808.132369000
Publishing: 'Hello World: 11'
Publishing: 'Hello World: 12'
Publishing: 'Hello World: 13'
Publishing: 'Hello World: 14'
Publishing: 'Hello World: 15'
Publishing: 'Hello World: 16'
Publishing: 'Hello World: 17'
Publishing: 'Hello World: 18'
Publishing: 'Hello World: 19'
Publishing: 'Hello World: 20'
^Csignal_handler(2)
Eventually the warnings settle out, but they never exchange messages. This seems new to me, at one point communication between the two was working on OS X.
I noticed these anomalous results: https://ci.ros2.org/job/ci_windows/10042/consoleText
The weird part is [test_replier-1] [ERROR] [1586854796.352196400] [rclcpp]: executor taking a service server request from service '/test_time_08_59_55/test/service/Arrays' unexpectedly failed: error not set
It seems like maybe the test should be reporting failure, not success.
19: Starting test run test_communication.test_requester_replier__rclpy__rclcpp__rmw_connext_cpp_Release.launch_tests[Arrays]
19: [INFO] [launch]: All log files can be found below C:\Users\ContainerAdministrator\.ros/log\2020-04-14-08-59-55-155571-ee5129ec6efc-12504
19: [INFO] [launch]: Default logging verbosity is set to INFO
19: test_requester_finishes_in_a_finite_amount_of_time[Arrays] (test_communication.TestRequesterReplier)
19: Test that the second executable terminates after a finite amount of time. ... [INFO] [test_replier-1]: process started with pid [9268]
19: [INFO] [test_requester-2]: process started with pid [14304]
19: [test_replier-1] [ERROR] [1586854796.352196400] [rclcpp]: executor taking a service server request from service '/test_time_08_59_55/test/service/Arrays' unexpectedly failed: error not set
19: [INFO] [test_requester-2]: process has finished cleanly [pid 14304]
19: [WARNING] [test_replier-1]: 'SIGINT' sent to process[test_replier-1] not supported on Windows, escalating to 'SIGTERM'
19: [INFO] [test_replier-1]: sending signal 'SIGTERM' to process[test_replier-1]
19: [ERROR] [test_replier-1]: process has died [pid 9268, exit code 1, cmd 'C:/ci/ws/build/test_communication/Release/test_replier_cpp.exe Arrays /test_time_08_59_55'].
19: ok
Similar to the tests for pub/sub and services.
Required Info:
Since ros2/sros2#208 the end-to-end tests of sros2 are enabled on the CI of this repo (daily master branch test + PR testing).
While the code is located at https://github.com/ros2/system_tests/tree/master/test_security I'm reporting it here because that's the repo watched by he sros2 maintainer and the test failures are happening on the CI of this repo.
Some tests time out making CI fail, this seems unrelated to the changes being tested, some examples of this:
41 - test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_dynamic_cpp__secure_comm_0 (Timeout)
19 - test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_1 (Timeout)
30 - test_secure_publisher_subscriber__Empty__rmw_fastrtps_dynamic_cpp__secure_comm_0 (Timeout)
30 - test_secure_publisher_subscriber__Empty__rmw_fastrtps_dynamic_cpp__secure_comm_0 (Timeout)
Looking at the ROS2 CI from ci.ros2.org over the last week it seems to fail very often but only on linux and only for fastrtps. It goes unnoticed because tests are ran multiple times:
33 - test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_3 (Timeout)
43 - test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_2 (Timeout)
41 - test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_0 (Timeout)
43 - test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_2 (Timeout)
43 - test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_2 (Timeout)
Required Info:
Run rclcpp
-to-rclpy
pub/sub tests using Connext on Windows:
colcon test --event-handlers=console_direct+ --packages-select test_communication --ctest-args " -R" " ^test_publisher_subscriber__rclcpp__rclpy__rmw_connext_cpp"
All tests pass.
All test pass but it can be clearly seen that the subscriber fails to match incoming WString messages with the following traceback:
21: [test_subscriber-24] raise RuntimeError('received unexpected message %r' % msg)
21: INFO:test_subscriber-24-stderr:[test_subscriber-24] raise RuntimeError('received unexpected message %r' % msg)
21: --- Logging error ---
21: Traceback (most recent call last):
21: File "C:\Python37\lib\logging\__init__.py", line 1037, in emit
21: stream.write(msg + self.terminator)
21: File "C:\Python37\lib\encodings\cp1252.py", line 19, in encode
21: return codecs.charmap_encode(input,self.errors,encoding_table)[0]
21: UnicodeEncodeError: 'charmap' codec can't encode character '\u4e09' in position 136: character maps to <undefined>
21: Call stack:
21: File "C:\Python37\lib\runpy.py", line 193, in _run_module_as_main
21: "__main__", mod_spec)
21: File "C:\Python37\lib\runpy.py", line 85, in _run_code
21: exec(code, run_globals)
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\launch_test.py", line 165, in <module>
21: main()
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\launch_test.py", line 142, in main
21: results = runner.run()
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\test_runner.py", line 231, in run
21: results[run] = worker.run()
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\test_runner.py", line 143, in run
21: self._launch_service.run() # This will block until the test thread stops it
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\launch_service.py", line 350, in run
21: self.__loop_from_run_thread.run_until_complete(run_loop_task)
21: File "C:\Python37\lib\asyncio\base_events.py", line 571, in run_until_complete
21: self.run_forever()
21: File "C:\Python37\lib\asyncio\base_events.py", line 539, in run_forever
21: self._run_once()
21: File "C:\Python37\lib\asyncio\base_events.py", line 1775, in _run_once
21: handle._run()
21: File "C:\Python37\lib\asyncio\events.py", line 88, in _run
21: self._context.run(self._callback, *self._args)
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\launch_service.py", line 177, in _process_one_event
21: await self.__process_event(next_event)
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\launch_service.py", line 186, in __process_event
21: entities = event_handler.handle(event, self.__context)
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\event_handlers\on_process_io.py", line 75, in handle
21: return self.__on_stderr(event)
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\actions\execute_process.py", line 327, in __on_process_stderr
21: self.__output_format.format(line=line, this=self)
21: Message: "[test_subscriber-24] RuntimeError: received unexpected message test_msgs.msg.WStrings(wstring_value='', array_of_wstrings=['1', 'two', '\u4e09'], bounded_sequence_of_wstrings=['one', '\u4e8c'], unbounded_sequence_of_wstrings=['.', '..', '...', '\u56db'])"
21: Arguments: ()
21: --- Logging error ---
21: Traceback (most recent call last):
21: File "C:\Python37\lib\logging\__init__.py", line 1037, in emit
21: stream.write(msg + self.terminator)
21: File "C:\Python37\lib\encodings\cp1252.py", line 19, in encode
21: return codecs.charmap_encode(input,self.errors,encoding_table)[0]
21: UnicodeEncodeError: 'charmap' codec can't encode character '\u4e09' in position 155: character maps to <undefined>
21: Call stack:
21: File "C:\Python37\lib\runpy.py", line 193, in _run_module_as_main
21: "__main__", mod_spec)
21: File "C:\Python37\lib\runpy.py", line 85, in _run_code
21: exec(code, run_globals)
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\launch_test.py", line 165, in <module>
21: main()
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\launch_test.py", line 142, in main
21: results = runner.run()
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\test_runner.py", line 231, in run
21: results[run] = worker.run()
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch_testing\Lib\site-packages\launch_testing\test_runner.py", line 143, in run
21: self._launch_service.run() # This will block until the test thread stops it
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\launch_service.py", line 350, in run
21: self.__loop_from_run_thread.run_until_complete(run_loop_task)
21: File "C:\Python37\lib\asyncio\base_events.py", line 571, in run_until_complete
21: self.run_forever()
21: File "C:\Python37\lib\asyncio\base_events.py", line 539, in run_forever
21: self._run_once()
21: File "C:\Python37\lib\asyncio\base_events.py", line 1775, in _run_once
21: handle._run()
21: File "C:\Python37\lib\asyncio\events.py", line 88, in _run
21: self._context.run(self._callback, *self._args)
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\launch_service.py", line 177, in _process_one_event
21: await self.__process_event(next_event)
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\launch_service.py", line 186, in __process_event
21: entities = event_handler.handle(event, self.__context)
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\event_handlers\on_process_io.py", line 75, in handle
21: return self.__on_stderr(event)
21: File "C:\Users\Administrator\nuclearsandwich_ws\install\launch\Lib\site-packages\launch\actions\execute_process.py", line 327, in __on_process_stderr
21: self.__output_format.format(line=line, this=self)
21: Message: "[test_subscriber-24] RuntimeError: received unexpected message test_msgs.msg.WStrings(wstring_value='', array_of_wstrings=['1', 'two', '\u4e09'], bounded_sequence_of_wstrings=['one', '\u4e8c'], unbounded_sequence_of_wstrings=['.', '..', '...', '\u56db'])"
21: Arguments: ()
21: INFO:test_subscriber-24-stderr:[test_subscriber-24] RuntimeError: received unexpected message test_msgs.msg.WStrings(wstring_value='', array_of_wstrings=['1', 'two', '\u4e09'], bounded_sequence_of_wstrings=['one', '\u4e8c'], unbounded_sequence_of_wstrings=['.', '..', '...', '\u56db'])
21: [ERROR] [test_subscriber-24]: process has died [pid 5580, exit code 1, cmd 'C:\Python37\python.exe C:/Users/Administrator/nuclearsandwich_ws/src/ros2/system_tests/test_communication/test/subscriber_py.py WStrings /test_time_20_56_09'].
This issue does not manifest itself on rclcpp
-to-rclcpp
nor rclpy
-to-rclpy
tests.
The issue is twofold, as not only an encoding/decoding step is wrong but the test shouldn't be passing. The latter is due to an inappropriately constructed assertion.
Between two builds of OS X this test failed:
The previous build passed that one test.
http://ci.ros2.org/job/ros2_batch_ci_osx/193/
There were small changes which could have affected this, but I think they're unrelated since this only affects one implementation and the changes should have affected both static and dynamic connext if they were the root cause. I don't have time to dig into it right now so I created this issue.
In ros2/rmw_fastrtps#48 (test_client_scope_cpp
failures), the first client received the response but the second client did not. It can be fixed by making a change to the QoS reliability settings of the second client.
The second client in that demo should have the same behaviour as the first. I have made a regression test that currently fails for fastrtps: 0b024f5
it can be fixed by setting
rmw_qos_profile.reliability = RMW_QOS_POLICY_RELIABILITY_SYSTEM_DEFAULT;
for the second client.
should I open a PR even though the test is failing? or is this ticket sufficient?
Starting with build: http://ci.ros2.org/view/nightly/job/nightly_linux_repeated/696/ , the test known as test_client_scope_cpp__rmw_connext_cpp started failing. If you search for 'test_client_scope_cpp' in the console output, you'll see that this test started timing out.
Some private or yet-to-be released RMW implementations may intend to interoperate with other RMW implementations. For example, DDS based implementations probably want to communicate with the other DDS implemenations. This is a ticket to enable them to use tests in this repo to determine whether that actually works.
See discussion on #428
One option is to by default create tests for all RMW implementations and blacklist the ones which are known not to work. This enables accessing the cross-vendor tests by building this repo in a workspace with all RMW implementations they want to test compatibility with. It has a drawback in that this repo contains more than cross-vendor tests, so someone wanting to test an rmw implementation that is not expected to communicate with other vendors, such as iceoryx
, would build and run many tests that are expected to fail.
Another option might be to create a package containing tools for generating a cross-vendor tests. This separates the test suite from the build of an individual package. test_communication
could generate the cross-vendor test suite for the RMW implementations we use, and other RMW implementers could use the same tools to generate tests for the combinations they expect to work.
While working on ros2/rclcpp#236, I ran this build: http://ci.ros2.org/job/ci_osx/1232/testReport/, which shows some fastrtps test failures. Looking at the console log (http://ci.ros2.org/job/ci_osx/1232/consoleFull#console-section-200), among much other output, I noticed this section:
23:52:30 test 13
23:52:30 Start 13: test_parameter_server_cpp__rmw_connext_cpp
23:52:30
23:52:30 13: Test command: /Users/osrf/jenkins/workspace/ci_osx/venv/bin/python3 "-u" "/Users/osrf/jenkins/workspace/ci_osx/ws/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/Users/osrf/jenkins/workspace/ci_osx/ws/build/test_rclcpp/test_results/test_rclcpp/test_parameter_server_cpp__rmw_connext_cpp.xunit.xml" "--output-file" "/Users/osrf/jenkins/workspace/ci_osx/ws/build/test_rclcpp/ament_cmake_nose/test_parameter_server_cpp__rmw_connext_cpp.txt" "--env" "RCL_ASSERT_RMW_ID_MATCHES=rmw_connext_cpp" "--command" "/Users/osrf/jenkins/workspace/ci_osx/venv/bin/python3" "/Users/osrf/jenkins/workspace/ci_osx/venv/bin/nosetests" "/Users/osrf/jenkins/workspace/ci_osx/ws/build/test_rclcpp/test_parameter_server_cpp__rmw_connext_cpp_.py" "--with-xunit" "--xunit-file=/Users/osrf/jenkins/workspace/ci_osx/ws/build/test_rclcpp/test_results/test_rclcpp/test_parameter_server_cpp__rmw_connext_cpp.xunit.xml" "--xunit-testsuite-name=test_rclcpp.nosetests"
23:52:30 13: Test timeout computed to be: 15
23:52:30 13: -- run_test.py: extra environment variables:
23:52:30 13: - RCL_ASSERT_RMW_ID_MATCHES=rmw_connext_cpp
23:52:30 13: -- run_test.py: invoking following command in '/Users/osrf/jenkins/workspace/ci_osx/ws/src/ros2/system_tests/test_rclcpp':
23:52:30 13: - /Users/osrf/jenkins/workspace/ci_osx/venv/bin/python3 /Users/osrf/jenkins/workspace/ci_osx/venv/bin/nosetests /Users/osrf/jenkins/workspace/ci_osx/ws/build/test_rclcpp/test_parameter_server_cpp__rmw_connext_cpp_.py --with-xunit --xunit-file=/Users/osrf/jenkins/workspace/ci_osx/ws/build/test_rclcpp/test_results/test_rclcpp/test_parameter_server_cpp__rmw_connext_cpp.xunit.xml --xunit-testsuite-name=test_rclcpp.nosetests
23:52:45 13/35 Test #13: test_parameter_server_cpp__rmw_connext_cpp .................***Timeout 15.09 sec
That sure looks to me like the connext variant of the parameter_server test failed. But there's no report of any connext failures in Jenkins. I wonder whether the fastrtps and connext test variants are masking each other in the report in one direction or the other. In this case, the fastrtps variant of the parameter_server test also timed out and it was reported as a failure.
There are a bunch of tests that can be moved closer to the repository where the source code is.
e.g. all tests in tests_rclcpp
can be moved to rclcpp
.
See ros2/rclcpp#931 (comment) discussion.
Probably, the sames apply to test_rclpy
.
Example:
I can reproduce this problem locally. test_replier
receives the 2 expected requests, which look like this:
$5 = (test_communication::srv::Primitives_Request_<std::allocator<void> > &) @0x6949d0: {bool_value = false, byte_value = 0 '\000',
char_value = 0 '\000', float32_value = 0, float64_value = 0, int8_value = 0 '\000', uint8_value = 0 '\000', int16_value = 0,
uint16_value = 0, int32_value = 0, uint32_value = 0, int64_value = 0, uint64_value = 0, string_value = "request"}
$6 = (test_communication::srv::Primitives_Request_<std::allocator<void> > &) @0x6949d0: {bool_value = false, byte_value = 0 '\000',
char_value = 0 '\000', float32_value = 0, float64_value = 0, int8_value = 0 '\000', uint8_value = 0 '\000', int16_value = 0,
uint16_value = 0, int32_value = 0, uint32_value = 0, int64_value = 0, uint64_value = 0, string_value = "request"}
and then receives a third, unexpected one, which looks bogus:
$8 = (test_communication::srv::Primitives_Request_<std::allocator<void> > &) @0x68eb70: {bool_value = false, byte_value = 214 '\326',
char_value = -1 '\377', float32_value = 4.59163468e-41, float64_value = 6.953355807310109e-310, int8_value = 112 'p',
uint8_value = 214 '\326', int16_value = -1, uint16_value = 32767, int32_value = -10608, uint32_value = 32767,
int64_value = 140737488344768, uint64_value = 6969568, string_value = ""}
Currently on OS X several of these tests fail:
http://54.183.26.131:8080/job/ros2_batch_ci_osx/149/testReport/
I've been trying to track down the failures with OpenSplice, but all of the opensplice processes simultaneously dump their logs to the same file:
This makes it hard to debug, and it is being generated into the source space, neither of which are desirable. After doing just a little bit of research it looks like we can use OSPL_ERRORFILE
and OSPL_INFOFILE
to direct the location of these files. The other option is to run each executable in its own folder to prevent them from using the same file. See:
http://forums.opensplice.org/index.php?/topic/1974-log-file-management/?p=2850
The above also hints that we can build a library that gives us even more control over OpenSplice's logging and maybe the long term solution to embedding the OpenSplice logs into the ROS 2 logging system.
Add a README with some explanation of how to add new system tests
I would like to be able to add a system test if I make an addition to rclcpp, but not completely sure how to do that. Please add basic instructions.
N/A
Sorry for basic question, how to run tests?
I'm using beta1 release and ament test ~/ros2_ws/src/ros2/system_tests/test_communication/
outputs many failures.
38: [test_subscriber] exception in subscriber:
38: [test_subscriber] Traceback (most recent call last):
38: [test_subscriber] File "/home/user/ros2_ws/src/ros2/system_tests/test_communication/test/subscriber_py.py", line 88, in <module>
38: [test_subscriber] listener(message_name=args.message_name)
38: [test_subscriber] File "/home/user/ros2_ws/src/ros2/system_tests/test_communication/test/subscriber_py.py", line 69, in listener
38: [test_subscriber] qos_profile_default)
38: [test_subscriber] File "/home/user/ros2_ws/install/lib/python3.5/site-packages/rclpy/node.py", line 62, in create_subscription
38: [test_subscriber] raise NoTypeSupportImportedException
38: [test_subscriber] rclpy.exceptions.NoTypeSupportImportedException: no type_support imported
38: (test_subscriber) rc 1
38: () tear down
38: (test_publisher) signal SIGINT
38: [test_publisher] signal_handler(2)
38: [test_publisher] published for 0.156372 seconds
38: (test_publisher) rc 0
The following tests FAILED:
13 - test_publisher_subscriber__BoundedArrayNested__rclpy__rmw_fastrtps_cpp (Failed)
14 - test_publisher_subscriber__BoundedArrayPrimitives__rclpy__rmw_fastrtps_cpp (Failed)
15 - test_publisher_subscriber__Builtins__rclpy__rmw_fastrtps_cpp (Failed)
16 - test_publisher_subscriber__DynamicArrayNested__rclpy__rmw_fastrtps_cpp (Failed)
17 - test_publisher_subscriber__DynamicArrayPrimitives__rclpy__rmw_fastrtps_cpp (Failed)
18 - test_publisher_subscriber__Empty__rclpy__rmw_fastrtps_cpp (Failed)
19 - test_publisher_subscriber__Nested__rclpy__rmw_fastrtps_cpp (Failed)
20 - test_publisher_subscriber__Primitives__rclpy__rmw_fastrtps_cpp (Failed)
21 - test_publisher_subscriber__StaticArrayNested__rclpy__rmw_fastrtps_cpp (Failed)
22 - test_publisher_subscriber__StaticArrayPrimitives__rclpy__rmw_fastrtps_cpp (Failed)
23 - test_requester_replier__Empty__rclpy__rmw_fastrtps_cpp (Failed)
24 - test_requester_replier__Primitives__rclpy__rmw_fastrtps_cpp (Failed)
25 - test_publisher_subscriber__BoundedArrayNested__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
26 - test_publisher_subscriber__BoundedArrayPrimitives__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
27 - test_publisher_subscriber__Builtins__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
28 - test_publisher_subscriber__DynamicArrayNested__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
29 - test_publisher_subscriber__DynamicArrayPrimitives__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
30 - test_publisher_subscriber__Empty__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
31 - test_publisher_subscriber__Nested__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
32 - test_publisher_subscriber__Primitives__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
33 - test_publisher_subscriber__StaticArrayNested__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
34 - test_publisher_subscriber__StaticArrayPrimitives__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
35 - test_requester_replier__Empty__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
36 - test_requester_replier__Primitives__rclpy__rclcpp__rmw_fastrtps_cpp (Failed)
37 - test_publisher_subscriber__BoundedArrayNested__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
38 - test_publisher_subscriber__BoundedArrayPrimitives__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
39 - test_publisher_subscriber__Builtins__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
40 - test_publisher_subscriber__DynamicArrayNested__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
41 - test_publisher_subscriber__DynamicArrayPrimitives__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
42 - test_publisher_subscriber__Empty__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
43 - test_publisher_subscriber__Nested__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
44 - test_publisher_subscriber__Primitives__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
45 - test_publisher_subscriber__StaticArrayNested__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
46 - test_publisher_subscriber__StaticArrayPrimitives__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
47 - test_requester_replier__Empty__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
48 - test_requester_replier__Primitives__rclcpp__rclpy__rmw_fastrtps_cpp (Failed)
Required Info:
How does this work with colcon
? I tried building with RMW_IMPLEMENTATION
(for something other than rmw_fastrtps_cpp
but that doesn't seem to affect this line which is how I assume I run test_communication
against other RMWs.
More precisely:
$ colcon list
...
rmw_cyclonedds_cpp
$ RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
$ colcon build --symlink-install --packages-up-to test_communication
$ grep -i cyclone log/latest_build/test_communication/*
(cyclone matches in logs)
no matches
My assumption is that log matches would indicate cyclone tests were built as indicated by the line in the CMakeLists.txt
Required Info:
Run rclpy
-to-rclcpp
pub/sub tests using Connext on Windows:
colcon test --event-handlers=console_direct+ --packages-select test_communication --ctest-args " -R" " ^test_publisher_subscriber__rclpy__rclcpp__rmw_connext_cpp"
All tests pass.
Subscription to a WString publisher fails with:
18: [test_subscriber-24] r
18: INFO:test_subscriber-24-stderr:[test_subscriber-24] r
18: [test_subscriber-24] eceived message does not match any expected message
18: INFO:test_subscriber-24-stderr:[test_subscriber-24] eceived message does not match any expected message
18: INFO:test_publisher-23-stdout:[test_publisher-23]
18: [ERROR] [test_subscriber-24]: process has died [pid 3156, exit code 3221226505, cmd 'C:/Users/Administrator/nuclearsandwich_ws/build/test_communication/Release/test_subscriber_cpp.exe WStrings /test_time_21_23_20'].
18: ERROR:test_subscriber-24:process has died [pid 3156, exit code 3221226505, cmd 'C:/Users/Administrator/nuclearsandwich_ws/build/test_communication/Release/test_subscriber_cpp.exe WStrings /test_time_21_23_20'].
This issue does not manifest itself on rclcpp
-to-rclcpp
nor rclpy
-to-rclpy
nor rclcpp
-to-rclpy
tests.
The process terminates almost silently. Looking at the rclcpp
subscriber implementation, this suggests that a crash occurs during rclcpp::shutdown
.
#190 arranges for installation of messages and helpful Python code that's now being used elsewhere (ros2/cli_tools#5). But it would be better to move that stuff into a different repo (#190 (comment)).
Required Info:
Run test_rclcpp tests on ARM64 or macOS and you have a chance of the test timing out rather than completing. Here's an example build where I reached into ament_cppcheck and bumped the timeout up to 240. The test completed in 131s.
As seen on CI
cppcheck runs to completion
cppcheck run times out before completion generating no test result output.
Failures of this test on CI is a known issue and is being tracked at ros2/build_farmer#166
Users who don't already know about this would think to look in this package, where the test is from.
Required Info:
colcon test --packages-select test_communication
If failures exist, each test case terminates in a short (<1s) time and reports a relevant failure message (something like "10 messages were sent but 0 were received").
On failure, the test takes a long time (10s) and the message only reports "timed out waiting for ... to finish". This makes it sound like the receiving process deadlocked. Additionally, the assertion uses a confusing string representation of the launch action object, where the subscriber executable name and arguments would be more appropriate.
Traceback (most recent call last):
File "/home/jenkins-agent/workspace/nightly_linux-aarch64_extra_rmw_release/ws/build/test_communication/test_publisher_subscriber__rclpy__rclcpp__rmw_fastrtps_dynamic_cpp__rmw_cyclonedds_cpp_Release.py", line 66, in test_subscriber_terminates_in_a_finite_amount_of_time
proc_info.assertWaitForShutdown(process=subscriber_process, timeout=10)
File "/home/jenkins-agent/workspace/nightly_linux-aarch64_extra_rmw_release/ws/install/launch_testing/lib/python3.6/site-packages/launch_testing/proc_info_handler.py", line 144, in assertWaitForShutdown
assert success, "Timed out waiting for process '{}' to finish".format(process)
AssertionError: Timed out waiting for process '<launch.actions.execute_process.ExecuteProcess object at 0xffff7e693e80>' to finish
Required Info:
Build test_security twice:
colcon build --merge-install --packages-select test_security
colcon build --merge-install --packages-select test_security
Compiles twice without error.
The second time, we see the following error:
--- stderr: test_security
CMake Error at CMakeLists.txt:341 (message):
Failed to generate security artifacts: Traceback (most recent call last):
File "C:\Users\jacob\dev\ws\ros\foxy\install\Scripts\ros2-script.py", line 11, in <module>
load_entry_point('ros2cli==0.9.3', 'console_scripts', 'ros2')()
File "C:\Users\jacob\dev\ws\ros\foxy\install\Lib\site-packages\ros2cli\cli.py", line 67, in main
rc = extension.main(parser=parser, args=args)
File "C:\Users\jacob\dev\ws\ros\foxy\install\Lib\site-packages\sros2\command\security.py", line 36, in main
return extension.main(args=args)
File "C:\Users\jacob\dev\ws\ros\foxy\install\Lib\site-packages\sros2\verb\generate_artifacts.py", line 47, in main
success = _artifact_generation.generate_artifacts(
File "C:\Users\jacob\dev\ws\ros\foxy\install\Lib\site-packages\sros2\api\_artifact_generation.py", line 31, in generate_artifacts
if not _key.create_key(keystore_path, identity):
File "C:\Users\jacob\dev\ws\ros\foxy\install\Lib\site-packages\sros2\api\_key.py", line 86, in create_key
_permission.create_permission_file(permissions_path, _utilities.domain_id(), policy_element)
File "C:\Users\jacob\dev\ws\ros\foxy\install\Lib\site-packages\sros2\api\_permission.py", line 65, in create_permission_file
cert_content = _utilities.load_cert(cert_path)
File "C:\Users\jacob\dev\ws\ros\foxy\install\Lib\site-packages\sros2\api\_utilities.py", line 131, in load_cert
return x509.load_pem_x509_certificate(
File "c:\python38\lib\site-packages\cryptography\x509\base.py", line 52, in load_pem_x509_certificate
return backend.load_pem_x509_certificate(data)
File "c:\python38\lib\site-packages\cryptography\hazmat\backends\openssl\backend.py", line 1222, in load_pem_x509_certificate
raise ValueError(
ValueError: Unable to load certificate. See
https://cryptography.io/en/latest/faq/#why-can-t-i-import-my-pem-file for
more details.
If I remove the build directory for test_security, then I can compile again.
Starting from a failure in a nightly job (http://ci.ros2.org/view/nightly/job/nightly_osx/147/consoleFull#console-section-170 - search for test_services_cpp__rmw_opensplice_cpp ..........................***Timeout
), I can locally reproduce an apparent deadlock in the services test like so:
while true; do nosetests3 -s ./build/test_rclcpp/test_services_cpp__rmw_opensplice_cpp.py; done
Eventually, it'll stop. Then if I attach gdb to each process, the server has this backtrace:
#0 0x0000000000416127 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7ffeb6060268,
__in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:545
#1 0x0000000000415fc6 in std::__shared_ptr<rclcpp::node::Node, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7ffeb6060260,
__in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:781
#2 0x0000000000415fe0 in std::shared_ptr<rclcpp::node::Node>::~shared_ptr (this=0x7ffeb6060260, __in_chrg=<optimized out>)
at /usr/include/c++/4.8/bits/shared_ptr.h:93
#3 0x00007ff01409578a in rclcpp::executor::Executor::get_next_timer(std::shared_ptr<rclcpp::executor::AnyExecutable>) ()
from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#4 0x00007ff014095c25 in rclcpp::executor::Executor::get_next_ready_executable() ()
from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#5 0x00007ff014095eda in rclcpp::executor::Executor::get_next_executable(std::chrono::duration<long, std::ratio<1l, 1000000000l> >)
() from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#6 0x00007ff01409dd5b in rclcpp::executors::single_threaded_executor::SingleThreadedExecutor::spin() ()
from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#7 0x00007ff01409928c in rclcpp::spin(std::shared_ptr<rclcpp::node::Node>) ()
from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#8 0x0000000000415d8d in main (argc=1, argv=0x7ffeb60605e8)
at /home/gerkey/ros2_ws/src/ros2/system_tests/test_rclcpp/test/test_services_server.cpp:50
Not surprising; it's waiting in spin()
.
The client's backtrace is:
#0 0x0000000000489fce in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::~_Sp_counted_base (this=0xff0d00,
__in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:114
#1 0x000000000049311f in std::_Sp_counted_ptr<test_rclcpp::srv::AddTwoInts_Response_<std::allocator<void> >*, (__gnu_cxx::_Lock_policy)2>::~_Sp_counted_ptr (this=0xff0d00, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:281
#2 0x000000000049314e in std::_Sp_counted_ptr<test_rclcpp::srv::AddTwoInts_Response_<std::allocator<void> >*, (__gnu_cxx::_Lock_policy)2>::~_Sp_counted_ptr (this=0xff0d00, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:281
#3 0x00000000004931ed in std::_Sp_counted_ptr<test_rclcpp::srv::AddTwoInts_Response_<std::allocator<void> >*, (__gnu_cxx::_Lock_policy)2>::_M_destroy (this=0xff0d00) at /usr/include/c++/4.8/bits/shared_ptr_base.h:294
#4 0x0000000000487520 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0xff0d00)
at /usr/include/c++/4.8/bits/shared_ptr_base.h:161
#5 0x00000000004860d3 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7ffd77070218,
__in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:546
#6 0x00007f65a1fc0780 in std::__shared_ptr<void, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() ()
from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#7 0x00007f65a1fc079a in std::shared_ptr<void>::~shared_ptr() ()
from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#8 0x00007f65a1fc28f6 in rclcpp::executor::Executor::execute_client(std::shared_ptr<rclcpp::client::ClientBase>) ()
from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#9 0x00007f65a1fc21c4 in rclcpp::executor::Executor::execute_any_executable(std::shared_ptr<rclcpp::executor::AnyExecutable>) ()
from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#10 0x00007f65a1fc1d52 in rclcpp::executor::Executor::spin_once(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) ()
from /home/gerkey/ros2_ws/install/lib/librclcpp__rmw_opensplice_cpp.so
#11 0x0000000000489ee0 in rclcpp::executor::Executor::spin_until_future_complete<std::pair<std::shared_ptr<test_rclcpp::srv::AddTwoInts_Request_<std::allocator<void> > >, std::shared_ptr<test_rclcpp::srv::AddTwoInts_Response_<std::allocator<void> > > >, std::ratio<1l, 1000l> > (this=0x7ffd77070400, future=..., timeout=...) at /home/gerkey/ros2_ws/install/include/rclcpp/executor.hpp:167
#12 0x0000000000488d30 in rclcpp::executors::spin_node_until_future_complete<std::pair<std::shared_ptr<test_rclcpp::srv::AddTwoInts_Request_<std::allocator<void> > >, std::shared_ptr<test_rclcpp::srv::AddTwoInts_Response_<std::allocator<void> > > >, std::ratio<1l, 1000l> > (executor=..., node_ptr=..., future=..., timeout=...) at /home/gerkey/ros2_ws/install/include/rclcpp/executors.hpp:67
#13 0x0000000000487440 in rclcpp::spin_until_future_complete<std::pair<std::shared_ptr<test_rclcpp::srv::AddTwoInts_Request_<std::allocator<void> > >, std::shared_ptr<test_rclcpp::srv::AddTwoInts_Response_<std::allocator<void> > > >, std::ratio<1l, 1000l> > (
node_ptr=..., future=..., timeout=...) at /home/gerkey/ros2_ws/install/include/rclcpp/executors.hpp:81
#14 0x00000000004841d1 in test_services_client_test_return_request_Test::TestBody (this=0xf88e40)
at /home/gerkey/ros2_ws/src/ros2/system_tests/test_rclcpp/test/test_services_client.cpp:76
#15 0x00000000004aeddc in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0xf88e40,
method=&virtual testing::Test::TestBody(), location=0x4bb49b "the test body") at /usr/src/gtest/src/gtest.cc:2090
#16 0x00000000004aa3cc in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0xf88e40,
method=&virtual testing::Test::TestBody(), location=0x4bb49b "the test body") at /usr/src/gtest/src/gtest.cc:2126
#17 0x0000000000497e83 in testing::Test::Run (this=0xf88e40) at /usr/src/gtest/src/gtest.cc:2162
#18 0x0000000000498588 in testing::TestInfo::Run (this=0xf88830) at /usr/src/gtest/src/gtest.cc:2338
#19 0x0000000000498ae4 in testing::TestCase::Run (this=0xf885d0) at /usr/src/gtest/src/gtest.cc:2445
#20 0x000000000049d58a in testing::internal::UnitTestImpl::RunAllTests (this=0xf882d0) at /usr/src/gtest/src/gtest.cc:4243
#21 0x00000000004afd1d in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (
object=0xf882d0,
method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x49d31c <testing::internal::UnitTestImpl::RunAllTests()>, location=0x4bbfd8 "auxiliary test code (environments or event listeners)") at /usr/src/gtest/src/gtest.cc:2090
#22 0x00000000004ab2d6 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (
object=0xf882d0,
method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x49d31c <testing::internal::UnitTestImpl::RunAllTests()>, location=0x4bbfd8 "auxiliary test code (environments or event listeners)") at /usr/src/gtest/src/gtest.cc:2126
#23 0x000000000049c4f5 in testing::UnitTest::Run (this=0x6dfc00 <testing::UnitTest::GetInstance()::instance>)
at /usr/src/gtest/src/gtest.cc:3880
#24 0x00000000004842f4 in main (argc=2, argv=0x7ffd77070938)
at /home/gerkey/ros2_ws/src/ros2/system_tests/test_rclcpp/test/test_services_client.cpp:83
It's waiting in spin_until_future_complete()
for the test_return_request
test (https://github.com/ros2/system_tests/blob/master/test_rclcpp/test/test_services_client.cpp#L70 - line numbers are different in my backtrace because I'm adding print statements).
From added print statements, I can see that the server gets the callback handle_add_two_ints_noreqid()
invoked once with args (1,2)
and gets the callback handle_add_two_ints_reqid()
invoked twice, both times with args (4,5)
. So all three service requests are getting to the server. But the third response doesn't make its way back to the client.
Hi @dirk-thomas,
This is Anita from Erle Robotics. Recently, under the supervision of @vmayoral I've been trying to port ROS 1 packages to ROS 2. However, working with services I get errors with non-primitive field types.
I followed these steps:
· Build the package with primitives. (In this case the package builds perfectly)
· Build the package with both primitives and non-primitives. (I get an AttributeError: 'ServiceSpecification' object has no attribute 'base_type'
error)
· Again, build the package just with primitives. (I still get errors)
Here is the link of the package: https://github.com/erlerobot/ros2_spider_msgs
I'm following the Migration guide from ROS 1, am I missing something?
Thank you,
Anita
In test_security, tests for access control plugin are missing. I am wondering if there is a plan to add that? I have some tests that target the access control security plugin, but my tests generate policy file at run time, and clean them up before tests are finished.
If you don't already have a plan for access control plugin, could you let me know if you'd prefer to have tests generate policy file at runtime or commit all security-related files? I'd like to contribute.
thanks.
There are new test failures in the linux nightly build:
First failures: https://ci.ros2.org/view/nightly/job/nightly_linux_debug/1456/ (Feb 11, 2020)
Failures are:
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Arrays]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BasicTypes]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BoundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Builtins]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Constants]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Defaults]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Empty]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[MultiNested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Nested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Strings]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[UnboundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[WStrings]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Arrays]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BasicTypes]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BoundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Builtins]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Constants]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Defaults]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Empty]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[MultiNested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Nested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Strings]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[UnboundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[WStrings]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Arrays]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BasicTypes]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BoundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Builtins]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Constants]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Defaults]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Empty]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[MultiNested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Nested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Strings]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[UnboundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[WStrings]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Arrays]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BasicTypes]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[BoundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Builtins]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Constants]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Defaults]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Empty]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[MultiNested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Nested]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[Strings]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[UnboundedSequences]
test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[WStrings]
Failure message:
Traceback (most recent call last):
File "/home/jenkins-agent/workspace/nightly_linux_debug/ws/build/test_communication/test_publisher_subscriber__rclpy__rmw_connext_cpp__rmw_fastrtps_cpp_Debug.py", line 66, in test_subscriber_terminates_in_a_finite_amount_of_time
proc_info.assertWaitForShutdown(process=subscriber_process, timeout=10)
File "/home/jenkins-agent/workspace/nightly_linux_debug/ws/install/launch_testing/lib/python3.6/site-packages/launch_testing/proc_info_handler.py", line 144, in assertWaitForShutdown
assert success, "Timed out waiting for process '{}' to finish".format(process)
AssertionError: Timed out waiting for process '<launch.actions.execute_process.ExecuteProcess object at 0x7f16457296a0>' to finish
There are 2 branches currently in this repository that start adding QoS tests:
These should probably be combined, but then we run into a combinatorial problem; we would test all combinations of RMW implementations x the rclcpp or rclpy x all QoS parameters, which would lead to a very long test. What we should probably do is combine the above, and then choose some subset of the tests that make sense and run those.
node_creation
successtest_security includes tests that target the authentication plugin. I see that it's making the node_creation fail when security env. variables are not set or are not set correctly. Perhaps we should also add the successful cases where a node is authentication and able to be created.
Two packages test command line arguments passed to a node: test_cli
tests parameters (ex: __params:=/path/to/initialvalues.yaml
) while test_cli_remapping
tests remap rules (ex: foo:=/bar/foo
). The remapping test should be moved into the test_cli
package.
Bonus points would be to reuse code between the two packages.
Now that Python Actions are available we can add tests across RMW implementations and between C++ and Python client libraries.
Related TODOs:
system_tests/test_communication/CMakeLists.txt
Lines 256 to 259 in e812374
system_tests/test_communication/CMakeLists.txt
Lines 318 to 320 in e812374
system_tests/test_communication/CMakeLists.txt
Lines 328 to 330 in e812374
Metaticket ros2/ros2#583
Links:
(Not sure whether this is the same problem as #39.)
Below is the console output from a failure seen in a nightly build on OSX, running the multithreaded test with connext.
Looks like this is the sequence of events (https://github.com/ros2/system_tests/blob/master/test_rclcpp/test/test_multithreaded.cpp#L78-L110):
I'm not sure of the correct fix here. Increase the heuristic delay at the beginning (https://github.com/ros2/system_tests/blob/master/test_rclcpp/test/test_multithreaded.cpp#L72)? Spin more before checking the initial delivery (https://github.com/ros2/system_tests/blob/master/test_rclcpp/test/test_multithreaded.cpp#L83)? Something else?
[ RUN ] test_multithreaded__.multi_consumer_intra_process
RTI Data Distribution Service Evaluation License issued to OSRF (OSRF01) [email protected] For non-production use only.
Expires on 08-jun-2016 See www.rti.com for more information.
/Users/osrf/jenkins/workspace/nightly_osx/ws/src/ros2/system_tests/test_rclcpp/test/test_multithreaded.cpp:84: Failure
Value of: subscriptions.size()
Actual: 4
Expected: counter.load()
Which is: 0
callback() 1 with message data 1
/Users/osrf/jenkins/workspace/nightly_osx/ws/src/ros2/system_tests/test_rclcpp/test/test_multithreaded.cpp:88: Failure
Value of: subscriptions.size()
Actual: 4
Expected: counter.load()
Which is: 1
callback() 1 with message data 1
callback() 2 with message data 1
callback() 3 with message data 1
callback() 4 with message data 1
callback() 5 with message data 1
callback() 6 with message data 1
callback() 7 with message data 1
callback() 8 with message data 2
callback() 9 with message data 2
callback() 10 with message data 2
callback() 11 with message data 2
callback() 12 with message data 3
callback() 13 with message data 3
callback() 14 with message data 3
callback() 15 with message data 3
callback() 16 with message data 4
callback() 17 with message data 4
callback() 18 with message data 4
callback() 19 with message data 4
callback() 20 with message data 5
callback() 21 with message data 5
callback() 22 with message data 5
callback() 23 with message data 5
/Users/osrf/jenkins/workspace/nightly_osx/ws/src/ros2/system_tests/test_rclcpp/test/test_multithreaded.cpp:110: Failure
Value of: 5 * subscriptions.size()
Actual: 20
Expected: counter.load()
Which is: 23
Required Info:
Here is an example failure:
Note, the "failed to set parameter" messages are expected output of the test.
I'm able to reproduce locally on Linux (note the test is flakey, but happens more often than not):
colcon test --packages-select test_rclcpp --ctest-args -R "test_parameter_server_cpp__rmw_fastrtps_cpp" --event-handlers console_direct+
Tests pass.
Tests fail (timeout)
With #419 checked out, we can see that the different tests hang at different points in the execution.
With additional print statements, I've been able to determine that the timeouts occur due to the parameter client waiting forever for a service response (GetParameters.srv, SetParameters.srv, or ListParameters.srv). It looks to me that the service request always makes it to the service server (which succesfully makes the rcl call to send the response), but the response sometimes never reaches the client.
So far, I've only been focused on Fast-RTPS, which is the RMW we see failing on the nightlies.
Also, I've only seen the *_sync
tests fail. I have not witnessed a test with an AsyncParametersClient
fail to get a response.
Required Info:
colcon test --packages-select test_communication
If the action server or action client is terminated ungracefully, tests should fail.
Tests pass despite log output that indicates cleanup failed:
ros2/rclpy#524
17: Test command: /usr/bin/python3 "-u" "/tmp/ws/install_isolated/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/tmp/ws/test_results/test_communication/test_action_client_server__rclpy__rmw_fastrtps_cpp.xunit.xml" "--package-name" "test_communication" "--output-file" "/tmp/ws/build_isolated/test_communication/launch_test/CHANGEME.txt" "--append-env" "LD_LIBRARY_PATH=/tmp/ws/build_isolated/test_communication" "--command" "/usr/bin/python3" "-m" "launch_testing.launch_test" "/tmp/ws/build_isolated/test_communication/test_action_client_server__rclpy__rmw_fastrtps_cpp_.py" "--junit-xml=/tmp/ws/test_results/test_communication/test_action_client_server__rclpy__rmw_fastrtps_cpp.xunit.xml" "--package-name=test_communication"
17: Test timeout computed to be: 60
17: -- run_test.py: extra environment variables to append:
17: - LD_LIBRARY_PATH+=/tmp/ws/build_isolated/test_communication
17: -- run_test.py: invoking following command in '/tmp/ws/build_isolated/test_communication':
17: - /usr/bin/python3 -m launch_testing.launch_test /tmp/ws/build_isolated/test_communication/test_action_client_server__rclpy__rmw_fastrtps_cpp_.py --junit-xml=/tmp/ws/test_results/test_communication/test_action_client_server__rclpy__rmw_fastrtps_cpp.xunit.xml --package-name=test_communication
17:
17: Starting test run test_communication.test_action_client_server__rclpy__rmw_fastrtps_cpp_.launch_tests[Fibonacci]
17: [INFO] [launch]: All log files can be found below /home/buildfarm/.ros/log/2020-03-12-04-03-39-582329-e523ecceac2a-5255
17: [INFO] [launch]: Default logging verbosity is set to INFO
17: test_client_finishes_in_a_finite_amount_of_time[Fibonacci] (test_communication.TestActionClientServer)
17: Test that the second executable terminates after a finite amount of time. ... [INFO] [test_action_server-1]: process started with pid [5258]
17: [INFO] [test_action_client-2]: process started with pid [5260]
17: [test_action_client-2] Sending goal for test number 0
17: [test_action_client-2] Exception ignored in: <function ActionClient.__del__ at 0x7f2115f12700>
17: [test_action_client-2] Traceback (most recent call last):
17: [test_action_client-2] File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/action/client.py", line 596, in __del__
17: [test_action_client-2] File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/action/client.py", line 589, in destroy
17: [test_action_client-2] File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/handle.py", line 124, in __enter__
17: [test_action_client-2] File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/handle.py", line 106, in _get_capsule
17: [test_action_client-2] rclpy.handle.InvalidHandle: Tried to use a handle that has been destroyed.
17: [INFO] [test_action_client-2]: process has finished cleanly [pid 5260]
17: ok
17:
17: ----------------------------------------------------------------------
17: Ran 1 test in 1.187s
17:
17: OK
17: [INFO] [test_action_server-1]: sending signal 'SIGINT' to process[test_action_server-1]
17: [test_action_server-1] Exception ignored in: <function ActionServer.__del__ at 0x7fce84bc3040>
17: [test_action_server-1] Traceback (most recent call last):
17: [test_action_server-1] File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/action/server.py", line 625, in __del__
17: [test_action_server-1] File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/action/server.py", line 618, in destroy
17: [test_action_server-1] File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/handle.py", line 124, in __enter__
17: [test_action_server-1] File "/tmp/ws/install_isolated/rclpy/lib/python3.8/site-packages/rclpy/handle.py", line 106, in _get_capsule
17: [test_action_server-1] rclpy.handle.InvalidHandle: Tried to use a handle that has been destroyed.
17: [INFO] [test_action_server-1]: process has finished cleanly [pid 5258]
17: test_processes_finished_gracefully[Fibonacci] (test_communication.TestActionClientServerAfterShutdown)
17: Test that both executables finished gracefully. ... ok
17:
17: ----------------------------------------------------------------------
17: Ran 1 test in 0.000s
17:
17: OK
Refactor the multi_target
macro to generate tests for the different configurations:
To do so, need to fix python char and test template to match current cpp tests
test_messages_c.cpp defines its own fixtures and is a monolothic >1000 line file. We should refactor it to use create executables similar to the c++ test_publisher/test_subscriber and move the fixtures to the test_msgs package
It has failed a few times on the CI, but locally I cannot reproduce it. I think we can just add more re-try loops, but first I want to understand why two is not enough.
System info:
https://ci.ros2.org/view/nightly/job/nightly_win_deb/1589/
The crash fails the test case associated with it.
The useful stack trace appears in the passing test case "test_subscriber_terminates_in_a_finite_amount_of_time[WStrings]"
139: test_subscriber_terminates_in_a_finite_amount_of_time[WStrings] (test_communication.TestPublisherSubscriber)
139: Test that the subscriber terminates after a finite amount of time. ... [INFO] [test_publisher-23]: process started with pid [5128]
139: [INFO] [test_subscriber-24]: process started with pid [1732]
139: [test_subscriber-24] exception in subscriber:
139: [test_subscriber-24] Traceback (most recent call last):
139: [test_subscriber-24] File "C:/ci/ws/src/ros2/system_tests/test_communication/test/subscriber_py.py", line 81, in <module>
139: [test_subscriber-24] listener(message_name=args.message_name, namespace=args.namespace)
139: [test_subscriber-24] File "C:/ci/ws/src/ros2/system_tests/test_communication/test/subscriber_py.py", line 65, in listener
139: [test_subscriber-24] rclpy.spin_once(node)
139: [test_subscriber-24] File "C:\ci\ws\install\Lib\site-packages\rclpy\__init__.py", line 171, in spin_once
139: [test_subscriber-24] executor.spin_once(timeout_sec=timeout_sec)
139: [test_subscriber-24] File "C:\ci\ws\install\Lib\site-packages\rclpy\executors.py", line 685, in spin_once
139: [test_subscriber-24] raise handler.exception()
139: [test_subscriber-24] File "C:\ci\ws\install\Lib\site-packages\rclpy\task.py", line 239, in __call__
139: [test_subscriber-24] self._handler.send(None)
139: [test_subscriber-24] File "C:\ci\ws\install\Lib\site-packages\rclpy\executors.py", line 397, in handler
139: [test_subscriber-24] arg = take_from_wait_list(entity)
139: [test_subscriber-24] File "C:\ci\ws\install\Lib\site-packages\rclpy\executors.py", line 325, in _take_subscription
139: [test_subscriber-24] msg = _rclpy.rclpy_take(capsule, sub.msg_type, sub.raw)
139: [test_subscriber-24] UnicodeDecodeError: 'utf-16-le' codec can't decode bytes in position 0-1: illegal encoding
139: [ERROR] [test_subscriber-24]: process has died [pid 1732, exit code 1, cmd 'C:\Python37\python_d.exe C:/ci/ws/src/ros2/system_tests/test_communication/test/subscriber_py.py WStrings /test_time_11_49_47'].
139: [WARNING] [test_publisher-23]: 'SIGINT' sent to process[test_publisher-23] not supported on Windows, escalating to 'SIGTERM'
139: [INFO] [test_publisher-23]: sending signal 'SIGTERM' to process[test_publisher-23]
139: [ERROR] [test_publisher-23]: process has died [pid 5128, exit code 1, cmd 'C:\Python37\python_d.exe C:/ci/ws/src/ros2/system_tests/test_communication/test/publisher_py.py WStrings /test_time_11_49_47'].
139: ok
A useless stack trace appears in the failed test case:
139: test_processes_finished_gracefully[WStrings] (test_communication.TestPublisherSubscriberAfterShutdown)
139: Test that both executables finished gracefully. ... FAIL
139:
139: ======================================================================
139: FAIL: test_processes_finished_gracefully[WStrings] (test_communication.TestPublisherSubscriberAfterShutdown)
139: Test that both executables finished gracefully.
139: ----------------------------------------------------------------------
139: Traceback (most recent call last):
139: File "C:\ci\ws\build\test_communication\test_publisher_subscriber__rclpy__rmw_fastrtps_cpp_Debug.py", line 77, in test_processes_finished_gracefully
139: 'test_subscriber'
139: File "C:\ci\ws\install\Lib\site-packages\launch_testing\asserts\assert_exit_codes.py", line 64, in assertExitCodes
139: info.returncode
139: AssertionError: Proc test_subscriber-24 exited with code 1
Unfortunately, searching these logs for the string "[ERROR]
" is made somewhat less useful since even a passing test cases send a SIGTERM to the subprocess, causing it to log an error.
http://ci.ros2.org/job/nightly_osx_repeated/863
projectroot.test_secure_publisher_subscriber__DynamicArrayNested__rmw_connext_cpp__secure_not_connecting_0
projectroot.test_secure_publisher_subscriber__DynamicArrayNested__rmw_connext_cpp__secure_not_connecting_1
projectroot.test_secure_publisher_subscriber__Empty__rmw_connext_cpp__secure_not_connecting_1
projectroot.test_secure_publisher_subscriber__DynamicArrayNested__rmw_fastrtps_cpp__secure_not_connecting_1
They are receiving connections when they're not supposed to. We know that some nodes can take a while to shutdown properly so perhaps it's cross-talk from re-running the tests multiple times. I'll add a namespace to the nodes and see if it fixes it.
Required Info:
colcon test --event-handlers console_direct+ --packages-select test_communication --ctest-args -R test_action_client_server__Fibonacci__rclcpp__rmw_fastrtps_cpp__rmw_connext_cpp
RMW_IMPLEMENTATION=rmw_fastrtps_cpp ./build/test_communication/test_action_client_cpp Fibonacci /foo/bar
RMW_IMPLEMENTATION=rmw_connext_cpp ./build/test_communication/test_action_server_cpp Fibonacci /foo/bar
Same behavior if the rmw implementations are flipped so it's a connext client with fastrtps server. If both are fastrtps or both are connext it works as expected. The service/client and subscriber/publisher tests appear to work fine across fastrtps and connext.
This is tracking the parts of test_security
not implemented in #230
#230 tests authentication and encryption in various scenarios but doesnt test access control.
Tests for access control should:
ros2 security
tool to generate keys, certificates, governance and permission files for these testsA declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.