The Distro 2022.11.0 will be the last one providing iCub_SIM (check the announcement).

Thus, we're required to update all those tests that now rely on iCub_SIM (if there are any) to the use of Gazebo instead and clean up any leftovers.

cc @Nicogene @valegagge

on win32, both rtf and icub-tests on devel branch:
when the project is generated with cmake gui I get a lot of errors like:

CMake Error at src/movementReferencesTest/CMakeLists.txt:22 (add_library):
  Target "movementReferencesTest" links to target "YARP::YARP_rtf" but the
  target was not found.  Perhaps a find_package() call is missing for an
  IMPORTED target, or an ALIAS target is missing?

The generation then completes but I'm not able to compile.

Error C1083: Cannot open include file: 'yarp/rtf/TestCase.h': 

Additionally, when compiling rtf I get the following warnings:

C:\software\robot-testing\src\plugins\dll\include\rtf/dll/DllPluginLoader.h(29): warning C4275: non dll-interface class 'RTF::plugin::PluginLoader' used as base for dll-interface class 'RTF::plugin::DllPluginLoader'
          C:\software\robot-testing\src\rtf\include\rtf/PluginLoader.h(32) : see declaration of 'RTF::plugin::PluginLoader'
          C:\software\robot-testing\src\plugins\dll\include\rtf/dll/DllPluginLoader.h(29) : see declaration of 'RTF::plugin::DllPluginLoader'

these warnings are typically a source of problems...
cc @Nicogene @apaikan

Installing the project via robotology-superbuild made evident how it's not straightforward to launch a test.

Take this docker recipe as an example, where the following points could be improved:

• Need for extending LD_LIBRARY_PATH
• The suites are not installed

cc @Nicogene @traversaro @mfussi66

After YARP 3 release.

add documentation to icub-tests (e.g. icub-tests\src\motor-tests)

Follow the procedure shown in: https://github.com/robotology/icub-tests to update documentation

For building test on external repository (for example codyco software or coman/walkman software), we need to expose the Yarp stuff.

We could create a yarp-tests library, given that this bits are iCub - indipendent, but probably requiring three different repos (robot-testing,yarp-tests,icub-tests) to be installed is an overkill, especially for users that don't use the superbuild.

Another option is to simply add the cmake logic to export them as a library directly from this repo for the moment.

I have yarp robot-testing and icub-main up-to-date on devel branch and the following test are not compiling:

• motor-striction (missing using namespace std;)
• opticalEncoders-drift (missing using namespace std;)
• motorEncodersSignCheck IPidControl API changed(@randaz81)
• jointLimits IPidControl API changed(@randaz81)

The project fails to compile on Macos, in the devel branch, with the error:

rc/positionControl-accuracy/PositionControlAccuracy.cpp -o /Users/nunoguedelha/dev/robotology-superbuild/build/robotology/icub-tests/src/positionControl-accuracy/iCub-Tests.build/Release/PositionControlAccuracy.build/Objects-normal/x86_64/PositionControlAccuracy.o
/Users/nunoguedelha/dev/robotology-superbuild/robotology/icub-tests/src/positionControl-accuracy/PositionControlAccuracy.cpp:108:17: error: no member named 'nanf' in namespace 'std'; did you mean simply 'nanf'?
    double p_Kp=std::nanf("");
                ^~~~~~~~~
                nanf

isnan(), nanf(), etc.., are defined in the namespacestd in the header <cmath.h> (*) but not in the header <math> (**) which is the one included in the PositionControlAccuracy.cpp file.

(*) C++ standard: https://en.cppreference.com/w/cpp/numeric/math/nan
(**) C stancdard: https://en.cppreference.com/w/c/numeric/math/nan

So either the include is changed to <cmath.h>, either the prefix std should be removed.

If I'm not mistaken this repo does not depend on icub-main. At the current moment, if icub-main is not installed, the build process of the devel branch fails with the following error:

-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done

CMake Error at src/positionControl-accuracy-ExternalPid/CMakeLists.txt:12 (find_package):
  By not providing "FindICUB.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "ICUB", but
  CMake did not find one.

  Could not find a package configuration file provided by "ICUB" with any of
  the following names:

    ICUBConfig.cmake
    icub-config.cmake

  Add the installation prefix of "ICUB" to CMAKE_PREFIX_PATH or set
  "ICUB_DIR" to a directory containing one of the above files.  If "ICUB"
  provides a separate development package or SDK, be sure it has been
  installed.

This behavior is due to one of the latest tests added, and particularly this line:

cc @randaz81

Actually, also this line is involved:

It was a while I was not updating the setup which includes icub-tests, and last summer I had no icub-main there.

cc @Nicogene

@randaz81 i got this error message while building icub-tests:

[ 46%] Building CXX object src/opticalEncoders-consistency/CMakeFiles/OpticalEncodersConsistency.dir/opticalEncodersConsistency.cpp.o
/home/apaikan/Install/robotology/icub-tests/src/opticalEncoders-consistency/opticalEncodersConsistency.cpp: In member function ‘virtual void OpticalEncodersConsistency::run()’:
/home/apaikan/Install/robotology/icub-tests/src/opticalEncoders-consistency/opticalEncodersConsistency.cpp:407:44: error: no matching function for call to ‘yarp::os::Bottle::read(yarp::sig::Vector)’
/home/apaikan/Install/robotology/icub-tests/src/opticalEncoders-consistency/opticalEncodersConsistency.cpp:407:44: note: candidates are:
In file included from /home/apaikan/Install/yarp-apaikan/src/libYARP_OS/include/yarp/os/Value.h:16:0,
                 from /home/apaikan/Install/yarp-apaikan/src/libYARP_OS/include/yarp/os/Network.h:16,
                 from /home/apaikan/Install/yarp-apaikan/src/libYARP_OS/include/yarp/os/ManagedBytes.h:14,
                 from /home/apaikan/Install/yarp-apaikan/src/libYARP_sig/include/yarp/sig/Vector.h:21,
                 from /home/apaikan/Install/yarp-apaikan/src/libYARP_math/include/yarp/math/Math.h:13,
                 from /home/apaikan/Install/robotology/icub-tests/src/opticalEncoders-consistency/opticalEncodersConsistency.cpp:14:

Building the devel branch against yarp:devel, I have the following error:

[  2%] Building CXX object example/cpp/ExampleTest/CMakeFiles/ExampleTest.dir/ExampleTest.cpp.o

In file included from /icub-tests/example/cpp/ExampleTest/ExampleTest.h:13:0,
                 from /icub-tests/example/cpp/ExampleTest/ExampleTest.cpp:10:
/usr/local/include/yarp/rtf/TestCase.h:10:26: fatal error: yarp/rtf/api.h: No such file or directory
compilation terminated.

The yarp/rtf/api.h header is not installed with yarp.

The temporary variable:

is never reset to false after every partial test. So the final result is not correct.

During the tests on the last iCub that should be out at the end of the month, we had to launch the following test:

ENCODERS TEST

From icubsrv, we typed:
cd $ROBOT_CODE/icub-tests/suites
and we executed the following command:
robottestingframework-testrunner -v -s encoders-icub.xml
NB: we modify the file encoders-icub.xml in order to run one test at a time to get a better view of each part.
here the original file encoders-icub.xml:

<?xml version="1.0" encoding="UTF-8"?>

<suite name="Encoders Test Suite">
    <description> Testing encoders</description>
    <environment>--robotname icub</environment>

    <test type="dll" param="--from optical_encoders_drift_left_arm.ini">  OpticalEncodersDrift </test>
    <test type="dll" param="--from optical_encoders_drift_right_arm.ini"> OpticalEncodersDrift </test>
    <test type="dll" param="--from optical_encoders_drift_left_leg.ini">  OpticalEncodersDrift </test>
    <test type="dll" param="--from optical_encoders_drift_right_leg.ini"> OpticalEncodersDrift </test>
    <test type="dll" param="--from optical_encoders_drift_torso.ini">     OpticalEncodersDrift </test>

    <test type="dll" param="--from motorEncoderConsistency_left_arm.ini">  MotorEncodersConsistency </test>
    <test type="dll" param="--from motorEncoderConsistency_right_arm.ini"> MotorEncodersConsistency </test>
    <test type="dll" param="--from motorEncoderConsistency_left_leg.ini">  MotorEncodersConsistency </test>
    <test type="dll" param="--from motorEncoderConsistency_right_leg.ini"> MotorEncodersConsistency </test>
    <test type="dll" param="--from motorEncoderConsistency_torso.ini">     MotorEncodersConsistency </test>
    <test type="dll" param="--from motorEncodersConsistency_face.ini">     MotorEncodersConsistency </test> 
    <test type="dll" param="--from motorEncodersConsistency_head.ini">     MotorEncodersConsistency </test>
</suite>

Every single test is actually refering to some specific .ini file that can be located into the repository:
cd $ROBOT_CODE/icub-tests/suites/contexts/icub/
Let's look at the following the file optical_encoders_drift_left_arm.ini as example

robot     ${robotname}
part      left_arm
joints    (0 1 2 3)
home      (-30 30 10 45)
speed     (20 20 20 20)
max       (-20 40 20 55)
min       (-40 20 0  35)
cycles    1000
tolerance 1.0 
plot_enabled 0

From what we understood TWEAKING the parameter TOLERANCE could prevent the entire test to FAIL miserably as it was the case until we have been notified to modify it.

QUESTIONS

So here are my questions regarding THIS specific test to be addressed to the creator:

1. is there any documentation for this test?
2. why don't the test check by itself different values for TOLERANCE to compatibility give the final result of [PASS] or [FAIL] with no tweak required and in case of FAILURE, giving some advice on what is the problem or what exactly makes this test fail?
3. Most probably, the second question brings a solution on maybe it has not yet been implemented, so I would be grateful if it was possible to get the range of values for which we can consider the test really passed?
   MIN_VALUE<=TOLERANCE<=MAX_VALUE

JOINTLIMIT TEST

From icubsrv, we typed:
cd $ROBOT_CODE/icub-tests/suites
and we executed the following command:
robottestingframework-testrunner -v -s jointsLimits_robotEth.xml
NB: we modify the file jointsLimits_robotEth.xml in order to run one test at a time to get a better view of each part.
here the original file jointsLimits_robotEth.xml:

<?xml version="1.0" encoding="UTF-8"?>

<suite name="Motor Control Suite">
    <description>Testing robot motors</description>
    <environment>--robotname icub --context icubEth</environment>
    <!--fixture param="--fixture icub-fixture.xml"> yarpmanager </fixture-->

    <!-- Joint Limits -->
    <test type="dll" param="--from joint_limits_right_arm.ini"> JointLimits </test>
    <test type="dll" param="--from joint_limits_left_arm.ini"> JointLimits </test>
    <test type="dll" param="--from joint_limits_right_leg.ini"> JointLimits </test>
    <test type="dll" param="--from joint_limits_left_leg.ini"> JointLimits </test>
    <test type="dll" param="--from joint_limits_torso.ini"> JointLimits </test>
    <test type="dll" param="--from joint_limits_head.ini"> JointLimits </test> 
    <!-- only for robot with mc4plus on forearm use following test 
    <test type="dll" param="--from joint_limits_left_forearm.ini"> JointLimits </test>-->
</suite>

Every single test is actually referring to some specific .ini file that can be located into the repository:
cd $ROBOT_CODE/icub-tests/suites/contexts/icubEth/
Let's look at the following the file joint_limits_head.ini as an example:

robot     ${robotname}
part      head
joints    (0 1 2 3 4 5)
home      (0 0 0 0 0 0)
speed     (20 20 20 20 20 20)
outputLimitPercent (50 50 90 90 90 90)
outOfBoundPosition ( 2  2  2  2  2  2)
tolerance 1 

Here again, the first time we launched the test it FAILS, and therefore we needed to tweak the value in order to make it passed BUT unfortunately we dont know if the value for TOLERANCE is TOLERABLE, an indication on the boundaries of TOLERANCE will be much appreciated.

Except from the TOLERANCE parameter I have to say that the test is really PROBLEMATIC because it makes the robot at risk of breaking some part (mainly cover) as the boundaries for some critical joints are NOT correct. (for example: j1 arms can go below 15 degrees, either from the yarpmotorgui, which is even WORSE in my opinion than the test itself, or by the commands requested by the tests which first get the joint limit from IControlLimits::getLimits() which gets it from the values onto the configuration file:
ex:
cd $ROBOT_CODE/robotd-configuration/iCubHongKong01/hardware/mechanicals/
In the file left_arm-eb1-j0_3-mec.xml, one section of it refers to the hardware limit of the robot:

<group name="LIMITS">
        <!--                                    0                1              2              3            -->
        <param name="hardwareJntPosMax">        8              160             80            106       </param>  
        <param name="hardwareJntPosMin">      -95.5              0            -32             15       </param>
        <param name="rotorPosMin">              0                0              0              0       </param> 
        <param name="rotorPosMax">              0                0              0              0       </param>
    </group>

The problem is for joint1 if the test requests to go to the minimum limit, the arm just BREAKS the torso cover 100% sure and probably also the arm cover. If the test is done WITHOUT COVER I am not 100% sure that this test is safe as well.

j1 of the arm is an example but more joints could be faulty as well

QUESTIONS

Here is my thoughts about this test:
In my opinion, this test is not responsible for the bad numbers, values not correctly inserted into the configurations file of the robot - but if it would have been tested thoroughly it would have made emerged that something was off.
I would love to have someone taking care of double-checking every joint of the robot:

1. the hardware limit of the robot WITHOUT and WITH COVER and modify some files accordingly
2. redefine boundaries for the yarpmotorgui to prevent unexpected collisions if moved carelessly
3. Find out the tolerances for which collisions are avoided and by software prevents them.
   4, 5 and 6 are basically the copy of the questions 1, 2 and 3 from encoders test.

Hope this will benefit and ease the production tests and the external users

cc @pattacini , @randaz81 , @valegagge
fyi @Fabrizio69, @davidetome

Related to #50.

Was robotology/icub-main#64

Things to check:

• call to methods that are supposed to read from local buffers (and not from RPC) should be checked for the time spent in the call. This could also be a good example for adding some time related assert/check macros to RTF.

cc @randaz81 @francesco-romano @apaikan @lornat75 @barbalberto

From the installation instructions:
https://robotology.github.io/icub-tests/doxygen/doc/html/installation.html

In case CMake fails to detect the RTF package check that you installed RTF or that you set the RTF_DIR environment variables to point to the build directory. Also check that RTF was compiled with YARP support enables (set ENABLE_MIDDLEWARE_PLUGINS and verify that ENABLE_YARP is ON).

but ENABLE_MIDDLEWARE_PLUGINS, ENABLE_YARP does not exist in RTF...

...this makes robinterface_stream.ini/PortsFrequency fail.

Why is /icubSim/skinManager/skin_events:o present at all?

The installation instructions written in https://robotology.github.io/icub-tests/doxygen/doc/html/installation.html are non corrected.

Moreover the note, that specifies which yarp flags need to be enabled, seems belong only to installation instructions for windows

Some xml fixture files have to be updated in order to work after the latest distro (12.19) that introduced a wide adoption of mas interfaces.

It seems that Travis compilation was broken by the requirement of CMake 3.12 (see robotology/community#364). Instead of fixing it, perhaps it could make sense to just migrate the repo to GitHub Actions (see https://github.com/robotology/assistive-rehab/blob/master/.github/workflows/ci.yml for an example of another repo that uses GitHub Actions).

Some yarp methods are time critical.
We would like to check, for example, that setControlMode() lasts no more than xxx milliseconds (see also issue robotology/icub-main#234).
We have to decide if this kind of tests should be implemented separately (create new tests) or integrated in the existing tests suits (e.g. add calls to yarp::os::Time:now() inside motor-tests, controlModes etc)

@randaz81

I have just noticed that src/torqueControl-accuracy tests has a cmake file but there is not any source code in the folder 🙈

iCubSim takes some time to stabilize framerate: first execution of camera test fails because of this. Maybe we should add the possibilty to add a delay after the fixture:

<environment>--robotname icubSim</environment>
<fixture param="--fixture icubsim-fixture.xml"> yarpmanager </fixture>
<--! add delay about here -->
<!-- Camera -->
<test type="dll" param="--from camera_right.ini"> CameraTest </test>

Write a test which performs the following actions:

PART1:

• set a small pre-defined impedance value (e.g. stiffness 0.2 damping 0.2) on all joints of the robot part.
• read back the impedance from all the joints. Retrieved value should not differ more than (0.05 0.05) from the original command.
• Repeat using a large pre-defined impedance value (e.g. stiffness 10.0 damping 10.0).
• Threshold and test values should be stored in a ini file.

PART2:

• call function yarp::dev::IImpedanceControl::getCurrentImpedanceLimit (int j,double * min_stiff,double * max_stiff,double * min_damp,double * max_damp ))
• repeat the tests using the max/min values obtained by getCurrentImpedanceLimit() in place of the predefined values used in PART1.

Missing installation documentation for LD_LIBRARY_PATH equivalent for macOS and Windows in
https://robotology.github.io/icub-tests/doxygen/doc/html/installation.html

I have just updated my setup which also contains icub-tests and I'm getting the following error

CMake Error at CMakeLists.txt:26 (find_package):
  By not providing "FindRTF.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "RTF", but
  CMake did not find one.

  Could not find a package configuration file provided by "RTF" (requested
  version 1.3.3) with any of the following names:

    RTFConfig.cmake
    rtf-config.cmake

  Add the installation prefix of "RTF" to CMAKE_PREFIX_PATH or set "RTF_DIR"
  to a directory containing one of the above files.  If "RTF" provides a
  separate development package or SDK, be sure it has been installed.


-- Configuring incomplete, errors occurrefrad!
See also "/home/gromualdi/robot-code/robotology-superbuild/build/robotology/icub-tests/CMakeFiles/CMakeOutput.log".

RTF, YARP, icub-main and icub-test are in devel.
Actually, I think that the problem is related to the fact that RFT is installing a .cmake file called

RobotTestingFrameworkConfig.cmake

while icub-test is looking for RTFConfig.cmake
@nunoguedelha

Remember to add a check on the return values of yarp methods related to the motor interfaces.
e.g. getEncoders(), getPid() ecc
http://wiki.icub.org/yarpdoc/group__dev__iface__motor.html

To be propagated on all tests.

see also: robotology/yarp#495

Using conda binaries should speed it up

Let's separate the configuration files for testing icub CAN from icub ETH. This because icub ETH has (for now) a different tolerance when reaching a reference position and a different PWM limit. The option --context will be removed from the suits configuration files, meaning that the user will be required to explicitly set it when running the test from the command line.

When running tests as suggested in the documentation:

testrunner --verbose --suit ./suits/basics-icubSim.xml

I get the following error:

[testrunner] cannot load plugin yarpmanager.so; error (load) : yarpmanager.so: cannot open shared object file: No such file or directory