EGM causes robot controller shutdown with error message "10014 System Failure" on IRB14000 about abb_libegm HOT 15 CLOSED

Here's a video of the joint demo motions (RobotStudio EGM demo motions video.zip).

A 🕺 YuMi. Nice.

from abb_libegm.

I'm not saying this is the cause, but the title of your issue claims:

EGM causes robot controller shutdown

while later you write:

A fully functioning system was achieved with ROS2 Dashing. However, when transitioning to ROS2 Eloquent, the same code crashes the robot controller.

These two don't match.

Unless @jontje recognises this immediately, I would suggest to add more details about your code, the way you are wrapping it with pieces of ROS 2 infrastructure, perhaps even a wireshark trace for both the working and the non-working situations, information on how you installed ROS 2, etc.

Is the same PC running Dashing and Eloquent? Anything else that changed between the two?

from abb_libegm.

Unless @jontje recognises this immediately

I haven't seen EGM cause such a behaviour before, at least as far as I can remember.

A fully functioning system was achieved with ROS2 Dashing. However, when transitioning to ROS2 Eloquent, the same code crashes the robot controller.

I have only tested abb_libegm in ROS2 Dashing against RobotStudio simulations before, which worked fine. I haven't had time to try out ROS2 Eloquent yet, but I must say I find it a bit that strange that only a change in ROS2 distribution would cause the crashes. E.g. since the robot controller doesn't "know" about what the external computer is running.

When a motion command is issued (joint position reference) to the robot controller, it activates the joint brakes on both arms and freezes for a couple of seconds before crashing (FlexPendant reboots etc). It then automatically reboots, but gets stuck trying to connect to the robot controller. When manually powered down and restarted, the FlexPendant informs us the robot controller is in a System Failure state.

@Mariunil, can you provide the robot controller's event logs? E.g. in RobotStudio:

And check All logs when saving:

I would suggest to add more details about your code, the way you are wrapping it with pieces of ROS 2 infrastructure, perhaps even a wireshark trace for both the working and the non-working situations, information on how you installed ROS 2, etc.

Is the same PC running Dashing and Eloquent? Anything else that changed between the two?

In addition to @gavanderhoorn suggestions, have you made any modifications to the StateMachine Add-In's RAPID code? And have you tested what happens if you run your ROS2 implementation against a virtual controller in RobotStudio?

from abb_libegm.

Looks like it's Mariunil/abb_yumi and/or yumi-crew/yumi.

@Mariunil: random suggestion, but: I would avoid copying entire repositories into your own repository. There are tools to manage dependencies, even when building everything from source.

from abb_libegm.

I'm not saying this is the cause, but the title of your issue claims:

EGM causes robot controller shutdown

while later you write:

A fully functioning system was achieved with ROS2 Dashing. However, when transitioning to ROS2 Eloquent, the same code crashes the robot controller.

These two don't match.

The robot controller crashes when a EGM motion command is issued through abb_libegm. As the robot controller and the desktop PC is communicating via the EGM Sensor protocol (handled by abb_libegm), the robot is unaware of the ROS 2 Eloquent system running on the desktop computer. It is only aware of the EGM server spawned by abb_libegm (and the RWS client spawned by abb_librws, but no issues occur here).

The ROS 2 code is organized in the ros2_control framework with a simple joint_position_controller (handled by the controller manager).

ROS 2 dashing and eloquent were both installed following instruction from: https://index.ros.org/doc/ros2/Installation/Dashing/Linux-Install-Debians/ and https://index.ros.org/doc/ros2/Installation/Eloquent/Linux-Install-Debians/

abb_libegm dependency versions:

• Protobuf: 3.6.1
• Boost: 1.65.1.0

As the robot controller is only communicating with abb_libegm resources, and crashing exclusively when issued motion commands, we experience this issue as a abb_libegm issue, and not a ROS2 Eloquent issue. The abb_libegm methods are utilized in the robot's hardware interface in ROS 2.

The same computer was running Dashing and Eloquent. The same behavior have also occurred for two different computers.

I have only tested abb_libegm in ROS2 Dashing against RobotStudio simulations before, which worked fine. I haven't had time to try out ROS2 Eloquent yet, but I must say I find it a bit that strange that only a change in ROS2 distribution would cause the crashes. E.g. since the robot controller doesn't "know" about what the external computer is running.

Agree, we are unable to see how ROS 2 Eloquent can be able to affect the EGM communication, however it seem to be doing just that. We suspect we somehow are sending faulty messages to the robot.

In addition to @gavanderhoorn suggestions, have you made any modifications to the StateMachine Add-In's RAPID code? And have you tested what happens if you run your ROS2 implementation against a virtual controller in RobotStudio?

The StateMachine Add-In's have not been modified. Have not yet tested against a virtual robot in RobotStudio due to issues with RWS calls towards virtual robots on windows machines.

These folders contain wireshark traces and event logs for runs with respectively Dashing and Eloquent.

eloquent.zip
dashing.zip

from abb_libegm.

Looks like it's Mariunil/abb_yumi and/or yumi-crew/yumi.

The code in those repositories are very similar, however it is the code from yumi-crew/yumi that are being executed.

@Mariunil: random suggestion, but: I would avoid copying entire repositories into your own repository. There are tools to manage dependencies, even when building everything from source.

Thank you for good advice, will correct this.

from abb_libegm.

In addition to @gavanderhoorn suggestions, have you made any modifications to the StateMachine Add-In's RAPID code? And have you tested what happens if you run your ROS2 implementation against a virtual controller in RobotStudio?

The StateMachine Add-In's have not been modified. Have not yet tested against a virtual robot in RobotStudio due to issues with RWS calls towards virtual robots on windows machines.

Just for testing, then you can manually set any desired EGM parameter values in the RAPID code and then manually trigger the IO-signal to start EGM communication (i.e. the same signal that RWS sets). This can for example be done by using the IO simulator in RobotStudio:

Otherwise, one way around the virtual controller blocking RWS commands is to set up a proxy server on your Windows machine.

These folders contain wireshark traces and event logs for runs with respectively Dashing and Eloquent.

eloquent.zip
dashing.zip

It seems that only the robot controller's Common event log is included in your zip-files. That log is often too general, and I was interested in all logs.

from abb_libegm.

It seems that only the robot controller's Common event log is included in your zip-files. That log is often too general, and I was interested in all logs.

Apologies, my mistake. These zips contain all log files.
dashing_all_logs.zip
eloquent_all_logs.zip

Just for testing, then you can manually set any desired EGM parameter values in the RAPID code and then manually trigger the IO-signal to start EGM communication (i.e. the same signal that RWS sets). This can for example be done by using the IO simulator in RobotStudio:

Ok, will test the Eloquent version against a virtual YuMi in RobotStudio and get back to you.

from abb_libegm.

Something we seem to be ignoring here: the ros2_control and related packages used by @Mariunil are community contributed / privately adapted versions and appear to be very early ports of some basic functionality. These ros2_control packages has seem some updates to work again in Eloquent. Those changes are also very new.

Looking at upstream ros2_control (here) the updates can sometimes be quite invasive. I haven't checked the specific packages @Mariunil is using, but they would probably need similar changes.

Stating that "the only thing" that changed was Dashing to Eloquent is ignoring all of this. It could very well be there are threading, locking or other problems introduced in those updates.

As abb_libegm is the only thing that hasn't changed here (although @Mariunil seems to be using a port to ROS 2 by Lars and some other ppl, possibly with his own changes added) my own workflow would be to check the surrounding infrastructure first.

@Mariunil: it would be good if you could clarify which packages you are using, where they come from (if: forks, custom adaptations or official upstreams), which versions they are at and perhaps also provide diffs from official versions to the custom ones.

As you've copied things around a lot (I get that impression from your repositories at least) this may not be that easy any more.

I'm also not asking you to do all of this immediately. It would be good to first realise this is the situation @Mariunil is in. Getting a high-level overview of which packages he's using -- and in which state those are -- would be good though.

from abb_libegm.

Apologies, my mistake. These zips contain all log files.
dashing_all_logs.zip
eloquent_all_logs.zip

No worries! 😃

The only thing that stood out in the logs, as far as I could see, is that the robot controller complains about missing AbsAcc data (e.g. Absolute accuracy is activated but no data exists for robot ROB_L). Have you experienced any issues when moving the robot with RAPID commands (e.g. MoveAbsJ or MoveL)?

As abb_libegm is the only thing that hasn't changed here (although @Mariunil seems to be using a port to ROS 2 by Lars and some other ppl, possibly with his own changes added) my own workflow would be to check the surrounding infrastructure first.

It might also be reasonable to just run some EGM demo motions from a minimal node. E.g. I tested this (with the master branch) to see if the EGM communication channels works as expected:

#include "rclcpp/rclcpp.hpp"
#include "abb_libegm/egm_controller_interface.h"

int main(int argc, char **argv)
{
  rclcpp::init(argc, argv);

  // Boost components for managing asynchronous UDP socket(s).
  boost::asio::io_service io_service;
  boost::thread_group thread_group;

  // Create a configuration to make the EGM interfaces use demo outputs (sinusoidal motions).
  abb::egm::BaseConfiguration configuration;
  configuration.use_demo_outputs = true;

  // Create EGM interfaces, and if using StateMachine Add-In (with a YuMi robot) then:
  // - Left arm = port 6511.
  // - Right arm = port 6512.
  abb::egm::EGMControllerInterface egm_interface_l(io_service, 6511, configuration);
  abb::egm::EGMControllerInterface egm_interface_r(io_service, 6512, configuration);

  // Spin up a background thread for managing the EGM interfaces.
  thread_group.create_thread(boost::bind(&boost::asio::io_service::run, &io_service));

  // Spin up a node.
  rclcpp::spin(rclcpp::Node::make_shared("minimal_egm_demo"));

  // Stop the background thread.
  io_service.stop();
  thread_group.join_all();

  return 0;
}

The line configuration.use_demo_outputs = true; makes the EGM interfaces to send sinusoidal motion references, as soon as the EGM communication channels have been established.

Please note that I recommend, for a YuMi robot, to use the joint demo motions (sinus references, for each joint, with a magnitude of 2 degrees). This is because the Cartesian demo motions requires more care with the initial starting position.

Here's a video of the joint demo motions (RobotStudio EGM demo motions video.zip).

from abb_libegm.

We found the cause of the robot controller shutdown to be a "Nan" being sent to robot controller, likely due to a bug in our code. This happened in Eloquent but somehow not in Dashing.

Eloquent

Dashing

Crashes stopped when we changed the NaN value to be a valid number. The robot controller seems to not be able to handle invalid joint commands.

We tested the faulty commands on a virtual robot in RobotStudio as well, which suffered the same issues as the physical robot.
Screenshot: Robot controller stopped during execution

Screenshot: Robot controller unable to start up again after warm-restart.

from abb_libegm.

Nice find.

@jontje: should not be too difficult to add a check to the outgoing joint position values before sending them out, correct?

from abb_libegm.

Something we seem to be ignoring here: the ros2_control and related packages used by @Mariunil are community contributed / privately adapted versions and appear to be very early ports of some basic functionality. These ros2_control packages has seem some updates to work again in Eloquent. Those changes are also very new.

Looking at upstream ros2_control (here) the updates can sometimes be quite invasive. I haven't checked the specific packages @Mariunil is using, but they would probably need similar changes.

(@gavanderhoorn )
Ok, I suspect our bug is caused by us using the older version. We will look into our dependencies and move to the latest versions.

The only thing that stood out in the logs, as far as I could see, is that the robot controller complains about missing AbsAcc data (e.g. Absolute accuracy is activated but no data exists for robot ROB_L). Have you experienced any issues when moving the robot with RAPID commands (e.g. MoveAbsJ or MoveL)?

(@jontje)
Have not tested the RAPID commands. We noticed the absacc file to be missing a while back, but have not been able to do anything about it yet.

from abb_libegm.

We found the cause of the robot controller shutdown to be a "Nan" being sent to robot controller, likely due to a bug in our code. This happened in Eloquent but somehow not in Dashing.

Great that you managed to figure out the cause!

@jontje: should not be too difficult to add a check to the outgoing joint position values before sending them out, correct?

Yes, it should be straight-forward to add it to the methods here and here.

I will open an issue for it.

(@jontje)
Have not tested the RAPID commands. We noticed the absacc file to be missing a while back, but have not been able to do anything about it yet.

AbsAcc is important to ensure +- 1 mm accuracy of the TCP, so it might be worth to fix it if you experience any problems with that. And it's probably easiest to just contact your local ABB Robotics sales organization to get help with that.

from abb_libegm.

As this appears to not be an issue with the library in this repository, but a problem in user code, I'm closing this.

We're tracking the enhancement to check outgoing commands in #69.

@Mariunil: feel free to keep commenting on the issue of course.

from abb_libegm.