Comments (15)
Here's a video of the joint demo motions (RobotStudio EGM demo motions video.zip).
A ๐บ YuMi. Nice.
from abb_libegm.
I'm not saying this is the cause, but the title of your issue claims:
EGM causes robot controller shutdown
while later you write:
A fully functioning system was achieved with ROS2
Dashing
. However, when transitioning to ROS2Eloquent
, the same code crashes the robot controller.
These two don't match.
Unless @jontje recognises this immediately, I would suggest to add more details about your code, the way you are wrapping it with pieces of ROS 2 infrastructure, perhaps even a wireshark trace for both the working and the non-working situations, information on how you installed ROS 2, etc.
Is the same PC running Dashing and Eloquent? Anything else that changed between the two?
from abb_libegm.
Unless @jontje recognises this immediately
I haven't seen EGM cause such a behaviour before, at least as far as I can remember.
A fully functioning system was achieved with ROS2
Dashing
. However, when transitioning to ROS2Eloquent
, the same code crashes the robot controller.
I have only tested abb_libegm
in ROS2 Dashing
against RobotStudio simulations before, which worked fine. I haven't had time to try out ROS2 Eloquent
yet, but I must say I find it a bit that strange that only a change in ROS2 distribution would cause the crashes. E.g. since the robot controller doesn't "know" about what the external computer is running.
When a motion command is issued (joint position reference) to the robot controller, it activates the joint brakes on both arms and freezes for a couple of seconds before crashing (FlexPendant reboots etc). It then automatically reboots, but gets stuck trying to connect to the robot controller. When manually powered down and restarted, the FlexPendant informs us the robot controller is in a System Failure state.
@Mariunil, can you provide the robot controller's event logs? E.g. in RobotStudio:
And check All
logs when saving:
I would suggest to add more details about your code, the way you are wrapping it with pieces of ROS 2 infrastructure, perhaps even a wireshark trace for both the working and the non-working situations, information on how you installed ROS 2, etc.
Is the same PC running Dashing and Eloquent? Anything else that changed between the two?
In addition to @gavanderhoorn suggestions, have you made any modifications to the StateMachine Add-In
's RAPID code? And have you tested what happens if you run your ROS2 implementation against a virtual controller in RobotStudio?
from abb_libegm.
Looks like it's Mariunil/abb_yumi and/or yumi-crew/yumi.
@Mariunil: random suggestion, but: I would avoid copying entire repositories into your own repository. There are tools to manage dependencies, even when building everything from source.
from abb_libegm.
I'm not saying this is the cause, but the title of your issue claims:
EGM causes robot controller shutdown
while later you write:
A fully functioning system was achieved with ROS2 Dashing. However, when transitioning to ROS2 Eloquent, the same code crashes the robot controller.
These two don't match.
The robot controller crashes when a EGM motion command is issued through abb_libegm. As the robot controller and the desktop PC is communicating via the EGM Sensor protocol (handled by abb_libegm), the robot is unaware of the ROS 2 Eloquent system running on the desktop computer. It is only aware of the EGM server spawned by abb_libegm (and the RWS client spawned by abb_librws, but no issues occur here).
The ROS 2 code is organized in the ros2_control framework with a simple joint_position_controller (handled by the controller manager).
ROS 2 dashing and eloquent were both installed following instruction from: https://index.ros.org/doc/ros2/Installation/Dashing/Linux-Install-Debians/ and https://index.ros.org/doc/ros2/Installation/Eloquent/Linux-Install-Debians/
abb_libegm dependency versions:
- Protobuf: 3.6.1
- Boost: 1.65.1.0
As the robot controller is only communicating with abb_libegm resources, and crashing exclusively when issued motion commands, we experience this issue as a abb_libegm issue, and not a ROS2 Eloquent issue. The abb_libegm methods are utilized in the robot's hardware interface in ROS 2.
The same computer was running Dashing and Eloquent. The same behavior have also occurred for two different computers.
I have only tested abb_libegm in ROS2 Dashing against RobotStudio simulations before, which worked fine. I haven't had time to try out ROS2 Eloquent yet, but I must say I find it a bit that strange that only a change in ROS2 distribution would cause the crashes. E.g. since the robot controller doesn't "know" about what the external computer is running.
Agree, we are unable to see how ROS 2 Eloquent can be able to affect the EGM communication, however it seem to be doing just that. We suspect we somehow are sending faulty messages to the robot.
In addition to @gavanderhoorn suggestions, have you made any modifications to the StateMachine Add-In's RAPID code? And have you tested what happens if you run your ROS2 implementation against a virtual controller in RobotStudio?
The StateMachine Add-In's have not been modified. Have not yet tested against a virtual robot in RobotStudio due to issues with RWS calls towards virtual robots on windows machines.
These folders contain wireshark traces and event logs for runs with respectively Dashing and Eloquent.
from abb_libegm.
Looks like it's Mariunil/abb_yumi and/or yumi-crew/yumi.
The code in those repositories are very similar, however it is the code from yumi-crew/yumi that are being executed.
@Mariunil: random suggestion, but: I would avoid copying entire repositories into your own repository. There are tools to manage dependencies, even when building everything from source.
Thank you for good advice, will correct this.
from abb_libegm.
In addition to @gavanderhoorn suggestions, have you made any modifications to the StateMachine Add-In's RAPID code? And have you tested what happens if you run your ROS2 implementation against a virtual controller in RobotStudio?
The StateMachine Add-In's have not been modified. Have not yet tested against a virtual robot in RobotStudio due to issues with RWS calls towards virtual robots on windows machines.
Just for testing, then you can manually set any desired EGM parameter values in the RAPID code and then manually trigger the IO-signal to start EGM communication (i.e. the same signal that RWS sets). This can for example be done by using the IO simulator in RobotStudio:
Otherwise, one way around the virtual controller blocking RWS commands is to set up a proxy server on your Windows machine.
These folders contain wireshark traces and event logs for runs with respectively Dashing and Eloquent.
It seems that only the robot controller's Common
event log is included in your zip-files. That log is often too general, and I was interested in all logs.
from abb_libegm.
It seems that only the robot controller's Common event log is included in your zip-files. That log is often too general, and I was interested in all logs.
Apologies, my mistake. These zips contain all log files.
dashing_all_logs.zip
eloquent_all_logs.zip
Just for testing, then you can manually set any desired EGM parameter values in the RAPID code and then manually trigger the IO-signal to start EGM communication (i.e. the same signal that RWS sets). This can for example be done by using the IO simulator in RobotStudio:
Ok, will test the Eloquent version against a virtual YuMi in RobotStudio and get back to you.
from abb_libegm.
Something we seem to be ignoring here: the ros2_control
and related packages used by @Mariunil are community contributed / privately adapted versions and appear to be very early ports of some basic functionality. These ros2_control
packages has seem some updates to work again in Eloquent. Those changes are also very new.
Looking at upstream ros2_control
(here) the updates can sometimes be quite invasive. I haven't checked the specific packages @Mariunil is using, but they would probably need similar changes.
Stating that "the only thing" that changed was Dashing to Eloquent is ignoring all of this. It could very well be there are threading, locking or other problems introduced in those updates.
As abb_libegm
is the only thing that hasn't changed here (although @Mariunil seems to be using a port to ROS 2 by Lars and some other ppl, possibly with his own changes added) my own workflow would be to check the surrounding infrastructure first.
@Mariunil: it would be good if you could clarify which packages you are using, where they come from (if: forks, custom adaptations or official upstreams), which versions they are at and perhaps also provide diffs from official versions to the custom ones.
As you've copied things around a lot (I get that impression from your repositories at least) this may not be that easy any more.
I'm also not asking you to do all of this immediately. It would be good to first realise this is the situation @Mariunil is in. Getting a high-level overview of which packages he's using -- and in which state those are -- would be good though.
from abb_libegm.
Apologies, my mistake. These zips contain all log files.
dashing_all_logs.zip
eloquent_all_logs.zip
No worries! ๐
The only thing that stood out in the logs, as far as I could see, is that the robot controller complains about missing AbsAcc data (e.g. Absolute accuracy is activated but no data exists for robot ROB_L
). Have you experienced any issues when moving the robot with RAPID commands (e.g. MoveAbsJ
or MoveL
)?
As
abb_libegm
is the only thing that hasn't changed here (although @Mariunil seems to be using a port to ROS 2 by Lars and some other ppl, possibly with his own changes added) my own workflow would be to check the surrounding infrastructure first.
It might also be reasonable to just run some EGM demo motions from a minimal node. E.g. I tested this (with the master branch) to see if the EGM communication channels works as expected:
#include "rclcpp/rclcpp.hpp"
#include "abb_libegm/egm_controller_interface.h"
int main(int argc, char **argv)
{
rclcpp::init(argc, argv);
// Boost components for managing asynchronous UDP socket(s).
boost::asio::io_service io_service;
boost::thread_group thread_group;
// Create a configuration to make the EGM interfaces use demo outputs (sinusoidal motions).
abb::egm::BaseConfiguration configuration;
configuration.use_demo_outputs = true;
// Create EGM interfaces, and if using StateMachine Add-In (with a YuMi robot) then:
// - Left arm = port 6511.
// - Right arm = port 6512.
abb::egm::EGMControllerInterface egm_interface_l(io_service, 6511, configuration);
abb::egm::EGMControllerInterface egm_interface_r(io_service, 6512, configuration);
// Spin up a background thread for managing the EGM interfaces.
thread_group.create_thread(boost::bind(&boost::asio::io_service::run, &io_service));
// Spin up a node.
rclcpp::spin(rclcpp::Node::make_shared("minimal_egm_demo"));
// Stop the background thread.
io_service.stop();
thread_group.join_all();
return 0;
}
The line configuration.use_demo_outputs = true;
makes the EGM interfaces to send sinusoidal motion references, as soon as the EGM communication channels have been established.
Please note that I recommend, for a YuMi robot, to use the joint demo motions (sinus references, for each joint, with a magnitude of 2 degrees). This is because the Cartesian demo motions requires more care with the initial starting position.
Here's a video of the joint demo motions (RobotStudio EGM demo motions video.zip).
from abb_libegm.
We found the cause of the robot controller shutdown to be a "Nan" being sent to robot controller, likely due to a bug in our code. This happened in Eloquent but somehow not in Dashing.
Crashes stopped when we changed the NaN value to be a valid number. The robot controller seems to not be able to handle invalid joint commands.
We tested the faulty commands on a virtual robot in RobotStudio as well, which suffered the same issues as the physical robot.
Screenshot: Robot controller stopped during execution
Screenshot: Robot controller unable to start up again after warm-restart.
from abb_libegm.
Nice find.
@jontje: should not be too difficult to add a check to the outgoing joint position values before sending them out, correct?
from abb_libegm.
Something we seem to be ignoring here: the ros2_control and related packages used by @Mariunil are community contributed / privately adapted versions and appear to be very early ports of some basic functionality. These ros2_control packages has seem some updates to work again in Eloquent. Those changes are also very new.
Looking at upstream ros2_control (here) the updates can sometimes be quite invasive. I haven't checked the specific packages @Mariunil is using, but they would probably need similar changes.
(@gavanderhoorn )
Ok, I suspect our bug is caused by us using the older version. We will look into our dependencies and move to the latest versions.
The only thing that stood out in the logs, as far as I could see, is that the robot controller complains about missing AbsAcc data (e.g. Absolute accuracy is activated but no data exists for robot ROB_L). Have you experienced any issues when moving the robot with RAPID commands (e.g. MoveAbsJ or MoveL)?
(@jontje)
Have not tested the RAPID commands. We noticed the absacc file to be missing a while back, but have not been able to do anything about it yet.
from abb_libegm.
We found the cause of the robot controller shutdown to be a "Nan" being sent to robot controller, likely due to a bug in our code. This happened in Eloquent but somehow not in Dashing.
Great that you managed to figure out the cause!
@jontje: should not be too difficult to add a check to the outgoing joint position values before sending them out, correct?
Yes, it should be straight-forward to add it to the methods here and here.
I will open an issue for it.
(@jontje)
Have not tested the RAPID commands. We noticed the absacc file to be missing a while back, but have not been able to do anything about it yet.
AbsAcc is important to ensure +- 1 mm accuracy of the TCP, so it might be worth to fix it if you experience any problems with that. And it's probably easiest to just contact your local ABB Robotics sales organization to get help with that.
from abb_libegm.
As this appears to not be an issue with the library in this repository, but a problem in user code, I'm closing this.
We're tracking the enhancement to check outgoing commands in #69.
@Mariunil: feel free to keep commenting on the issue of course.
from abb_libegm.
Related Issues (20)
- Wait for an EGM communication session to start... HOT 5
- Setting up EGM sessions for abb_libegm_samples HOT 4
- Rapid code corresponding to abb_libegm HOT 6
- What does the duration really mean? HOT 2
- EGMTrajectoryInterface::addTrajectory fails inconsistently HOT 1
- Why not add a path correction module๏ผ HOT 1
- RobotWare 6.13 EGM UdpUc error HOT 17
- Question: Can I use this library on Windows for testing with a simulated robot?
- Regulate robot speed - Robot too slow during the executions with EGM HOT 1
- Possibility about apply this library in Matlab
- Support for Four Axis SCARA Arms (Eg IRB910SC)
- Append multiple trajectories creates stop during motion between trajectories
- Problem when running `start_egm_joint`
- Example for controlling GoFa 15000 with C++ in Ubuntu
- State Machine add-in is not available any more in the RoboApps ( Robot Studio ) HOT 4
- Problems with compiling and using the abb_libegm library on Windows HOT 2
- Execution speed and EGM parameter documentation
- Error 41830 when try to use EGM Position Guidance Joint mode with an external axis rail
- I cant use it to control the pose ,but i can control the joint,what's wrong with me
- Impossible to use feed_forward velocity control alone - Gofa robot
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abb_libegm.