usdot-fhwa-stol / carma-platform Goto Github PK

CARMA Platform is built on robot operating system (ROS) and utilizes open source software (OSS) that enables Cooperative Driving Automation (CDA) features to allow Automated Driving Systems to interact and cooperate with infrastructure and other vehicles through communication. Doxygen Source Code Documentation :

Home Page: https://usdot-fhwa-stol.github.io/documentation/carma-platform/

CMake 2.89% C++ 86.09% Shell 3.09% Python 7.83% Dockerfile 0.06% C 0.03%

ros cpp cooperative-driving-automation automated-vehicles autonomous-driving autonomous-vehicles self-driving open-source

carma-platform's Introduction

Develop CI/CD Processes

DockerHub Image Build	Unit Tests Workflow	Sonar Code Quality

CARMA

CARMA^SM advances research and development to accelerate market readiness and deployment of cooperative driving automation, while advancing automated driving technology safety, security, data, and artificial intelligence. It encourages collaboration and participation by a community of engineers and researchers to advance understanding of cooperative driving automation using open source software (OSS) and agile project management practices.

CARMA is a reusable, extensible platform for controlling SAE level 2 connected, automated vehicles (AVs). It provides a rich, generic API for third party plugins that implement vehicle guidance algorithms to plan vehicle trajectories. It is written in C++ and runs in a Robot Operating System (ROS) environment on Ubuntu. The platform can be reused on a variety of properly equipped vehicles. Migration has begun from the ROS 1 framework to ROS 2.

This readme updated: October 23, 2020

What Is CARMA Platform

Managing automated vehicle motion involves three aspects. The first is navigation, also known as localization, which is the act of determining where the vehicle currently is with respect to the earth and with respect to the desired path of travel (its planned route). The second is guidance, also known as trajectory planning, which includes the processes of determining how the vehicle is to move from its current location to its destination. The destination and route will be handed to the guidance algorithms, and they then determine how the vehicle’s motion needs to be adjusted at any time in order to follow the route. The third aspect of automated vehicle motion is control, which covers the actuation of the vehicle’s physical devices to induce changes in motion (for land vehicles these are typically causing the wheels to rotate faster or slower and turning the steering wheel). Therefore, the navigation solution becomes an input to the guidance function, and the guidance solution becomes an input to the control function. As the vehicle moves, obviously its location changes so that the navigation function constantly needs to update its solution and the cycle iterates as quickly as necessary to produce a smooth and accurate vehicle motion. The rate of iteration is largely determined by the expected speed of the vehicle.

CARMA Platform provides the navigation and guidance functions for its host vehicle, as well as some of the control functions. It depends on low level controller hardware to provide the rest of the control function. The current version of CARMA Platform provides SAE level 3+ autonomy, with both speed and steering control.

Documentation

Documentation of the setup, operation, and design of the CARMA Platform can be found on the project Confluence pages. Additional documentation can be found on the Doxygen Source Code Documentation page.

Release Notes

The current version and release history of the CARMA software platform: CARMA Release Notes

Repo Structure Note: The master and develop branches of this repo (and all other repos in the usdot-fhwa-stol GitHub organization) now reflects the third generation of CARMA code and documentation, called CARMA3. It uses Autoware to provide SAE level 2 and 3 automation capability. The legacy CARMA2 code is still being supported. It was used extensively by FHWA during 2018 for SAE level 1 experiments. To work with that code base, please check out the CARMA2-integration branch and make pull requests to it (using the contribution process outlined below).

ROS Version Note: Release 3.11 was the final version of CARMA Platform to contain only ROS1 code. The develop branches and future versions will contain a mix of ROS1 and ROS2 code in a mixed system until all code has been fully ported to ROS2. You can learn more about this system and the ROS2 migration efforts on confluence here: CARMA Platform ROS2 Docs

Roadmap

The current CARMA ecosystem development direction and release plans can be found here: CARMA Roadmap

Architecture Guide

The documentation describes the software architecture within a single CARMA vehicle. There may be several of these vehicles operating in concert, communicating with each other via DSRC or cellular means. They can also communicate with roadside infrastructure using DSRC or cellular. The communication among software components within a single vehicle, however, is mostly done via the Robot Operating System (ROS) framework. There are no ROS communications between neighboring vehicles. CARMA3 System Architecture

For information on the CARMA2 architecture, please see CARMA2 System Architecture

Detailed Design Documents

Please see the Detail Design document page for detailed design specification for CARMA and other additional information. Detail Design Page

Developers Guide

Useful development documentation and instructions for the setup of a development environment for CARMA Platform can be found at the following pages:

A plug-in developers guide for CARMA3 will be coming soon.

For a description of the approach to develop a plug-in for CARMA2, please see CARMA 2.7 Developers Guide

Users Guide

A user guide for CARMA3 will be coming soon.

For a description of the functionality specific to CARMA2 on the Cadillac SRX, see CARMA User Guide

Administrator Guide

An administrator guide for CARMA3 will be coming soon.

For administrative information on CARMA2, including vehicle and developer PC configuration, build, deployment and testing steps necessary to install and run CARMA Platform, please see Administrator Guide

Other CARMA Packages

CARMA Platform^SM is a downloadable, open source software (OSS) platform architected to be extensible and reusable for a wide variety of research purposes to advance innovation for cooperative driving automation. It enables communication between vehicles, road users such as pedestrians, bicyclists, and scooters, and infrastructure devices capable of communication. It promotes collaboration between a community of engineers and researchers to accelerate the development, testing, and evaluation of cooperative driving automation while advancing the safety, security, data, and use of artificial intelligence in automated driving technology.

The CARMA Platform is distributed as a set of multiple independent packages hosted in separate Github repositories. These packages facilitate operation of the CARMA Platform with different hardware configurations or allow it to support different modes of operation. To include one of these packages in your build of the CARMA Platform system please clone the Github repository into the same Catkin workspace src/ folder as this repository. The Catkin build system will verify that dependencies are resolved appropriately and build the newly included package when you next run catkin_make. An incomplete listing of available packages for CARMA3 includes:

Vehicle Controller Interface Drivers

carma-ssc-interface-wrapper
dataspeed_controller_driver (Externally managed)
dataspeed_can_driver (Externally managed)

Sensor Drivers

carma-cohda-dsrc-driver
carma-velodyne-lidar-driver
avt_vimba_camera
carma-delphi-srr2-driver
novatel_gps_driver
gstreamer_camera_driver (Externally managed)
ouster_lidar_driver (Externally managed)
inertiallabs_gnss_driver (Externally managed)

General System Utilites

Vehicle Configurations

carma-config

A full list of available packages may be found at in the USDOT FHWA STOL Github organization.

CARMA Cloud^SM is a downloadable, cloud-based open source software (OSS) service that provides information to support and enable cooperative driving automation. It enables communication with cloud services and vehicles, road users such as pedestrians, bicyclists, and scooters, and infrastructure devices capable of communication. It promotes collaboration between a community of engineers and researchers to accelerate the development, testing, and evaluation of cooperative driving automation while advancing the safety, security, data, and use of artificial intelligence in automated driving technology. The source code for CARMA Cloud can be found here on GitHub.

Contribution

Welcome to the CARMA contributing guide. Please read this guide to learn about our development process, how to propose pull requests and improvements, and how to build and test your changes to this project. CARMA Contributing Guide

Code of Conduct

Please read our CARMA Code of Conduct which outlines our expectations for participants within the CARMA community, as well as steps to reporting unacceptable behavior. We are committed to providing a welcoming and inspiring community for all and expect our code of conduct to be honored. Anyone who violates this code of conduct may be banned from the community.

Attribution

The development team would like to acknowledge the people who have made direct contributions to the design and code in this repository. CARMA Attribution

License

By contributing to the Federal Highway Administration (FHWA) Connected Automated Research Mobility Applications (CARMA), you agree that your contributions will be licensed under its Apache License 2.0 license. CARMA License

Code.gov Registration Info

Agency: DOT

Short Description: Cooperative Automation Research Mobility Applications (CARMA) is an open source software (OSS) platform that runs on a computer inside of a vehicle, and the computer interacts with the vehicle's devices to enable cooperative automated vehicle research, development, and testing. The computer then gives the vehicle the information needed in order to interact and cooperate with infrastructure and other vehicles.Phase I of this project initiated the next generation of the CARMA fleet by developing a common platform infrastructure that will support the FHWA's strategic vision for Cooperative Automated Driving Systems (CADS). Phase I also developed the Integrated Highway Prototype (IHP) by integrating the existing applications and a new vehicle type, the heavy truck, into the CARMA fleet. The following five applications are major research areas: (1) Vehicle Platooning, (2) Signalized Intersection Approach and Departure, (3) Infrastructure guided Speed Harmonization, and (4) Cooperative Lane Change, and (5) Cooperative Ramp Merge.Phase II of this project will advance CADS, which was developed on the FHWA's CARMA research platform to allow testing of arbitration and negotiation strategies. Ultimately, this project will enhance infrastructure efficiency and reduce traffic congestion. One of the goals of this project is to accelerate the technology transfer and deployment of cooperative vehicle applications through open source collaboration with industry and stakeholders. This project will enable Society of Engineers (SAE) Level 2 and Level 3 (L2/3) light vehicle testing capabilities and develop a fleet of four SAE L2/3 heavy vehicles. The project will also develop a high-level concept for Transportation System Management and Operations (TSMO) around the use of automated driving technology. Furthermore, this project will identify Traffic Incident Management (TIM) scenarios that provide strategy improvements and new strategies for first responder use cases interacting with ADS and using ADS. The new FHWA fleet will be used to prototype a scenario for each of the following use case categories: (1) Basic Travel, (2) Work Zone, (3) Traffic Incident Management and (4) Weather. These tests will provide objective data to inform stakeholders about the impacts of ADS and possible strategies to use ADS to improve operations.

Status: Beta

Tags: transportation, cooperative driving automation, connected vehicles, intelligent transportation systems, C++, java, python, docker

Labor hours: 200

Contact Name: James Lieu

Contact Phone: (202) 366-3000

Contact

Please click on the CARMA logo below to visit the Federal Highway Adminstration(FHWA) CARMA website. For more information, contact [email protected].

Support

For technical support from the CARMA team, please contact the CARMA help desk at [email protected].

carma-platform's People

Stargazers

Watchers

Forkers

rvalienter90 perryyo armando170a qwert1234zx damonmin snehagn nadellian atgreenwood zhangaigh diffblue-benchmarks mgoli-cav usdot-fhwa-nrc sichitong michaleb vsn3876 majiaqi jackhallccta kfarokhi ub-cavas turen009 ksadabadi itabr v2x-dc shashankreddysunkara labimage trendingtechnology jilinzhou adam-mcc mdomox dexter575 helomair yomo11 jspring dzhan85 typekid brucealmighty-debug oaosman fyj97 darrell-dorman rafailagln abey-yoseph keshava ronzh lonfr lucianzhong bujanovic pardhuaj tlochrane sirvelo maidoufu harderthan cangfeng110 matthewji carlosdbarrera pirazor qqsskk zhuhaijun753 exu1977 pmusau17 thuns001 hctrd242 abanglzu snail333 adamlm-forks blockspacer linecode zichaotong ruyiren hlchen1043 zhuiyuehun helloparanoidandroid 378281820qq hitleon ros2outdoor neophack ovasens arunbalas liucsg vt-asim-lab inugi-source hiyx zhaohui-liang arseniy-sonar frankfanslc xianglunkai riponcse10 arslan-z soar-robotics jackautocq ricetea xin0808 shyding saikrishnabairamoni asymingt xuhan417 ucla-mobility xiaxin2000 codygarver donrv kbr-carma

carma-platform's Issues

Multi-Vehicle Platooning requires vehicles to engage in order

Current version of platooning plugin requires vehicle to engage one by one from the very first vehicle to the very last vehicle. The first two vehicle can engage in any order, but the third vehicle should engage when the first two vehicle have already formed a platoon (yellow light is on). The forth vehicle should wait to engage until the first three vehicle have already been in the same platoon.
The reason is that when a vehicle is searching for a platoon, it only cares if the rear position of any platoon is in front of the host vehicle, which means that it will not care about whether there is an object between the host vehicle and the front platoon.
In three vehicle case, if they started in random order, there will be a case where the third vehicle is joining the first vehicle. And the second vehicle is never able to join them because it is between the first and the third.
#827 is a fix for that, but the sensor fusion anomaly prevent it from working properly. (See #842)

Sensor fusion reports connected vehicle in a wrong lane

When we are doing two-vehicle platooning testing, the first vehicle is in the same lane of the second one broadcasting BSM. The seconds vehicle see one object in front with empty BSM id and see another vehicle with BSM in a different lane (lane 4). Here are logs file for the second vehicle:
F_silver_test1.txt
The log looks like this:

2018-05-08 07:30:43:873 | DEBUG | PlatooningPlugin | NONE | Found obstacle in lane 0 and its dtd is 226.15253250848778
2018-05-08 07:30:43:874 | DEBUG | PlatooningPlugin | NONE | The found obstacle type is MessageImpl<cav_msgs/ConnectedVehicleType>
2018-05-08 07:30:43:874 | DEBUG | PlatooningPlugin | NONE | The BSM id is of size 0
2018-05-08 07:30:43:874 | DEBUG | PlatooningPlugin | NONE | Found obstacle in lane 4 and its dtd is 187.2083069788181
2018-05-08 07:30:43:874 | DEBUG | PlatooningPlugin | NONE | The found obstacle type is MessageImpl<cav_msgs/ConnectedVehicleType>
2018-05-08 07:30:43:874 | DEBUG | PlatooningPlugin | NONE | The BSM id is of size 4
2018-05-08 07:30:43:874 | DEBUG | PlatooningPlugin | NONE | The content at 0 is 73
2018-05-08 07:30:43:875 | DEBUG | PlatooningPlugin | NONE | The content at 1 is 54
2018-05-08 07:30:43:875 | DEBUG | PlatooningPlugin | NONE | The content at 2 is -59
2018-05-08 07:30:43:875 | DEBUG | PlatooningPlugin | NONE | The content at 3 is -70

I confirmed that the reported BSM ID in lane 4 is what the first vehicle is boardcasting.

Display of system alert on startup

When trying to start Carma this morning we ran into several instances of the UI informing "unable to find SYSTEM READY message" or something similar. The root cause was the pinpoint was in a fault status, but required looking through logs to determine. It would be nice to have the UI display the content from the SYSTEM_ALERT message when one is received to make diagnosis quicker.

CTE consequences should be handled by Tracking

Route computes cross track error, which is appropriate, but it should not be initiating system shutdown based on the value. Guidance.Tracking is where decisions are made about whether the vehicle is too far off plan. All of this decision logic should be consolidated in this one class.

Need to filter actual speed data

The lane change code, and soon the platooning code, will make decisions based on small differences in host vehicle speed and that of neighbor vehicles. We have lots of experience from Glidepath that measured speed data at 10 Hz contains enough noise to throw these subtle calculations into unrecoverable chaos. A large part of the solution there was to implement a sophisticated data filter to smooth out not only the speed but the acceleration derived from it. The hybrid polynomial/Holoborodko filter used there provides good filtering at that frequency for both the data stream (speed) and its derivative. This algorithm, or something similar, needs to be adopted in the Carma code base.

Speedharm Plugin should not throw Fatal when SpeedHarm Server is unresponsive

While testing on local VM, got this Fatal Error when :

carma UI running and engaged while speed harm server running.
speed harm is receiving vehiclecommands and carma UI receiving SH commands.
stopped speed harm server (which is similar to server not being available during a run)
On the UI, driver disengages and hits continue. Then carma throws a fatal error.

Expected behavior should be to make the plugin unavailable and continue with the other plugins.

2018-05-03 16:59:58:768 | INFO | PluginLifecycleHandler | PLUGIN | Suspending Cruising Plugin:1.0.2
2018-05-03 16:59:58:774 | DEBUG | YieldPlugin | NONE | Availability Listeners 1
2018-05-03 16:59:58:777 | INFO | PluginLifecycleHandler | PLUGIN | Suspending Yield Plugin:1.0.1
2018-05-03 16:59:58:777 | INFO | LightBarManager | NONE | Set light bar for Light Bar State Machine for LightBarIndicator GREEN with statusOFF
2018-05-03 16:59:58:778 | INFO | LightBarStateMachine | NONE | Event: GUIDANCE_DISENGAGED Prev State: ENGAGED New State: DISENGAGED
2018-05-03 16:59:58:778 | DEBUG | LightBarManager | NONE | Light Bar Manager is polling job queue.
2018-05-03 16:59:58:778 | FATAL | GuidanceMain | NONE | Guidance thread Speed Harmonization PluginDoSuspendPluginWorkerThread raised uncaught exception! Handling!!!
2018-05-03 16:59:58:778 | ERROR | GuidanceExceptionHandler | NONE | Global guidance exception handler caught an uncaught exception from thread: Speed Harmonization PluginDoSuspendPluginWorkerThread
org.springframework.web.client.ResourceAccessException: I/O error on DELETE request for "http://localhost:8081/rest/vehicles/15": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:728)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:671)
at org.springframework.web.client.RestTemplate.delete(RestTemplate.java:540)
at gov.dot.fhwa.saxton.carma.plugins.speedharm.SessionManager.endVehicleSession(SessionManager.java:81)
at gov.dot.fhwa.saxton.carma.plugins.speedharm.SpeedHarmonizationPlugin.onSuspend(SpeedHarmonizationPlugin.java:176)
at gov.dot.fhwa.saxton.carma.guidance.plugins.SuspendPluginTask.run(SuspendPluginTask.java:29)
at gov.dot.fhwa.saxton.carma.guidance.plugins.PluginLifecycleHandler$PluginWorker.run(PluginLifecycleHandler.java:299)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:714)
... 7 more
2018-05-03 16:59:58:779 | DEBUG | GuidanceStateMachine | GUIDANCE_STATE | Guidance state machine reveiced PANIC at state: DRIVERS_READY
2018-05-03 16:59:58:779 | DEBUG | GuidanceStateMachine | GUIDANCE_STATE | Guidance transited to state SHUTDOWN
2018-05-03 16:59:58:779 | DEBUG | GuidanceStateMachine | GUIDANCE_STATE | Guidance is taking action PANIC_SHUTDOWN
2018-05-03 16:59:58:782 | DEBUG | LaneChangePlugin | NONE | Availability Listeners 1
2018-05-03 16:59:58:784 | INFO | PluginLifecycleHandler | PLUGIN | Suspending Lane Change Plugin:1.0.2
2018-05-03 16:59:58:785 | INFO | YieldPlugin | NONE | Yield plugin suspended
2018-05-03 16:59:58:787 | DEBUG | RouteFollowingPlugin | NONE | Availability Listeners 1

WARNING: | CommandReceiver | NONE | Unable to wait for server speed command, received exception.
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8081/rest/commands/8": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:728)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:671)
at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:319)
at gov.dot.fhwa.saxton.carma.plugins.speedharm.CommandReceiver.run(CommandReceiver.java:53)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:714)
... 4 more

May 03, 2018 12:01:30 PM org.ros.internal.node.RosoutLogger info
INFO: | TrajectoryExecutor | NONE | Received RouteState. New downtrack distance: 0.8565578851848841
May 03, 2018 12:01:30 PM org.ros.internal.node.RosoutLogger info
INFO: | MessageConsumer | NONE | We encode BSM
May 03, 2018 12:01:30 PM org.ros.internal.node.RosoutLogger info
INFO: | Tracking | NONE | Current vehicle speed is -0.0117899441719055 m/s
May 03, 2018 12:01:30 PM org.ros.internal.node.RosoutLogger info
INFO: | Arbitrator | NONE | Received RouteState:MessageImpl<cav_msgs/RouteState>
May 03, 2018 12:01:31 PM org.ros.internal.node.RosoutLogger info
INFO: | TrajectoryExecutorWorker | NONE | Finished downtrack distance update with no trajectory.
May 03, 2018 12:01:31 PM org.ros.internal.node.RosoutLogger info
INFO: | PluginLifecycleHandler | PLUGIN | Suspending Speed Harmonization Plugin:1.0.0
May 03, 2018 12:01:31 PM org.ros.internal.node.RosoutLogger info
INFO: | PluginLifecycleHandler | PLUGIN | Suspending Yield Plugin:1.0.1
May 03, 2018 12:01:31 PM org.ros.internal.node.RosoutLogger fatal
SEVERE: | GuidanceMain | NONE | Guidance thread Speed Harmonization PluginDoSuspendPluginWorkerThread raised uncaught exception! Handling!!!
May 03, 2018 12:01:31 PM org.ros.internal.node.RosoutLogger info
INFO: | YieldPlugin | NONE | Yield plugin suspended
May 03, 2018 12:01:31 PM org.ros.internal.node.RosoutLogger error
SEVERE: | GuidanceExceptionHandler | NONE | Global guidance exception handler caught an uncaught exception from thread: Speed Harmonization PluginDoSuspendPluginWorkerThread
org.springframework.web.client.ResourceAccessException: I/O error on DELETE request for "http://localhost:8081/rest/vehicles/8": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:728)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:671)
at org.springframework.web.client.RestTemplate.delete(RestTemplate.java:540)
at gov.dot.fhwa.saxton.carma.plugins.speedharm.SessionManager.endVehicleSession(SessionManager.java:81)
at gov.dot.fhwa.saxton.carma.plugins.speedharm.SpeedHarmonizationPlugin.onSuspend(SpeedHarmonizationPlugin.java:176)
at gov.dot.fhwa.saxton.carma.guidance.plugins.SuspendPluginTask.run(SuspendPluginTask.java:29)
at gov.dot.fhwa.saxton.carma.guidance.plugins.PluginLifecycleHandler$PluginWorker.run(PluginLifecycleHandler.java:299)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:714)
... 7 more

Complex Replan Trigger Does Not Delete Itself

Complex replan triggers setup in Arbitrator are supposed to delete themselves from the trajectory progress callback list after being called, but this is not the case, leading to 1 invocation, then 2, then 3, then 4, escalating with each trajectory planned.

Performance impact is likely low, as callback is very small and not observed to get above 10-12 executions, and only done once per trajectory, but is notable issue.

Platooning Plugin depends on the same route file

Current implementation of platooning is letting each vehicle report their own downtrack distance. It would cause a failure in some cases:

If two vehicles are using two different route files, they will have different starting points for the downtrack distance.
If two vehicles are using the same route file, but one vehicle join the route before it is actually on any route segment, that vehicle will have a different starting point with the vehicle who join the route when it is already on the route.
In both cases, for the same location, they will report different dtd, which means they should not platoon together.
The current workaround is to avoid those two cases. But it should be fixed when the sensor fusion is functionally ready. See #842

Guidance silently refused to plan some trajectories

During I-95 test on NB 65 mph route, arbitrator stops to plan any trajectories after planning trajectory spanning [10420.896801400624, 10481.931957650624), which causes the vehicle never able to join the platoon. But the CARMA guidance is always ENGAGED.
guidance.txt

SpeedHarmonization server Linear algorithm behavior did not match expected behavior. (Aberdeen SpeedHarm Test)

When testing the Speed Harmonization Plugin at ATEF the Linear Speed Change Algorithm did not seem to change its behavior based on modifications to the algorithms.properties file located in the same folder as the jar. This made testing difficult but a workaround was found by creating a new algorithm which reduced the vehicles speed by 5mph.

Update license & contact info in package.xml

Our package.xml files still contain "todo"s and example info in various fields such as license info and maintainer contact info. We need to populate all of these with real info before community-wide distro in September 2018. Probably need to work w/Taylor to figure out appropriate maintainer contact info.

Platooning Gap is Abnormal

During I-95 testing, we observed abnormal gaps when we are testing 0.84 time headway at 65mph.
The gap between vehicles from front bumper to rear bumper should be around 20 meters. But from radar data, it seems that the gap drops to 5~10 meters at some time for the third and forth vehicle.

Route specification cannot handle edge case

The route specification cannot describe a lane change where the number of lanes does not change and the lane index does not change. (The vehicle is not aware of this lane change)

Speed Limit Counter not working

Per ATEF drivers, not counting up or down and just displaying 0.

Build system doesn't run gradle tests

catkin_make run_tests doesn't trigger the execution and collection of the tests normally run by the ./gradlew test command. This means CircleCI doesn't get a report on our test results.

#77 attempts to address this but is incomplete.

Ramp Metering - Stopping Issue

Takes minutes for "merge" vehicle to stop by coasting to 3MPH, and jerks until then.

Give TORC an email/call on controlling a vehicle to a stop. This is potentially a scope creep but try to address now if possible. Assign to Ed to contact Tammy R. to resolve this somehow and have Mike to followup.

Need a proper way to handle trajectory re-plan in complex maneuver

Currently, if arbitrator got a notification on trajectory failure when it is running a complex maneuver, the speed change from the complex maneuver to a regular longitudinal is not handled. It might result in a max_accel deceleration from the current speed.

Light bar not turning on for merging vehicle when in platoon

In TO26 I-95 testing, The light bar does not turn-on reliably for platooning after merge. This is because the RSU is sending commands to do a lane change as the route geometry does not match the road (rsu thinks vehicle is in the wrong lane). The requested steering command causes the lane change plugin to request the light bar as shown below. This prevents from platooning getting access till much later.

2018-06-25 13:32:35:074 | INFO | LightBarManager | NONE | Platooning Plugin failed to set the LightBarIndicator YELLOW as this was already controlled by Lane Change Plugin

Truck light bar is inverted

The light bar on the truck is using inverted commands. On means off and vise versa when making the set_lights service call. Additionally, the green lights do not behave in a reliable fashion. The green lights only sometimes turn on or off.

This was first found to be an issue during Aberdeen testing in July 2018.

Improve specificity of get_drivers_with_capabilities

Attempt to match the whole query string for capabilities
Allow wildcards for sections of the capability strings: comms/*/j2735/inbound_binary_msg
Increase specificity of inbound_binary_msg and outbound_binary_msg to include message set and carrying channel: dsrc/j2735/inbound_binary_msg

Plugin params do not have unique prefix

The ros params used by plugins are all assigned to the private guidance_main namespace. However, unless the plugin specifies a prefix like example_plugin, then the parameters could clash and be overwritten.

For example if a plugin has a timeout it will look like
/saxton_cav/guidance/guidance_main/timeout

but instead it we should make it
/saxton_cav/guidance/guidance_main/example_plugin/timeout

Clarification of SpeedAccel.msg

The first element in the indicated message spec is named "speed", which implies it is the current speed of something. Reading the header comment kinda implies it is intended to represent the commanded speed. Let's either expand the description of this field or rename it (e.g. speed_cmd) to clarify which it is.

Guidance pub sub needs message type validation for topics and services

Because the underlying rosjava subscription/publication object is shared between guidance components, it is possible for one component to cause a class cast exception in another if the two components expect different message types on the same topic. This generates a very cryptic error such as the one below.

java.lang.ClassCastException: org.ros.internal.message.$Proxy61 cannot be cast to cav_msgs.MobilityResponse
    at gov.dot.fhwa.saxton.carma.guidance.mobilityrouter.MobilityRouter.lambda$fireMobilityRequestCallback$0(MobilityRouter.java:196)
    at java.lang.Thread.run(Thread.java:748)

To avoid this the pub sub manager should cache all subscribed topic names, and assign a message type. This topic name should probably be fully defined to allow for namespaces. If a future user wishes to use a different message type a clearly described error should be thrown.

Truck CAN interface configuration depends on the order of CAN device initialization

Each driver needs to connect to a specific PC CAN interface. But truck CAN interface will be reordered if CAN devices initialized in a different order. For example if we unplug the radar USB PCAN, it will change the name of truck controller driver CAN interface name (from can1 to can0) and truck CAN driver (from can2 to can1).
Ideally, we should set up some udev rules on the computer to lock in the devices regardless of what order they are initialized.

Redistributable installer

We need an installer for Carma that is redistributable and not dependent on the presence of a development environment to copy from. The idea is to package all files needed for the vehicle PC onto a memory stick or similar, copy from there to the vehicle PC and run the installer to build the PC - no other computers online. Think InstallShield, although it doesn't have to be that pretty.

Guidance components have to implement logic for Guidance state machine themselves

Guidance components wishing to be triggered by guidance state events are responsible for implementing the onStateChange function themselves. This is strange given they also implement the called functions and every guidance component implements the function in basically the same way.
This implementation should be moved to a GuidanceComponent base class or similarly refactored out of the components themselves.

     */
    @Override
    public void onStateChange(GuidanceAction action) {
        log.debug("GUIDANCE_STATE", getComponentName() + " received action: " + action);
        switch (action) {
        case INTIALIZE:
            jobQueue.add(this::onSystemReady);
            break;
        case ACTIVATE:
            jobQueue.add(this::onActive);
            break;
        case DEACTIVATE:
            jobQueue.add(this::onDeactivate);
            break;
        case ENGAGE:
            jobQueue.add(this::onEngaged);
            break;
        case SHUTDOWN:
            jobQueue.add(this::onShutdown);
            break;
        case PANIC_SHUTDOWN:
            jobQueue.add(this::onPanic);
            break;
        case RESTART:
            jobQueue.add(this::onCleanRestart);
            break;
        default:
            throw new RosRuntimeException(
                    getComponentName() + "received unknown instruction from guidance state machine.");
        }
    }

Speed harm commands sometimes ignored on vehicle

During overnight testing on I-95 IHP the morning of 6/12, lead vehicle Green demonstrated at least one instance where it seemed to be ignoring incoming speed harm commands; it was receiving speed harm commands at ~22 m/s but GuidanceCommand was issuing commands to the controller at 29 m/s. Quick scan of the logs seemed to indicate that the trajectory planning was ignoring speed harm, as no complex maneuver appeared in the trajectory. Not clear why. One example is in green log 20180612-014837 at time 01:55:44.

Platooning Plugin Has Conflicts with Cooperative Merge Plugin

On I-95 testing, the complex maneuver planned by cooperative merge plugin is deleted by Platooning Plugin. The intention is to join a platoon after we finished merging. However, platooning plugin started to plan a new trajectory once it finished negotiation with the pass-by platoon, when it is still on the ramp (it should be under control of RSU).

Add manual switch for light bar (Aberdeen CR 10)

Drivers would like a manual toggle on the dashboard to activate the light bar to signal a hazardous situation. Its activation should be independent from anything going on in the software. Related to issue #9. 11/14/17 CCB: Need to deconflict this with software-controlled activation of the lights. Maybe yellow lights are manually controlled for safety, and only green light is used for automation indicators.

License display on UI

We should have a tab on the UI for license information. There are quite a few packages used in Carma, each with their own license info, most of which requires reproduction of certain clauses by the executable software. See CARMA OSADP repository for more license info.

Logging IO Exception

 2018-06-12 03:14:18:654 | SEVERE | Channel exception: java.io.IOException: Connection reset by peer 
 2018-06-12 03:14:18:659 | WARNING | GuidanceStateHandler | GUIDANCE_STATE | GuidanceStateHandlerreceived SHUTDOWN 
 2018-06-12 03:14:18:660 | INFO | GuidanceStateHandler | SHUTDOWN | Guidance state handler waiting5000ms for guidance thread shutdown... 
 2018-06-12 03:14:18:660 | INFO | GuidanceStateHandler | NONE | Publishing system_alert shutdown to ensure whole system shutdown 
 2018-06-12 03:14:18:663 | INFO | ManeuverInputs | NONE | Guidance.Maneuvers.ManeuverInputs shutting down normally. 
 2018-06-12 03:14:18:665 | INFO | GuidanceCommands | NONE | Guidance.Commands shutting down normally. 
 2018-06-12 03:14:18:671 | INFO | Tracking | NONE | Guidance.Tracking shutting down normally. 
 2018-06-12 03:14:18:673 | INFO | TrajectoryExecutor | NONE | Guidance.TrajectoryExecutor shutting down normally. 
 2018-06-12 03:14:18:688 | INFO | MobilityRouter | NONE | MobilityRouter shutting down normally. 
 2018-06-12 03:14:18:689 | INFO | LightBarManager | NONE | Light Bar Manager shutting down normally. 
 2018-06-12 03:14:18:689 | INFO | VehicleAwareness | NONE | VehicleAwareness shutting down normally. 
 2018-06-12 03:14:18:692 | INFO | I/O exception (java.net.ConnectException) caught when processing request: Connection refused (Connection refused) 
 2018-06-12 03:14:18:692 | INFO | Retrying request 
 2018-06-12 03:14:18:692 | INFO | Arbitrator | NONE | Guidance.Arbitrator shutting down normally. 
 2018-06-12 03:14:18:692 | INFO | I/O exception (java.net.ConnectException) caught when processing request: Connection refused (Connection refused) 
 2018-06-12 03:14:18:692 | INFO | Retrying request 
 2018-06-12 03:14:18:693 | INFO | I/O exception (java.net.ConnectException) caught when processing request: Connection refused (Connection refused) 
 2018-06-12 03:14:18:693 | INFO | Retrying request 
 2018-06-12 03:14:18:694 | SEVERE | PluginManager | NONE | Guidance.PluginManager is about to SHUTDOWN! 
 2018-06-12 03:14:18:695 | WARNING | GuidanceStateHandler | GUIDANCE_STATE | GuidanceStateHandlerreceived SHUTDOWN 
 2018-06-12 03:14:18:699 | INFO | I/O exception (java.net.ConnectException) caught when processing request: Connection refused (Connection refused)

This might be a new issue related to the new logging change. Need further investigate if it happens a lot.

Document startup procedure (Aberdeen CR 8)

Suggestion to provide a checklist of procedures on the tablet. This may be better served by handing out a printed QRC, since instructions may need to be changed frequently. 11/14/17 CCB: This is better as a document. Can have a hand-held sheet, but also make it a PDF on the tablet.

Platoon Vehicles - Inactive state

Notes from our Task Meeting 5/14/2018:

Inactive State - 220ms controller timeouts which may be caused by volume of messages. Operations message with command speeds. With BSMs and Path messages off, there was only 3 out of 4 that was inactive. Speed cmds are always sent at 100ms, regardless if the message is received or not. It takes 0.001s Message to Mobility. There is 0.3s delay from MobilityRouter to Platooning. It's a ROS delay.

TODO: Need to ask Tim to create a double lap route file with 4-5 vehicles next week.

Gap display

Tim Tiernan has requested that the UI always display the sensed gap if a forward vehicle is in view in host's lane (show distance in meters). Ideally, this would show just the track 1 data from the Cadillac radar, but if we can't easily get that isolated, then whatever data we have coming out of sensor fusion.

Mobility messages need way of conveying non-path point information

In the TO26 cooperative merge application an RSU broadcasts a mobility request with the meter point. This request message is incorrectly processed as a vehicle path by mobility router causing numerous replans on all vehicles. PR #930 adds a band-aid fix for this by preventing replans for single point paths, but this demonstrates a need for mobility requests to convey messages without an included path. Perhaps a flag could be added to disable the concept of a path in a request message. Additionally, other messages types should probably support some kind of single point location information.

Arbitrator does not support a replan request while planning

The arbitrator currently does not allow for plugins to perform a replan request while trajectory planning is in progress. If a plugin has already planned, but receives new information it has no way of forcing a replan at that point. The arbitrator should probably have a queue based list of future tasks so that when the trajectory is planned, re-planning can immediately began as expected by the plugin. This issue was discovered as one cause of Issue #905.

Relocate "Remote start" checkbox

The Remote Start checkbox on the UI startup page is right below the OK button, making it pretty easy to touch the wrong one. Since this checkbox is rarely used, I would prefer to see it moved way down to the bottom of the page, maybe in a corner.

Abrupt Stopping Issue at the End of Route

On 5/22/2018 ATEF Testing, Phil (Volpe) said him and the drivers are concerned about the issue when the vehicles just suddenly stop at the end of route, especially when there's a platoon behind them. This doesn't happen all the time but when it does, it increases the likelihood of an accident.

Attached are the 3 car platoon logs from today where Gray exhibited INACTIVITY and after re-engaging at the end of the route, it immediately stopped.

As a workaround, Tim said to split into two lanes towards the end of the route.

System shutdown generates NPE errors

It appears that most of the nodes, after they have signalled shutdown underway, generate an unhandled NPE in one of their onNewMessage methods.

YAML route names cannot include underscores (Aberdeen CR 29)

The YAML library doesn't allow underscore characters in names assigned inside the file (filenames themselves are okay). We need to include user instructions to clarify this requirement for people building route definitions. Alternatively, this may be another reason to switch to using CSV format for route files.

Improve tuning of ACC

The ACC system of carma could have it's stability improved. After, the speed controller and acc were tuned at Aberdeen on April 16th and 17th of 2018. The speed controller comfort was dramatically improved. ACC fixes were implemented in PR #764. ACC was brought to an acceptable level according to Tim. We were running at 45 and 60 mph.

However, their is still room for improvement. The system demonstrates large oscillations and some instability. This seems to mostly be due to a lag between guidance commands and the speed controller. This is likely due to the ACC system utilizing a PID on speed commands. This controller results in small changes in the target speed of the TORC speed controller. This leads to low errors and poor response time. The best fix would be moving the ACC system to wrench effort based commands. This would remove the stacked PID loops and allow for a better response and more effective tuning of the system.

Tracking did not work properly when it is tracking speed and downtrack distance

Tracking compares the current downtrack distance with a wrong number, which will show a wrong/large downtrack error. The same thing happened in tracking speed error. Something must be wrong in the logic of error tracking. Here is the log file:
20180214-145722.txt

Need for a clean shutdown of nodes on Sensor Fusion.

During verification test on 11/7/2017, we discovered both Guidance and Sensor Fusion wasn't cleanly shutting down.

No indication of E-Stop being pressed.

A GoPro camera fell from its ceiling mounted position and hit the E-Stop button. This resulted in a system shutdown when the controller reported a FAULT. However discovering the source of this issue was time consuming as no one has observed the button being pressed. Some indication of E-Stop compression needs to be available in the vehicle such as an LED indicator.

Truck CAN/controller driver interface is not the same as SRX drivers

Truck drivers and SRX drivers have different topic name for some output data. For example, truck controller subscribes to speed_accel topic but SRX drivers subscribes to cmd_speed topic for cmd_speed input.
Truck drivers and SRX drivers have different units on the same topic. For example, truck CAN reports current speed in km/h but SRX reports them in m/s.
There are more mismatch on unit and topic name. We need to verify all necessary topic to make sure they have the same interface.

Longitudinal maneuvers should allow controller to define speed profile

Current implementations of the speedup and slowdown maneuvers compute a linear interpolation between start speed and end speed then feed that interpolated command to the controller on every time step. The controller would perform better if we gave it the final target and let its PID naturally produce the critically damped response. But because of the controller's need for an input every 200 ms we need to consider whether a frequent re-issuance of the target command is any better than providing the linear command change at each time step. @watters-torc , your input would be valuable here.
This is not a critical, short-term need, but something to consider longer term.

Wifi capability on vehicle computers

If vehicles are to use I2 connection at Aberdeen they will need to connect to the wifi signal there. We need to get wifi adapters/dongles for each computer. Whether we use this method or direct ethernet cable (not sure if that will be available at Aberdeen), we will need careful and foolproof procedures in place to be sure that the vehicle computer is never connected to I2 and another network (e.g. Verizon aircard) simultaneously. Having a dongle that uses the same USB slot as an aircard would be a good way to prevent accidentally using them both (block other slots).

Trajectory extension for request messages needs to better obey Mobility Strategy

When a MobilityRequest is received beyond the end of the current trajectory, the trajectory is extended to check for conflicts. However, the only plugin capable of responding to a detected conflict during the replan is the default conflict handler (Yield). This is undesirable as the plugin the request was intended for should be the one allowed to plan in that region. We need a way to associate the request message data with the detected conflict. Some of this is already saved like the sender id and plan id in ConflictManager, but we also need the strategy string to route the planning to the correct plugin.

Occasionally Arbitrator Ignores SpeedHarm Plugin

During a roughly 4 min I-95 test run, Arbitrator is not asking SpeedHarm plugin to plan a new trajectory for two times. It is not clear what cause that. But it makes the vehicle be out of control of the speed harm server and back to the control of cruising.
The timestamp is 2018-06-19-10-28-44. Here is a snippet of the log:

 2018-06-19 10:53:37:744 | WARNING | Arbitrator | NONE | Using experimental replan method, replanning with window: 38987.46337911795 
 2018-06-19 10:53:37:744 | INFO | ArbitratorStateMachine | NONE | Arbitrator processing event: TRAJECTORY_FAILED_EXECUTION 
 2018-06-19 10:53:37:744 | INFO | ArbitratorStateMachine | NONE | Arbitrator transitioned to state REPLAN_DUE_TO_FAILED_TRAJECTORY 
 2018-06-19 10:53:37:744 | INFO | Arbitrator | NONE | Arbitrator replanning trajectory because of failed execution! 
 2018-06-19 10:53:37:745 | INFO | Arbitrator | NONE | Arbitrator planning new trajectory spanning [19190.049137347014, 30569.189513248486) 
 2018-06-19 10:53:37:745 | INFO | Arbitrator | NONE | Allowing plugin: Platooning Plugin to plan trajectory. 
 2018-06-19 10:53:37:746 | INFO | PlatooningPlugin | NONE | Plan Trajectory from Trajectory [startLocation=19190.049137347014, endLocation=30569.189513248486] in state PlatoonLeaderState 
 2018-06-19 10:53:37:746 | INFO | ConflictManager | NONE | Adding requested path 
 2018-06-19 10:53:37:746 | INFO | ConflictManager | NONE | addPath 
 2018-06-19 10:53:37:747 | INFO | ConflictManager | NONE | Preparing to insert points 
 2018-06-19 10:53:37:747 | INFO | ConflictManager | NONE | Inserting path with size 1 
 2018-06-19 10:53:37:747 | INFO | ConflictManager | NONE | Done inserting 
 2018-06-19 10:53:37:748 | INFO | Arbitrator | NONE | Allowing plugin: Route Following Plugin to plan trajectory. 
 2018-06-19 10:53:37:748 | INFO | RouteFollowingPlugin | NONE | Route Following Plugin planning trajectory! 
 2018-06-19 10:53:37:748 | INFO | RouteFollowingPlugin | NONE | Identified 0 lane changes in trajectory 
 2018-06-19 10:53:37:748 | INFO | RouteFollowingPlugin | NONE | Changes:  
 2018-06-19 10:53:37:748 | INFO | RouteFollowingPlugin | NONE | Planning lane keeping maneuver planning between: [19190.05, 30569.19) 
 2018-06-19 10:53:37:749 | INFO | RouteFollowingPlugin | NONE | Route Following planning success! 
 2018-06-19 10:53:37:749 | INFO | Arbitrator | NONE | Allowing plugin: Cruising Plugin to plan trajectory. 
 2018-06-19 10:53:37:749 | INFO | CruisingPlugin | NONE | Found 1 empty gaps in trajectory [19190.049137347014, 30569.189513248486] 
 2018-06-19 10:53:37:749 | INFO | CruisingPlugin | NONE | Gap: [startLocation = 19190.049137347014; endLocation = 30569.189513248486; startSpeed = 24.785999298095703] 
 2018-06-19 10:53:37:750 | INFO | VehicleAwareness | NONE | Notified of a forced replan, cleaning invalid trajectories! 
 2018-06-19 10:53:37:752 | INFO | CruisingPlugin | NONE | Found 1 speed limits in Gap: [startLocation = 19190.049137347014; endLocation = 30569.189513248486; startSpeed = 24.785999298095703] 
 2018-06-19 10:53:37:752 | INFO | CruisingPlugin | NONE | SpeedLimit: [location = 30569.189513248486, limit = 29.0576] 
 2018-06-19 10:53:37:752 | INFO | CruisingPlugin | NONE | Trying to plan maneuver {start=19190.05,end=30569.19,startSpeed=24.79,endSpeed=29.06} 
 2018-06-19 10:53:37:753 | INFO | CruisingPlugin | NONE | Planned maneuver from [24.79, 29.06) m/s over [19190.05, 19276.56) m 
 2018-06-19 10:53:37:753 | INFO | CruisingPlugin | NONE | Planned STEADY-SPEED maneuver at 29.06 m/s over [19276.56, 30569.19) m 
 2018-06-19 10:53:37:754 | INFO | Arbitrator | NONE | New trajectory planned in 9 ms. Planning finished at 0.007996015576228123% 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | Trajectory executor commanded to abort trajectory 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | TrajectoryExecutor received new trajectory! 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | Maneuver #1 from [19190.049137347014, 30569.189513248486) of type LATERAL 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | Maneuver #2 from [19190.049137347014, 19276.560727675307) of type LONGITUDINAL 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | Speeds from 24.785999298095703 to 29.0576 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | Maneuver #3 from [19276.560727675307, 30569.189513248486) of type LONGITUDINAL 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | Speeds from 29.0576 to 29.0576 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | TrajectoryExecutor running trajectory! 
 2018-06-19 10:53:37:754 | INFO | ArbitratorStateMachine | NONE | Arbitrator processing event: FINISHED_TRAJECTORY_PLANNING 
 2018-06-19 10:53:37:755 | INFO | ArbitratorStateMachine | NONE | Arbitrator transitioned to state AWAITING_REPLAN

Arbitrator does not respect the presence of complex maneuvers when replanning

During TO 26 testing the cooperative merge plugin tried to place complex maneuvers in a trajectory, however route following would always try to extend the trajectory further. Because maneuvers cannot be placed after a complex maneuver these trajectories fail. This is repeated until 5 trajectories fail and arbitrator throws a exception. Ideally route following would not be allowed to try to extend the trajectory in this case.

Edit: This issue was identified as the root cause of Issue #905