usdot-fhwa-stol / carma-platform Goto Github PK

The arbitrator currently does not allow for plugins to perform a replan request while trajectory planning is in progress. If a plugin has already planned, but receives new information it has no way of forcing a replan at that point. The arbitrator should probably have a queue based list of future tasks so that when the trajectory is planned, re-planning can immediately began as expected by the plugin. This issue was discovered as one cause of Issue #905.

We should have a tab on the UI for license information. There are quite a few packages used in Carma, each with their own license info, most of which requires reproduction of certain clauses by the executable software. See CARMA OSADP repository for more license info.

The Remote Start checkbox on the UI startup page is right below the OK button, making it pretty easy to touch the wrong one. Since this checkbox is rarely used, I would prefer to see it moved way down to the bottom of the page, maybe in a corner.

catkin_make run_tests doesn't trigger the execution and collection of the tests normally run by the ./gradlew test command. This means CircleCI doesn't get a report on our test results.

#77 attempts to address this but is incomplete.

Current version of platooning plugin requires vehicle to engage one by one from the very first vehicle to the very last vehicle. The first two vehicle can engage in any order, but the third vehicle should engage when the first two vehicle have already formed a platoon (yellow light is on). The forth vehicle should wait to engage until the first three vehicle have already been in the same platoon.
The reason is that when a vehicle is searching for a platoon, it only cares if the rear position of any platoon is in front of the host vehicle, which means that it will not care about whether there is an object between the host vehicle and the front platoon.
In three vehicle case, if they started in random order, there will be a case where the third vehicle is joining the first vehicle. And the second vehicle is never able to join them because it is between the first and the third.
#827 is a fix for that, but the sensor fusion anomaly prevent it from working properly. (See #842)

Currently, if arbitrator got a notification on trajectory failure when it is running a complex maneuver, the speed change from the complex maneuver to a regular longitudinal is not handled. It might result in a max_accel deceleration from the current speed.

Guidance components wishing to be triggered by guidance state events are responsible for implementing the onStateChange function themselves. This is strange given they also implement the called functions and every guidance component implements the function in basically the same way.
This implementation should be moved to a GuidanceComponent base class or similarly refactored out of the components themselves.

     */
    @Override
    public void onStateChange(GuidanceAction action) {
        log.debug("GUIDANCE_STATE", getComponentName() + " received action: " + action);
        switch (action) {
        case INTIALIZE:
            jobQueue.add(this::onSystemReady);
            break;
        case ACTIVATE:
            jobQueue.add(this::onActive);
            break;
        case DEACTIVATE:
            jobQueue.add(this::onDeactivate);
            break;
        case ENGAGE:
            jobQueue.add(this::onEngaged);
            break;
        case SHUTDOWN:
            jobQueue.add(this::onShutdown);
            break;
        case PANIC_SHUTDOWN:
            jobQueue.add(this::onPanic);
            break;
        case RESTART:
            jobQueue.add(this::onCleanRestart);
            break;
        default:
            throw new RosRuntimeException(
                    getComponentName() + "received unknown instruction from guidance state machine.");
        }
    }

The light bar on the truck is using inverted commands. On means off and vise versa when making the set_lights service call. Additionally, the green lights do not behave in a reliable fashion. The green lights only sometimes turn on or off.

This was first found to be an issue during Aberdeen testing in July 2018.

Notes from our Task Meeting 5/14/2018:

Inactive State - 220ms controller timeouts which may be caused by volume of messages. Operations message with command speeds. With BSMs and Path messages off, there was only 3 out of 4 that was inactive. Speed cmds are always sent at 100ms, regardless if the message is received or not. It takes 0.001s Message to Mobility. There is 0.3s delay from MobilityRouter to Platooning. It's a ROS delay.

TODO: Need to ask Tim to create a double lap route file with 4-5 vehicles next week.

Complex replan triggers setup in Arbitrator are supposed to delete themselves from the trajectory progress callback list after being called, but this is not the case, leading to 1 invocation, then 2, then 3, then 4, escalating with each trajectory planned.

Performance impact is likely low, as callback is very small and not observed to get above 10-12 executions, and only done once per trajectory, but is notable issue.

When trying to start Carma this morning we ran into several instances of the UI informing "unable to find SYSTEM READY message" or something similar. The root cause was the pinpoint was in a fault status, but required looking through logs to determine. It would be nice to have the UI display the content from the SYSTEM_ALERT message when one is received to make diagnosis quicker.

Current implementations of the speedup and slowdown maneuvers compute a linear interpolation between start speed and end speed then feed that interpolated command to the controller on every time step. The controller would perform better if we gave it the final target and let its PID naturally produce the critically damped response. But because of the controller's need for an input every 200 ms we need to consider whether a frequent re-issuance of the target command is any better than providing the linear command change at each time step. @watters-torc , your input would be valuable here.
This is not a critical, short-term need, but something to consider longer term.

Tim Tiernan has requested that the UI always display the sensed gap if a forward vehicle is in view in host's lane (show distance in meters). Ideally, this would show just the track 1 data from the Cadillac radar, but if we can't easily get that isolated, then whatever data we have coming out of sensor fusion.

The lane change code, and soon the platooning code, will make decisions based on small differences in host vehicle speed and that of neighbor vehicles. We have lots of experience from Glidepath that measured speed data at 10 Hz contains enough noise to throw these subtle calculations into unrecoverable chaos. A large part of the solution there was to implement a sophisticated data filter to smooth out not only the speed but the acceleration derived from it. The hybrid polynomial/Holoborodko filter used there provides good filtering at that frequency for both the data stream (speed) and its derivative. This algorithm, or something similar, needs to be adopted in the Carma code base.

During overnight testing on I-95 IHP the morning of 6/12, lead vehicle Green demonstrated at least one instance where it seemed to be ignoring incoming speed harm commands; it was receiving speed harm commands at ~22 m/s but GuidanceCommand was issuing commands to the controller at 29 m/s. Quick scan of the logs seemed to indicate that the trajectory planning was ignoring speed harm, as no complex maneuver appeared in the trajectory. Not clear why. One example is in green log 20180612-014837 at time 01:55:44.

During I-95 testing, we observed abnormal gaps when we are testing 0.84 time headway at 65mph.
The gap between vehicles from front bumper to rear bumper should be around 20 meters. But from radar data, it seems that the gap drops to 5~10 meters at some time for the third and forth vehicle.

When testing the Speed Harmonization Plugin at ATEF the Linear Speed Change Algorithm did not seem to change its behavior based on modifications to the algorithms.properties file located in the same folder as the jar. This made testing difficult but a workaround was found by creating a new algorithm which reduced the vehicles speed by 5mph.

When a MobilityRequest is received beyond the end of the current trajectory, the trajectory is extended to check for conflicts. However, the only plugin capable of responding to a detected conflict during the replan is the default conflict handler (Yield). This is undesirable as the plugin the request was intended for should be the one allowed to plan in that region. We need a way to associate the request message data with the detected conflict. Some of this is already saved like the sender id and plan id in ConflictManager, but we also need the strategy string to route the planning to the correct plugin.

Per ATEF drivers, not counting up or down and just displaying 0.

Takes minutes for "merge" vehicle to stop by coasting to 3MPH, and jerks until then.

Give TORC an email/call on controlling a vehicle to a stop. This is potentially a scope creep but try to address now if possible. Assign to Ed to contact Tammy R. to resolve this somehow and have Mike to followup.

Because the underlying rosjava subscription/publication object is shared between guidance components, it is possible for one component to cause a class cast exception in another if the two components expect different message types on the same topic. This generates a very cryptic error such as the one below.

java.lang.ClassCastException: org.ros.internal.message.$Proxy61 cannot be cast to cav_msgs.MobilityResponse
    at gov.dot.fhwa.saxton.carma.guidance.mobilityrouter.MobilityRouter.lambda$fireMobilityRequestCallback$0(MobilityRouter.java:196)
    at java.lang.Thread.run(Thread.java:748)

To avoid this the pub sub manager should cache all subscribed topic names, and assign a message type. This topic name should probably be fully defined to allow for namespaces. If a future user wishes to use a different message type a clearly described error should be thrown.

During I-95 test on NB 65 mph route, arbitrator stops to plan any trajectories after planning trajectory spanning [10420.896801400624, 10481.931957650624), which causes the vehicle never able to join the platoon. But the CARMA guidance is always ENGAGED.
guidance.txt

It appears that most of the nodes, after they have signalled shutdown underway, generate an unhandled NPE in one of their onNewMessage methods.

 2018-06-12 03:14:18:654 | SEVERE | Channel exception: java.io.IOException: Connection reset by peer 
 2018-06-12 03:14:18:659 | WARNING | GuidanceStateHandler | GUIDANCE_STATE | GuidanceStateHandlerreceived SHUTDOWN 
 2018-06-12 03:14:18:660 | INFO | GuidanceStateHandler | SHUTDOWN | Guidance state handler waiting5000ms for guidance thread shutdown... 
 2018-06-12 03:14:18:660 | INFO | GuidanceStateHandler | NONE | Publishing system_alert shutdown to ensure whole system shutdown 
 2018-06-12 03:14:18:663 | INFO | ManeuverInputs | NONE | Guidance.Maneuvers.ManeuverInputs shutting down normally. 
 2018-06-12 03:14:18:665 | INFO | GuidanceCommands | NONE | Guidance.Commands shutting down normally. 
 2018-06-12 03:14:18:671 | INFO | Tracking | NONE | Guidance.Tracking shutting down normally. 
 2018-06-12 03:14:18:673 | INFO | TrajectoryExecutor | NONE | Guidance.TrajectoryExecutor shutting down normally. 
 2018-06-12 03:14:18:688 | INFO | MobilityRouter | NONE | MobilityRouter shutting down normally. 
 2018-06-12 03:14:18:689 | INFO | LightBarManager | NONE | Light Bar Manager shutting down normally. 
 2018-06-12 03:14:18:689 | INFO | VehicleAwareness | NONE | VehicleAwareness shutting down normally. 
 2018-06-12 03:14:18:692 | INFO | I/O exception (java.net.ConnectException) caught when processing request: Connection refused (Connection refused) 
 2018-06-12 03:14:18:692 | INFO | Retrying request 
 2018-06-12 03:14:18:692 | INFO | Arbitrator | NONE | Guidance.Arbitrator shutting down normally. 
 2018-06-12 03:14:18:692 | INFO | I/O exception (java.net.ConnectException) caught when processing request: Connection refused (Connection refused) 
 2018-06-12 03:14:18:692 | INFO | Retrying request 
 2018-06-12 03:14:18:693 | INFO | I/O exception (java.net.ConnectException) caught when processing request: Connection refused (Connection refused) 
 2018-06-12 03:14:18:693 | INFO | Retrying request 
 2018-06-12 03:14:18:694 | SEVERE | PluginManager | NONE | Guidance.PluginManager is about to SHUTDOWN! 
 2018-06-12 03:14:18:695 | WARNING | GuidanceStateHandler | GUIDANCE_STATE | GuidanceStateHandlerreceived SHUTDOWN 
 2018-06-12 03:14:18:699 | INFO | I/O exception (java.net.ConnectException) caught when processing request: Connection refused (Connection refused) 

This might be a new issue related to the new logging change. Need further investigate if it happens a lot.

Drivers would like a manual toggle on the dashboard to activate the light bar to signal a hazardous situation. Its activation should be independent from anything going on in the software. Related to issue #9. 11/14/17 CCB: Need to deconflict this with software-controlled activation of the lights. Maybe yellow lights are manually controlled for safety, and only green light is used for automation indicators.

When we are doing two-vehicle platooning testing, the first vehicle is in the same lane of the second one broadcasting BSM. The seconds vehicle see one object in front with empty BSM id and see another vehicle with BSM in a different lane (lane 4). Here are logs file for the second vehicle:
F_silver_test1.txt
The log looks like this:

2018-05-08 07:30:43:873 | DEBUG | PlatooningPlugin | NONE | Found obstacle in lane 0 and its dtd is 226.15253250848778
2018-05-08 07:30:43:874 | DEBUG | PlatooningPlugin | NONE | The found obstacle type is MessageImpl<cav_msgs/ConnectedVehicleType>
2018-05-08 07:30:43:874 | DEBUG | PlatooningPlugin | NONE | The BSM id is of size 0
2018-05-08 07:30:43:874 | DEBUG | PlatooningPlugin | NONE | Found obstacle in lane 4 and its dtd is 187.2083069788181
2018-05-08 07:30:43:874 | DEBUG | PlatooningPlugin | NONE | The found obstacle type is MessageImpl<cav_msgs/ConnectedVehicleType>
2018-05-08 07:30:43:874 | DEBUG | PlatooningPlugin | NONE | The BSM id is of size 4
2018-05-08 07:30:43:874 | DEBUG | PlatooningPlugin | NONE | The content at 0 is 73
2018-05-08 07:30:43:875 | DEBUG | PlatooningPlugin | NONE | The content at 1 is 54
2018-05-08 07:30:43:875 | DEBUG | PlatooningPlugin | NONE | The content at 2 is -59
2018-05-08 07:30:43:875 | DEBUG | PlatooningPlugin | NONE | The content at 3 is -70

I confirmed that the reported BSM ID in lane 4 is what the first vehicle is boardcasting.

The first element in the indicated message spec is named "speed", which implies it is the current speed of something. Reading the header comment kinda implies it is intended to represent the commanded speed. Let's either expand the description of this field or rename it (e.g. speed_cmd) to clarify which it is.

In TO26 I-95 testing, The light bar does not turn-on reliably for platooning after merge. This is because the RSU is sending commands to do a lane change as the route geometry does not match the road (rsu thinks vehicle is in the wrong lane). The requested steering command causes the lane change plugin to request the light bar as shown below. This prevents from platooning getting access till much later.

2018-06-25 13:32:35:074 | INFO | LightBarManager | NONE | Platooning Plugin failed to set the LightBarIndicator YELLOW as this was already controlled by Lane Change Plugin 

On 5/22/2018 ATEF Testing, Phil (Volpe) said him and the drivers are concerned about the issue when the vehicles just suddenly stop at the end of route, especially when there's a platoon behind them. This doesn't happen all the time but when it does, it increases the likelihood of an accident.

Attached are the 3 car platoon logs from today where Gray exhibited INACTIVITY and after re-engaging at the end of the route, it immediately stopped.

As a workaround, Tim said to split into two lanes towards the end of the route.

The route specification cannot describe a lane change where the number of lanes does not change and the lane index does not change. (The vehicle is not aware of this lane change)

Route computes cross track error, which is appropriate, but it should not be initiating system shutdown based on the value. Guidance.Tracking is where decisions are made about whether the vehicle is too far off plan. All of this decision logic should be consolidated in this one class.

On I-95 testing, the complex maneuver planned by cooperative merge plugin is deleted by Platooning Plugin. The intention is to join a platoon after we finished merging. However, platooning plugin started to plan a new trajectory once it finished negotiation with the pass-by platoon, when it is still on the ramp (it should be under control of RSU).

If vehicles are to use I2 connection at Aberdeen they will need to connect to the wifi signal there. We need to get wifi adapters/dongles for each computer. Whether we use this method or direct ethernet cable (not sure if that will be available at Aberdeen), we will need careful and foolproof procedures in place to be sure that the vehicle computer is never connected to I2 and another network (e.g. Verizon aircard) simultaneously. Having a dongle that uses the same USB slot as an aircard would be a good way to prevent accidentally using them both (block other slots).

Suggestion to provide a checklist of procedures on the tablet. This may be better served by handing out a printed QRC, since instructions may need to be changed frequently. 11/14/17 CCB: This is better as a document. Can have a hand-held sheet, but also make it a PDF on the tablet.

Truck drivers and SRX drivers have different topic name for some output data. For example, truck controller subscribes to speed_accel topic but SRX drivers subscribes to cmd_speed topic for cmd_speed input.
Truck drivers and SRX drivers have different units on the same topic. For example, truck CAN reports current speed in km/h but SRX reports them in m/s.
There are more mismatch on unit and topic name. We need to verify all necessary topic to make sure they have the same interface.

A GoPro camera fell from its ceiling mounted position and hit the E-Stop button. This resulted in a system shutdown when the controller reported a FAULT. However discovering the source of this issue was time consuming as no one has observed the button being pressed. Some indication of E-Stop compression needs to be available in the vehicle such as an LED indicator.

The ACC system of carma could have it's stability improved. After, the speed controller and acc were tuned at Aberdeen on April 16th and 17th of 2018. The speed controller comfort was dramatically improved. ACC fixes were implemented in PR #764. ACC was brought to an acceptable level according to Tim. We were running at 45 and 60 mph.

However, their is still room for improvement. The system demonstrates large oscillations and some instability. This seems to mostly be due to a lag between guidance commands and the speed controller. This is likely due to the ACC system utilizing a PID on speed commands. This controller results in small changes in the target speed of the TORC speed controller. This leads to low errors and poor response time. The best fix would be moving the ACC system to wrench effort based commands. This would remove the stacked PID loops and allow for a better response and more effective tuning of the system.

The ros params used by plugins are all assigned to the private guidance_main namespace. However, unless the plugin specifies a prefix like example_plugin, then the parameters could clash and be overwritten.

For example if a plugin has a timeout it will look like
/saxton_cav/guidance/guidance_main/timeout

but instead it we should make it
/saxton_cav/guidance/guidance_main/example_plugin/timeout

Each driver needs to connect to a specific PC CAN interface. But truck CAN interface will be reordered if CAN devices initialized in a different order. For example if we unplug the radar USB PCAN, it will change the name of truck controller driver CAN interface name (from can1 to can0) and truck CAN driver (from can2 to can1).
Ideally, we should set up some udev rules on the computer to lock in the devices regardless of what order they are initialized.

During verification test on 11/7/2017, we discovered both Guidance and Sensor Fusion wasn't cleanly shutting down.

We need an installer for Carma that is redistributable and not dependent on the presence of a development environment to copy from. The idea is to package all files needed for the vehicle PC onto a memory stick or similar, copy from there to the vehicle PC and run the installer to build the PC - no other computers online. Think InstallShield, although it doesn't have to be that pretty.

Current implementation of platooning is letting each vehicle report their own downtrack distance. It would cause a failure in some cases:

1. If two vehicles are using two different route files, they will have different starting points for the downtrack distance.
2. If two vehicles are using the same route file, but one vehicle join the route before it is actually on any route segment, that vehicle will have a different starting point with the vehicle who join the route when it is already on the route.
   In both cases, for the same location, they will report different dtd, which means they should not platoon together.
   The current workaround is to avoid those two cases. But it should be fixed when the sensor fusion is functionally ready. See #842

During TO 26 testing the cooperative merge plugin tried to place complex maneuvers in a trajectory, however route following would always try to extend the trajectory further. Because maneuvers cannot be placed after a complex maneuver these trajectories fail. This is repeated until 5 trajectories fail and arbitrator throws a exception. Ideally route following would not be allowed to try to extend the trajectory in this case.

Edit: This issue was identified as the root cause of Issue #905

While testing on local VM, got this Fatal Error when :

1. carma UI running and engaged while speed harm server running.
2. speed harm is receiving vehiclecommands and carma UI receiving SH commands.
3. stopped speed harm server (which is similar to server not being available during a run)
4. On the UI, driver disengages and hits continue. Then carma throws a fatal error.

Expected behavior should be to make the plugin unavailable and continue with the other plugins.

2018-05-03 16:59:58:768 | INFO | PluginLifecycleHandler | PLUGIN | Suspending Cruising Plugin:1.0.2
2018-05-03 16:59:58:774 | DEBUG | YieldPlugin | NONE | Availability Listeners 1
2018-05-03 16:59:58:777 | INFO | PluginLifecycleHandler | PLUGIN | Suspending Yield Plugin:1.0.1
2018-05-03 16:59:58:777 | INFO | LightBarManager | NONE | Set light bar for Light Bar State Machine for LightBarIndicator GREEN with statusOFF
2018-05-03 16:59:58:778 | INFO | LightBarStateMachine | NONE | Event: GUIDANCE_DISENGAGED Prev State: ENGAGED New State: DISENGAGED
2018-05-03 16:59:58:778 | DEBUG | LightBarManager | NONE | Light Bar Manager is polling job queue.
2018-05-03 16:59:58:778 | FATAL | GuidanceMain | NONE | Guidance thread Speed Harmonization PluginDoSuspendPluginWorkerThread raised uncaught exception! Handling!!!
2018-05-03 16:59:58:778 | ERROR | GuidanceExceptionHandler | NONE | Global guidance exception handler caught an uncaught exception from thread: Speed Harmonization PluginDoSuspendPluginWorkerThread
org.springframework.web.client.ResourceAccessException: I/O error on DELETE request for "http://localhost:8081/rest/vehicles/15": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:728)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:671)
at org.springframework.web.client.RestTemplate.delete(RestTemplate.java:540)
at gov.dot.fhwa.saxton.carma.plugins.speedharm.SessionManager.endVehicleSession(SessionManager.java:81)
at gov.dot.fhwa.saxton.carma.plugins.speedharm.SpeedHarmonizationPlugin.onSuspend(SpeedHarmonizationPlugin.java:176)
at gov.dot.fhwa.saxton.carma.guidance.plugins.SuspendPluginTask.run(SuspendPluginTask.java:29)
at gov.dot.fhwa.saxton.carma.guidance.plugins.PluginLifecycleHandler$PluginWorker.run(PluginLifecycleHandler.java:299)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:714)
... 7 more
2018-05-03 16:59:58:779 | DEBUG | GuidanceStateMachine | GUIDANCE_STATE | Guidance state machine reveiced PANIC at state: DRIVERS_READY
2018-05-03 16:59:58:779 | DEBUG | GuidanceStateMachine | GUIDANCE_STATE | Guidance transited to state SHUTDOWN
2018-05-03 16:59:58:779 | DEBUG | GuidanceStateMachine | GUIDANCE_STATE | Guidance is taking action PANIC_SHUTDOWN
2018-05-03 16:59:58:782 | DEBUG | LaneChangePlugin | NONE | Availability Listeners 1
2018-05-03 16:59:58:784 | INFO | PluginLifecycleHandler | PLUGIN | Suspending Lane Change Plugin:1.0.2
2018-05-03 16:59:58:785 | INFO | YieldPlugin | NONE | Yield plugin suspended
2018-05-03 16:59:58:787 | DEBUG | RouteFollowingPlugin | NONE | Availability Listeners 1

WARNING: | CommandReceiver | NONE | Unable to wait for server speed command, received exception.
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8081/rest/commands/8": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:728)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:671)
at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:319)
at gov.dot.fhwa.saxton.carma.plugins.speedharm.CommandReceiver.run(CommandReceiver.java:53)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:714)
... 4 more

May 03, 2018 12:01:30 PM org.ros.internal.node.RosoutLogger info
INFO: | TrajectoryExecutor | NONE | Received RouteState. New downtrack distance: 0.8565578851848841
May 03, 2018 12:01:30 PM org.ros.internal.node.RosoutLogger info
INFO: | MessageConsumer | NONE | We encode BSM
May 03, 2018 12:01:30 PM org.ros.internal.node.RosoutLogger info
INFO: | Tracking | NONE | Current vehicle speed is -0.0117899441719055 m/s
May 03, 2018 12:01:30 PM org.ros.internal.node.RosoutLogger info
INFO: | Arbitrator | NONE | Received RouteState:MessageImpl<cav_msgs/RouteState>
May 03, 2018 12:01:31 PM org.ros.internal.node.RosoutLogger info
INFO: | TrajectoryExecutorWorker | NONE | Finished downtrack distance update with no trajectory.
May 03, 2018 12:01:31 PM org.ros.internal.node.RosoutLogger info
INFO: | PluginLifecycleHandler | PLUGIN | Suspending Speed Harmonization Plugin:1.0.0
May 03, 2018 12:01:31 PM org.ros.internal.node.RosoutLogger info
INFO: | PluginLifecycleHandler | PLUGIN | Suspending Yield Plugin:1.0.1
May 03, 2018 12:01:31 PM org.ros.internal.node.RosoutLogger fatal
SEVERE: | GuidanceMain | NONE | Guidance thread Speed Harmonization PluginDoSuspendPluginWorkerThread raised uncaught exception! Handling!!!
May 03, 2018 12:01:31 PM org.ros.internal.node.RosoutLogger info
INFO: | YieldPlugin | NONE | Yield plugin suspended
May 03, 2018 12:01:31 PM org.ros.internal.node.RosoutLogger error
SEVERE: | GuidanceExceptionHandler | NONE | Global guidance exception handler caught an uncaught exception from thread: Speed Harmonization PluginDoSuspendPluginWorkerThread
org.springframework.web.client.ResourceAccessException: I/O error on DELETE request for "http://localhost:8081/rest/vehicles/8": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:728)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:671)
at org.springframework.web.client.RestTemplate.delete(RestTemplate.java:540)
at gov.dot.fhwa.saxton.carma.plugins.speedharm.SessionManager.endVehicleSession(SessionManager.java:81)
at gov.dot.fhwa.saxton.carma.plugins.speedharm.SpeedHarmonizationPlugin.onSuspend(SpeedHarmonizationPlugin.java:176)
at gov.dot.fhwa.saxton.carma.guidance.plugins.SuspendPluginTask.run(SuspendPluginTask.java:29)
at gov.dot.fhwa.saxton.carma.guidance.plugins.PluginLifecycleHandler$PluginWorker.run(PluginLifecycleHandler.java:299)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:714)
... 7 more

Our package.xml files still contain "todo"s and example info in various fields such as license info and maintainer contact info. We need to populate all of these with real info before community-wide distro in September 2018. Probably need to work w/Taylor to figure out appropriate maintainer contact info.

Tracking compares the current downtrack distance with a wrong number, which will show a wrong/large downtrack error. The same thing happened in tracking speed error. Something must be wrong in the logic of error tracking. Here is the log file:
20180214-145722.txt

During a roughly 4 min I-95 test run, Arbitrator is not asking SpeedHarm plugin to plan a new trajectory for two times. It is not clear what cause that. But it makes the vehicle be out of control of the speed harm server and back to the control of cruising.
The timestamp is 2018-06-19-10-28-44. Here is a snippet of the log:

 2018-06-19 10:53:37:744 | WARNING | Arbitrator | NONE | Using experimental replan method, replanning with window: 38987.46337911795 
 2018-06-19 10:53:37:744 | INFO | ArbitratorStateMachine | NONE | Arbitrator processing event: TRAJECTORY_FAILED_EXECUTION 
 2018-06-19 10:53:37:744 | INFO | ArbitratorStateMachine | NONE | Arbitrator transitioned to state REPLAN_DUE_TO_FAILED_TRAJECTORY 
 2018-06-19 10:53:37:744 | INFO | Arbitrator | NONE | Arbitrator replanning trajectory because of failed execution! 
 2018-06-19 10:53:37:745 | INFO | Arbitrator | NONE | Arbitrator planning new trajectory spanning [19190.049137347014, 30569.189513248486) 
 2018-06-19 10:53:37:745 | INFO | Arbitrator | NONE | Allowing plugin: Platooning Plugin to plan trajectory. 
 2018-06-19 10:53:37:746 | INFO | PlatooningPlugin | NONE | Plan Trajectory from Trajectory [startLocation=19190.049137347014, endLocation=30569.189513248486] in state PlatoonLeaderState 
 2018-06-19 10:53:37:746 | INFO | ConflictManager | NONE | Adding requested path 
 2018-06-19 10:53:37:746 | INFO | ConflictManager | NONE | addPath 
 2018-06-19 10:53:37:747 | INFO | ConflictManager | NONE | Preparing to insert points 
 2018-06-19 10:53:37:747 | INFO | ConflictManager | NONE | Inserting path with size 1 
 2018-06-19 10:53:37:747 | INFO | ConflictManager | NONE | Done inserting 
 2018-06-19 10:53:37:748 | INFO | Arbitrator | NONE | Allowing plugin: Route Following Plugin to plan trajectory. 
 2018-06-19 10:53:37:748 | INFO | RouteFollowingPlugin | NONE | Route Following Plugin planning trajectory! 
 2018-06-19 10:53:37:748 | INFO | RouteFollowingPlugin | NONE | Identified 0 lane changes in trajectory 
 2018-06-19 10:53:37:748 | INFO | RouteFollowingPlugin | NONE | Changes:  
 2018-06-19 10:53:37:748 | INFO | RouteFollowingPlugin | NONE | Planning lane keeping maneuver planning between: [19190.05, 30569.19) 
 2018-06-19 10:53:37:749 | INFO | RouteFollowingPlugin | NONE | Route Following planning success! 
 2018-06-19 10:53:37:749 | INFO | Arbitrator | NONE | Allowing plugin: Cruising Plugin to plan trajectory. 
 2018-06-19 10:53:37:749 | INFO | CruisingPlugin | NONE | Found 1 empty gaps in trajectory [19190.049137347014, 30569.189513248486] 
 2018-06-19 10:53:37:749 | INFO | CruisingPlugin | NONE | Gap: [startLocation = 19190.049137347014; endLocation = 30569.189513248486; startSpeed = 24.785999298095703] 
 2018-06-19 10:53:37:750 | INFO | VehicleAwareness | NONE | Notified of a forced replan, cleaning invalid trajectories! 
 2018-06-19 10:53:37:752 | INFO | CruisingPlugin | NONE | Found 1 speed limits in Gap: [startLocation = 19190.049137347014; endLocation = 30569.189513248486; startSpeed = 24.785999298095703] 
 2018-06-19 10:53:37:752 | INFO | CruisingPlugin | NONE | SpeedLimit: [location = 30569.189513248486, limit = 29.0576] 
 2018-06-19 10:53:37:752 | INFO | CruisingPlugin | NONE | Trying to plan maneuver {start=19190.05,end=30569.19,startSpeed=24.79,endSpeed=29.06} 
 2018-06-19 10:53:37:753 | INFO | CruisingPlugin | NONE | Planned maneuver from [24.79, 29.06) m/s over [19190.05, 19276.56) m 
 2018-06-19 10:53:37:753 | INFO | CruisingPlugin | NONE | Planned STEADY-SPEED maneuver at 29.06 m/s over [19276.56, 30569.19) m 
 2018-06-19 10:53:37:754 | INFO | Arbitrator | NONE | New trajectory planned in 9 ms. Planning finished at 0.007996015576228123% 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | Trajectory executor commanded to abort trajectory 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | TrajectoryExecutor received new trajectory! 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | Maneuver #1 from [19190.049137347014, 30569.189513248486) of type LATERAL 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | Maneuver #2 from [19190.049137347014, 19276.560727675307) of type LONGITUDINAL 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | Speeds from 24.785999298095703 to 29.0576 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | Maneuver #3 from [19276.560727675307, 30569.189513248486) of type LONGITUDINAL 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | Speeds from 29.0576 to 29.0576 
 2018-06-19 10:53:37:754 | INFO | TrajectoryExecutor | NONE | TrajectoryExecutor running trajectory! 
 2018-06-19 10:53:37:754 | INFO | ArbitratorStateMachine | NONE | Arbitrator processing event: FINISHED_TRAJECTORY_PLANNING 
 2018-06-19 10:53:37:755 | INFO | ArbitratorStateMachine | NONE | Arbitrator transitioned to state AWAITING_REPLAN

In the TO26 cooperative merge application an RSU broadcasts a mobility request with the meter point. This request message is incorrectly processed as a vehicle path by mobility router causing numerous replans on all vehicles. PR #930 adds a band-aid fix for this by preventing replans for single point paths, but this demonstrates a need for mobility requests to convey messages without an included path. Perhaps a flag could be added to disable the concept of a path in a request message. Additionally, other messages types should probably support some kind of single point location information.

The YAML library doesn't allow underscore characters in names assigned inside the file (filenames themselves are okay). We need to include user instructions to clarify this requirement for people building route definitions. Alternatively, this may be another reason to switch to using CSV format for route files.

• Attempt to match the whole query string for capabilities
• Allow wildcards for sections of the capability strings: comms/*/j2735/inbound_binary_msg
• Increase specificity of inbound_binary_msg and outbound_binary_msg to include message set and carrying channel: dsrc/j2735/inbound_binary_msg