valtech-sd / logstash-output-iothub Goto Github PK

When support for MQTT_WS was introduced, this project had to update the dependency of the Azure IoTHub SDK to a newer version. This version introduces what appears to be a new logging pattern an is extremely verbose.

For example, for every event sent to the output, the IoTHub transport outputs several entries. Some examples:

[INFO ] 2020-06-13 01:22:41.706 [[pipeline-02-adx]>worker1] Message - The message with messageid 6aa6417e-fd85-4ce7-83a9-01fc96cfe947 has expiry time as 3000 milliseconds and the message will expire on Sat Jun 13 01:22:44 UTC 2020, method name is setExpiryTime
[INFO ] 2020-06-13 01:22:41.753 [[pipeline-02-adx]>worker1] DeviceIO - Message with messageid 85a82faf-50bf-4889-bcd9-588b88054285 along with callback and callbackcontext is added to the queue, method name is sendEventAsync
[INFO ] 2020-06-13 01:22:43.075 [azure-iot-sdk-IotHubSendTask] IotHubTransport - Get the message from waiting message queue to be sent to IoT Hub, method name is sendMessages
[INFO ] 2020-06-13 01:22:43.161 [azure-iot-sdk-IotHubSendTask] IotHubTransport - Invoking the callback function for sent message, IoT Hub responded to message with status OK_EMPTY, method name is invokeCallbacks

The MS SDK discusses logging in their documentation at https://github.com/Azure/azure-iot-sdk-java/blob/master/logging.md.

However, it is unclear at this time how to properly control logging for a java library from Ruby.

A branch has been started with a possible file, though it is unknown right now if this file is the proper way to do this (and where the file should be placed.)

The work-in-progress branch is:
https://github.com/valtech-sd/logstash-output-iothub/tree/fix/to-reduce-iotsdk-java-verbosity

The issue was first observed with the image "docker.elastic.co/logstash/logstash:7.8.0" which was known to work previously (around 2020 to early 2021).

The specific error messages seen for 7.8.0 in the Docker console were not noted.

The Issue is also observed with the image "docker.elastic.co/logstash/logstash:7.17.0" which was not previously tested but was tried attempting to resolve this issue. The console shows errors stating the plugin is not "found" and a "log4j" error is also logged.

The specific error messages seen for 7.17.0 in the Docker console are: (truncated here for brevity)

(PluginLoadingError) Couldn't find any output plugin named 'iothub'.

Trying to load the iothub output plugin resulted in this error: undefined local variable or method `log4j' for main:Object"