boxfuse / cloudwatchlogs-java-appender Goto Github PK

I upgrade to 1.1.8.60 to get a hold of the custom MDC fields (thank you by the way)

However since, my messages in cloud watch are hard to read because they now contain all the meta data + the actual log message.

Before (version 1.1.5.46):
{
"image": "[imagename]",
"instance": "i-xxxxxxxx",
"level": "WARN",
"logger": "[logger]",
"message": "[actual log message",
"thread": "[threadid]"
}
Nice and clean...

After:

{
"instance": "i-xxxxx",
"image": "[image]",
"level": "INFO",
"message": "2018-02-01 18:07:27.579 boxfuse/xxx [application] {"image":"[boxfuse image]","instance":"i-xxxxxx","level":"INFO","logger":"[logger]","message":"[THE LOG MESSAGE"}"
}

Logback configuration is as follows:

<appender name="cloud-watch" class="com.boxfuse.cloudwatchlogs.logback.CloudwatchLogsLogbackAppender">
	<config>
		<stdoutFallback>true</stdoutFallback>
		<customMdcKey>my-custom-mdc-key</customMdcKey>
	</config>
</appender>

To auto-recover from errors such as

com.amazonaws.services.logs.model.DataAlreadyAcceptedException: The given batch of log events has already been accepted. The next batch can be sent with sequenceToken: 49576308050167475836136287314051956388877295249203948850 (Service: AWSLogs; Status Code: 400; Error Code: DataAlreadyAcceptedException; Request ID: 0394f028-9c6d-11e7-818f-cdfcfc027d79)

For example, with Log4j2 ver. 2.6.2 (which we were using in production), appender failed on first logging statement with Exception in thread "main" java.lang.NoSuchMethodError: org.apache.logging.log4j.core.LogEvent.getContextData()Lorg/apache/logging/log4j/util/ReadOnlyStringMap; - which is understandable, as it was only introduced in ver. 2.7, as far as I can see.

@axelfontaine
Would be it possible to configure the appender in logback.xml so that it will include a named MDC attribute (that is outside of the list of pre-determined MDC attributes)

<appender name="cloud-watch" class="com.boxfuse.cloudwatchlogs.logback.CloudwatchLogsLogbackAppender">
	<mdc key="my-mdc-key"/>
</appender>

We had a hard time last week to understand why our system went down regularly. But finally we managed by some black box testing to find at least which component seems to be responsible for our frequent downtimes.

It turns out that the log appender seems not to handle our amount of logs (>=20logs/sec) we are sending to it. When we send the logs, the app is not responding anymore and thus boxfuse eventually redeploys it.

We still wonder how this can be the case as the appender should be able to handle much more throughput. We are using the latest version 1.1.3.33, Unfortunately, we can't tell more about the exact error as we won't see anything in the logs nor can't we access the application. What we can tell from our debugging with New Relic and jxm though, is that there seems every thing alright with the memory, threads and open connections on our server.

We found that the logappender is to blame for our downtime when we reduced the functionality of some of our components to only perform some logging and nothing else. When we finally also disabled that, our system became stable again.

To be precisely, I can change the behavior of our backend from a mostly stable to an unstable system by changing:

<!-- boxfuse aws -->
  <appender name="Boxfuse-CloudwatchLogs" class="com.boxfuse.cloudwatchlogs.logback.CloudwatchLogsLogbackAppender"/>

  <root level="INFO">
    <appender-ref ref="Boxfuse-CloudwatchLogs" />
  </root>

to

<!-- boxfuse aws -->
  <appender name="Boxfuse-CloudwatchLogs" class="com.boxfuse.cloudwatchlogs.logback.CloudwatchLogsLogbackAppender"/>

  <root level="DEBUG">
    <appender-ref ref="Boxfuse-CloudwatchLogs" />
  </root>

Hi boxfuse team -

I tried creating a generic version of cloudwatch logs appender using boxfuse's appender as a starting point. I faced several issues during the process - one of them being that - some of the logs are dropped off and are not fed to CloudWatch. This is because the while loop in the run() method of the CloudwatchLogsLogEventPutter.java is polling for one event while there could be 100's of events in the BlockingQueue and it is very much possible that the program execution stops before the while loop can run as many times as there are log events in the queue. To fix this, I had modified the design to use BlockingQueue.drainTo(collection) approach and push all logs to cloudwatch at one go. Please see https://stackoverflow.com/a/48230308/829542 for implementation info. Hope this helps in enhancing the appender on your end for boxfuse environments. Thanks.

Please publish these appenders to Maven Central, so that the community can benefit from them without depending on your Maven repository.

Right now, CloudwatchLogsLog4J2Appender doesn't accept AWS region from configuration, even though CloudwatchLogsConfig supports it, and it is referenced correctly in CloudwatchLogsLogEventPutter.

This appender looks great but it's unfortunate that the log group appears to be mostly hard-coded.