log4j-cloudwatch-appender

Log4j Cloudwatch Appender is a simple to use Log4j Appender that allows you to stream your application logs directly into your AWS Cloudwatch Logs.

In order to optimise network usage and mitigate throttling issues with AWS Cloudwatch, the actual sending of logs may be deferred.

The central (and unique!) class of this library is com.github.speedwing.log4j.cloudwatch.appender.CloudwatchAppender and extends org.apache.log4j.AppenderSkeleton from which it inherits the following features:

• threshold filtering
• general filters
• lifecycle management

NOTE: a potential infinite-loop-kind of issue has been found when com.amazonaws classes push logs via CloudwatchAppen. Check below for more info

Programmatic Configuration

The CloudwatchAppender can be configure either via properties file (see next paragraph) or programmatically in your codebase.

In order to setup the CloudwatchAppender in your codebase it's necessary to add these imports:

import com.github.speedwing.log4j.cloudwatch.appender.CloudwatchAppender;
import org.apache.log4j.Layout;
import org.apache.log4j.Logger;
import org.apache.log4j.PatternLayout;
import org.apache.log4j.Priority;

Now we can now initialise and configure the appender:

PatternLayout layout = new org.apache.log4j.PatternLayout();
layout.setConversionPattern("%d [%t] %-5p %c - %m%n");
CloudwatchAppender cloudwatchAppender = new CloudwatchAppender(layout, "MyLogName", "MyStreamName");
cloudwatchAppender.setThreshold(Priority.WARN);
Logger.getRootLogger().addAppender(cloudwatchAppender);

Configuration via log4j.properties

A much more classic approach in configuring the appender is via the classic log4j.properties.

An example below:

log4j.appender.CloudW=com.github.speedwing.log4j.cloudwatch.appender.CloudwatchAppender
log4j.appender.CloudW.layout=org.apache.log4j.PatternLayout
log4j.appender.CloudW.layout.ConversionPattern=%d [%t] %-5p %c - %m%n
log4j.appender.CloudW.logGroupName=MyLogGroupName
log4j.appender.CloudW.logStreamName=MyLogStreamName
log4j.appender.CloudW.region=us-east-2
log4j.appender.CloudW.queueLength=2048
log4j.appender.CloudW.messagesBatchSize=512

log4j.rootLogger=INFO, CloudW

As you can see alongside logGroupName and logStreamName attributes, there are a couple of optional parameters queueLength and messagesBatchSize. These parameters are used to influence the way CloudwatchAppender communicate to AWS Cloutwatch.

The queueLength parameter

Since throttling can be a real issue when using the AWS Cloudwatch Logs service it's important to mitigate the RPM against it. For this reason an intermediate buffer has been introduced in order to gather a few log entries together and then send them over the wire in one go.

The parameter queueLength sets the maximum number of log entries that can be buffered at any one time by the CloudwatchAppender, subsequent log messages will be discarded.

A separate thread will be polling for messages the buffer and send them over to AWS Cloudwatch.

The default value for queueLength is 1024

The messagesBatchSize parameter

It's the maximum number of log entries to send in one go to AWS Cloudwatch. It's important to pick this large enough to avoid frequent AWS Cloudwatch Logs api calls.

The default value for messagesBatchSize is 128

The log4j.debug flag

The CloudwatchAppender honors the log4j.debug flag and will print extra logging info on the standard out / err.

Gotchas

First of all thanks to jramsey139 for finding and fixing a couple of issues.

The CloudwatchAppender soffers of a potential infinite-look-like issue when com.amazonaws classes push logs to the CloudwatchAppender. Of course if you're using the this Appender you want logs to be pushed to Amazon, but at the same time you don't want to break your service. For this reason I may recommend to either

• log com.amazonaws only to Console or File
• raise the level of logging for com.amazonaws at least to WARN, or even better to ERROR so that, when everything works fine, there are no risks to incur into crazy infinity loops.

Some example on how to tweak the configuration.

Logging only errors from com.amazonaws to Cloudwatch logs

log4j.appender.CW.com.amazonaws=ERROR

Turning off com.amazonaws logs altogether

log4j.appender.CW.com.amazonaws=off

How to release

mvn clean deploy