The Log4J Appender for Amazon Kinesis enables customers to publish logs from their Java applications into an Amazon Kinesis stream. We also provide a publisher application which uses the Log4J Appender for Amazon Kinesis to publish data from a file into an Amazon Kinesis stream.

• AWS SDK for Java 1.7
• Java 1.7 (Java SE 7) or later
• Apache Log4j 1.2.x

The Log4J Appender for Amazon Kinesis:

• Buffers log messages in memory
• Uses the Amazon Kinesis AsyncClient from the AWS SDK for Java
• Uses a ThreadPoolExecutor backed by a LinkedBlockingQueue. In most cases, this results in the log message simply being added to the in-memory queue. However, if the queue is full, the logging call will block until there is room in the buffer (e.g. some log entries are successfully sent to Amazon Kinesis and removed from the queue).
• Uses an exponential backoff strategy with a configurable number of maxRetries in case of failures.

If a message is not successfully sent to Amazon Kinesis even after the retry attempts, then it is sent to the fallback handler (and no longer sent to Amazon Kinesis).

Note that the current implementation doesn't insert records in the same order as they are logged. One way to get ordering is by including a timestamp in the log entries and then sorting them at the consumer side of the Amazon Kinesis stream.

We recommend that customers:

• Configure parameters such as buffer size, number of threads, and JVM heap size based on the needs of their use case.
• Configure an additional appender (for example: DailyRollingFileAppender), so that they will retain logs even in the rare cases when the log entries may not be successfully sent to Amazon Kinesis.
• Pick a shutdownTimeout that allows the appender to send all outstanding log entries to Amazon Kinesis before exiting.

For customers who connect to the Internet through a proxy server, the Log4J Appender for Amazon Kinesis will pick up the Java proxy system properties and update the underlying ClientConfiguration instance.

Supported system properies are http.proxyHost, http.proxyPort, http.proxyUser, http.proxyPassword, and http.auth.ntlm.domain. For NTLM authentication on Windows, the COMPUTERNAME environment variable is used to set the proxy workstation.

• Download and install Apache Maven 3.x

• Clone this github repository to a new directory (e.g. by default, this is kinesis-log4j-appender) by using following git command on shell prompt:

  git clone https://github.com/awslabs/kinesis-log4j-appender

  This would download the repository to kinesis-log4j-appender directory.

• On the shell prompt, run mvn clean install in kinesis-log4j-appender directory and wait for packages to build. If the build is successful, you should see a message like the following:

  ------
  [INFO]
  ------
  [INFO] ------------------------------------------------------------------------
  [INFO] BUILD SUCCESS
  [INFO] ------------------------------------------------------------------------
  [INFO] Total time: 5.091s
  [INFO] Finished at: Wed Feb 12 17:35:03 GMT 2014
  [INFO] Final Memory: 52M/185M
  [INFO] ------------------------------------------------------------------------

  This command creates a target directory inside kinesis-log4j-appender directory.

• After building this repository, copy kinesis-log4j-appender-1.0.0.jar from target directory to some other directory (such as KinesisLoggingTest).

• Create an Amazon Kinesis stream (such as testStream).

• Copy sample log4j configuration file from here and save it as log4j.properties in KinesisLoggingTest directory.

• Create the credentials file, AwsCredentials.properties in KinesisLoggingTest directory. It must contain following keys:

  accessKey=<AWS_ACCESS_KEY>
  secretKey=<AWS_SECRET_KEY>

• Run the following command on command prompt from KinesisLoggingTest directory:

  On Mac OS X/Linux/Unix:

  ${JAVA_HOME}/bin/java -cp .:./kinesis-log4j-appender-1.0.0.jar com.amazonaws.services.kinesis.log4j.FilePublisher <path_to_sample_log_file>

  On Windows:

  %JAVA_HOME%/bin/java -cp .;./kinesis-log4j-appender-1.0.0.jar com.amazonaws.services.kinesis.log4j.FilePublisher <path_to_sample_log_file>

  If Java is already in the environment configuration, you can use the following command directly (without reference to JAVA_HOME):

  java -cp ...

  The application begins publishing records to the configured Amazon Kinesis stream, with the following output:

  INFO [main] (FilePublisher.java:64) - Started reading: access_log_1
  [continues...]
  DEBUG [main] (FilePublisher.java:79) - Total 6100 records written to logger
  DEBUG [main] (FilePublisher.java:79) - Total 6200 records written to logger
  DEBUG [main] (FilePublisher.java:79) - Total 6300 records written to logger
  DEBUG [main] (FilePublisher.java:79) - Total 6400 records written to logger
  DEBUG [main] (FilePublisher.java:79) - Total 6500 records written to logger
  DEBUG [main] (FilePublisher.java:79) - Total 6600 records written to logger
  [continues...]
  DEBUG [pool-1-thread-12] (AsyncPutCallStatsReporter.java:62) - Appender (KINESIS) made 12000 successful put requests out of total 12000 in 1 minute, 14 seconds and 250 milliseconds since start
  [continues...]

• Amazon Kinesis Developer Guide
• Amazon Kinesis API Reference
• AWS SDK for Java
• Apache Log4j 1.2.x
• Building a Project with Maven