Don't use ConsoleAppender about logback-awslogs-json-encoder HOT 3 CLOSED

Thanks for your feedback!

It would be interesting if you could share some numbers like number of logs per minute to see what is a "fairly decent logging load" in your scenario.

A few thoughts from my side:

• I'm using this library for myself for more than 2 years without any missing logs. The applications in which I use it currently are typically producing round about 150 logs per minute at maximum.
• To be precisely, your problem is not the ConsoleAppender (stdout) itself, but the awslogs driver that obviously doesn't correctly fetch and process those logs. I suspect a bug in its implementation and would recommend to open a bug report at https://github.com/docker/for-linux. Especially (but not only) if you have reproducible problems like "Really long lines hang the awslogs driver".
• Using a library that directly connects to the CloudWatch API definitely has some advantages. But it also introduces several dependencies to your application. The library you switched to has 13 dependencies in addition to Logback with a total size of more than 5 MB. Some of those dependencies might cause incompatibilities with other (direct) dependencies of your application at some point in time (e.g. httpclient or jackson-core). That might be tolerable to you, but it wasn't to me. My library (logback-awslogs-json-encoder) has no dependencies beside Logback and has only 8 Kilobyte in size.
• This library allows an application designed in accordance to the twelve-factor methodology – see log handling as per twelve-factor. Following this methodology allows to have applications that are easily portable between different platforms (AWS, Azure, Google, Kubernetes On-Prem, etc.).

from logback-awslogs-json-encoder.

We're logging roughly about 1MB/s.

I agree with you that that logging should be portable between platforms, and that's the reason we tried really hard with finetuning to get things to work with the awslogs driver. But we ultimately could not get it to work properly.

We have a sample C application that hangs the container, it's quite simple: just call write(stdout, buffer, 128) a few thousand times without a single newline in the buffer. The other issue of characters dropping is less obvious to trigger in a sample application (but it happens often with our real workload). It gets real obvious when logging log lines as JSON data, and functionbeat can't parse the JSON because of missing quotes or curly brackets.
We think it's an issue of the awslogs driver not optimally batching PutLog requests, so it gets very easily throttled (there's a limit of 5 PutLog requests per second, and each may be max 1MB or so)

Ultimately we concluded we could not wait for AWS to fix the awslogs issues (even if we opened up support tickets, which we haven't, due to time constraints). So we decided to try integrating a logger framework that communicated directly with Cloudwatch, and it's working fine now.

This is more of a story of caution, for people that decided to go with the awslogs driver route.

As for dependencies: CloudWatch is relatively simple to communicate with. It can be done without any dependencies, just standard Java that does a HttpUrlConnection. It's just this particular implementation we chose decided to go full blown, which is OK for most enterprise applications. But it can be rewritten to be smaller, for sure.

from logback-awslogs-json-encoder.

Thanks for your feedback! It'd great if you could quickly open a ticket at AWS support and link to this issue. That would give them a chance to investigate and hopefully fix those issues.

from logback-awslogs-json-encoder.