Events less than 1 ms apart can be missed? about logstash-input-cloudwatch-logs HOT 3 OPEN

@lukewaite we're seeing randomly missed CloudWatch events and seem to be related to a timing issue. It sounds like we may be running into this same issue with the "increment by 1 millisecond", but there's no way for me to confirm that's the case. We're using version 1.0.1. I don't see a ticket on this. Wondering whether there's a fix in a more recent version (don't think so, based on the code) or maybe a workaround. We use CloudWatch logs to capture our Lambda log events and we rely on this plugin to make sure we don't miss any. We're probably missing a lot, but the ERROR/FATAL log events are the ones everyone notices when they're missed. Just for some context ;-) Thanks!

from logstash-input-cloudwatch-logs.

I think at best I would prefer to expose this as a setting. 'increment_by' or something which defaults to 1.

You're right in that there is a condition where events can be dropped, however the expectation to de-duplicate could be high for initial usage, and could see many users asking "why am I getting duplicates", since de-duplication (unless I am mistaken) would have to occur at the output to elasticsearch (or whatever other output) level, by using the CWL log id as the output log id.

from logstash-input-cloudwatch-logs.

Hi,
I would like to ask if there is any update regarding this. I have faced the same(and other similar strange behaviors) events consistency with version 7.17.0 and 8.
Thanks.

from logstash-input-cloudwatch-logs.