plugin sometimes stops uploading about logstash-output-s3 HOT 9 OPEN

Can you double check if its related to #55
?
Do you have time_file and/or size_file configured?

from logstash-output-s3.

I have time_file set to 1 minute.
There are no errors what so ever in the logs, so i can't tell you exactly what happens, but i can tell you for sure that the plugin continues to work, but is not uploading.
The plugin keeps writing events to the temp file, but never uploads them.
I think that maybe if the connection to S3 is broken in the middle of the script running, than it won't reconnect and you don't catch that error.

from logstash-output-s3.

I am seeing the same behavior, happens with both time_file and size_file.

I have tried each individually and also together (small file size and longer time) just to see if the size didn't catch it the time one would, but still having the issue

from logstash-output-s3.

So it looks like the problem is with how you are setting up the worker threads. They don't stay alive after they process an upload. I tested this by setting the number of upload workers, and every time I get that number of files and then things start queuing on disk.

logstash 2.1.1
logstash-output-s3-2.0.3

from logstash-output-s3.

We seam to have the exact problem reported by zot420.

Is there any advice here? At least a work around?

Thanks!

from logstash-output-s3.

Is this issue resolved ??

from logstash-output-s3.

Unfortunately i have no idea about the bug and it's resolution. However a quick work around for me was to restart logstash in case the folder start filling up more than expected.

I have nothing to share as we are currently migrating to another solution, therefor did no automation script (aside from standard disk monitoring) to manage that.

Hope this help at least a little.

from logstash-output-s3.

My team is also running into this bug consistently. I don't think it is related to workers though - we have multiple identical machines, but one of them stopped uploading at 998 files, and the other one is at 1400 files but is still working correctly. My guess is that the S3 connection returns some sort of unexpected error, and the plugin doesn't handle it correctly. It must be the S3 upload part - the file rotation still works as expected, just that the file doesn't get uploaded.

Restarting logstash does work, but is obviously not a good solution. We will likely look into alternatives until this is fixed.

from logstash-output-s3.

I am testing logstash under docker with this plugin under localstack.
I ran a test case and found there are files under /tmp/logstash, but they are not uploaded to s3.
When I restart logstash (kill -TERM) I see the tmplogstash is now empty; then I checked the s3 bucket and indeed there are docs there. The /tmp/logstash is still empty. It seems like this plugin has some state/buffering/flushing problems.

Here is my output config file:

output {
    s3 {

        endpoint                   => "http://localstack:4566"
        access_key_id              => "test"
        secret_access_key          => "test"
        additional_settings => { "force_path_style" => true }
        validate_credentials_on_root_bucket => false
        region => "us-east-1"
        bucket => "em-top-archive-us-east-1-local-localstack"
        codec => "json"
        canned_acl => "private"
        prefix => "year=%{[@metadata][index_year]}/month=%{[@metadata][index_month]}/day=%{[@metadata][index_day]}/type=%{[@metadata][index]}/project=%{[project.name]}/environment=%{[ecsenv]}"
    }
}

from logstash-output-s3.