Plugin lost file buffer when setting buffer type to file and split file by hourly about fluent-plugin-webhdfs HOT 5 CLOSED

Once you specify flush_interval, time_slice_wait is ignored, and your chunk will be flushed after flush_interval(1 hour by your configuration) from chunk generation. (this is fluentd's normal behavior.)
So you should wait to see whether chunk object is successfully flushed or not until 20:57 for last chunk file.

Do you understand this behavior?

from fluent-plugin-webhdfs.

I know it, even using flush_interval with default setting(default is 60s), the file buffer of last hour do not upload until 10 minute in new hour, like this:
write a data(1) at 8:56, and the buffer file(buffer-2015052808.xxxx) lower than 1G at 8:59:59

the data(1) in buffer flush data to hdfs until 9:11.

how about upload the buffer before new buffer using? I means flush data to hdfs in real time(less than 1min) and split file into 1G size.

from fluent-plugin-webhdfs.

Are you willing to write data per 1GB or per 1minute, right?
If so, you should configure like this:

• append false
• path /path/to/dir/filename-including-minute.%Y%m%d_%H%M.${hostname}.${chunk_id}.log
• buffer_chunk_limit 1g
• time_slice_wait 1m

If you wanna split files per hour with per-minute flush, you should enable append.

from fluent-plugin-webhdfs.

Sorry, I made a mistake. As you said above, just add config time_slice_wait 1m, the file will write into per hour with max size limit 1G.

The buffer(like data(1) which i referred) will flush to hdfs at 9:01

Thank you very much.

from fluent-plugin-webhdfs.

Good to hear that 😄

from fluent-plugin-webhdfs.