why fluent-plugin-webhdfs not choose ruby-hdfs as hdfs data transfer component? about fluent-plugin-webhdfs HOT 4 CLOSED

You can specify open/read timeout of http request.

• open_timeout (default:30)
• read_timeout (default: 60)

Or, you can limit data transfer size with buffer_chunk_limit (size per flush and transfer) or flush_interval.

'fluent-plugin-webhdfs' is plugin for webhdfs.
Of course, you can write native hdfs plugin to write over native hdfs.

from fluent-plugin-webhdfs.

Thanks for your reply.

I know there are timeout related settings in fluent-plugin-webhdfs.

I just wanna know why fluentd choose webhdfs as transfer component(There is no other hdfs plugin for fluentd.). For convenience or performance or stable?

from fluent-plugin-webhdfs.

Native HDFS protocol is highly affected with Hadoop versions.
We cannot maintain many versions for many hadoop minor versions. On the other hand, we can expect that webhdfs protocol is stable between hadoop versions.

from fluent-plugin-webhdfs.

Got it.
Many thanks. :)

from fluent-plugin-webhdfs.