Error reporting missing? about fluent-plugin-webhdfs HOT 10 CLOSED

Many of webhdfs request failures make HTTP error response, and it will be raised as fluentd errors.
Can you post these below here?:

• your configurations of hadoop's hdfs-site.xml (of course, with masking of sensitive parts)
• output of curl -v webhdfs operation result without username

from fluent-plugin-webhdfs.

Ah, forgot to mention that. When I run curl, it accepts the 307 redirect (to a datanode) from Hadoop, then stops. I don't know whether it actually sends the request to the datanode. When I send the redirect URL to the datanode, that works.

I do not have access to the webhdfs system right now.

This is Hadoop version 1.0.3.

from fluent-plugin-webhdfs.

It started working when I added the username parameter. Is there a bug in propagating this error?

from fluent-plugin-webhdfs.

Hmm, i cannot understand why curl without username sucesses, and fluent-plugin-webhdfs without username failes.
Can you try this operation on irb from your client?

require 'webhdfs'
client = WebHDFS::Client.new('your.namenode.hostname', 50070)
client.list('/')
client.append('/tmp/test_file_webhdfs_client.txt', 'aaaaaaaaaaaaaaaaaaaaaaaa')

from fluent-plugin-webhdfs.

I'm on centos with ruby 0.9.5. 'gem install webhdfs' gets a buffer overflow. I don't know how to get the system to where your test wants it to be.

from fluent-plugin-webhdfs.

What do you do? Ruby '0.9.5' may not exists in this world...

$ which ruby
$ ruby -v
$ which gem
$ gem -v

from fluent-plugin-webhdfs.

Hello.
Can I ask some question about this module?

I started to try to use this module,
but I couldn't recognize my data appended into hdfs directory.
When I restart td-agent,only at that time, some data in td-agent buffer is correctly wrote to hdfs directry,
But, when I keep td-agent started, td-agent write nothing to HDFS directory.(new data is continuously comming to this server)
So,appending looks like not working collectly.
Would you have some idea to work collectly?

Below is my td-agent.conf

machine A(data forwerder)

type forward
host machineB

machine B(reciever and writer to hdfs)

and all of hdfs cluster below is set true
-dfs.webhdfs.enabled
-dfs.support.append

-dfs.support.broken.append

from fluent-plugin-webhdfs.

@satousan:
specify flush_interval 1s instead of flash_interval (and please make new issue for more).

@LanceNorskog ping?

from fluent-plugin-webhdfs.

thanks ! for your help

from fluent-plugin-webhdfs.

Please re-open this issue if you have any problems now > @LanceNorskog

from fluent-plugin-webhdfs.