Compatability with Hadoop 2.3.0-cdh5.0.0 about fluent-plugin-webhdfs HOT 6 CLOSED

dfs.support.broken.append is not required for HDFS latest versions.
And your error Connection refused - connect(2) means that fluentd failed to connect on TCP layer, not on WebHDFS protocol layer. It's not problem of hadoop versions.

You should check the connectivities between your fluentd (td-agent) host and all of your cluster nodes (namenode and all of datanodes).

from fluent-plugin-webhdfs.

Hi Tagamoris,

Thanks for your response. In terms of TCP-connectivity, i am able to perform telnet and nc, so i dont think tcp-connectivity is an issue.

Is there something i might have missed in terms of testing tcp-connectivity?

Thanks in advance

Alex

from fluent-plugin-webhdfs.

How do you check tcp connectivity (in command)?

Or, can you put any data from fluentd host with curl?

curl -v -XPUT --data @file.txt "http://your.namenode.local:50070/webhdfs/v1/file.txt?op=CREATE&user.name=hdfs"
# => redirected
curl -v -XPUT --data @file.txt REDIRECTED_URL

from fluent-plugin-webhdfs.

Hi Tagomoris,

Thank you so much for your reply. I tried those cURL commands to PUT a file via webhdfs.

Basically it reports that:
Connected to datanode.host.com (my-ip) port 50075 (#0)

PUT /webhdfs/v1/foo.txt?op=CREATE HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: datanode.host.com:50075
Accept: /
Content-Length: 24
Content-Type: application/x-www-form-urlencoded

< HTTP/1.1 400 Bad Request
< Cache-Control: no-cache
< Expires: Tue, 06 May 2014 13:51:10 GMT
< Date: Tue, 06 May 2014 13:51:10 GMT
< Pragma: no-cache
< Expires: Tue, 06 May 2014 13:51:10 GMT
< Date: Tue, 06 May 2014 13:51:10 GMT
< Pragma: no-cache
< Content-Type: application/json
< Transfer-Encoding: chunked
< Server: Jetty(6.1.26)
<

• Connection #0 to host datanode.host.com left intact
• Closing connection #0
  {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"namenoderpcaddress is not specified."}}

So it is probably due to the dfs.namenode.servicerpc-address setting.

We tried it by specifying the port to 8022 and also tried the default value (empty), but both scenarios complain about the same error (namenoderpcaddress is not specified).

So are there certain gotcha's regarding HDFS settings that we need to be aware of apart from the ones mentioned here http://docs.fluentd.org/articles/out_webhdfs

Thanks in advance

Alex

from fluent-plugin-webhdfs.

Oops.... problem seems to be solved now.
It seems that if i create a directory beforehand by "hadoop fs" commands, with owner hdfs and mode 777, it actually wont let me write files to that directory.

However, if i dont create a directory beforehand, somehow it does allow me to put files in the specified paths in td-agent.conf

Thanks again for your help!

from fluent-plugin-webhdfs.

OK, I see. I'll close this issue.

from fluent-plugin-webhdfs.