Comments (3)
tagomoris
commented on June 4, 2024
The stacktrace you showed is typically caused by the missing value for the specified key message of the <format> @type single_value </format> formatter.
I'm not sure why the other compressor can work well but snappy cannot, but the value conversion from nil could be the reason.
(Other compressors don't hit the problem about value conversion from nil?)
Are you sure that the record contains the message key and its value?
If so, can I see the value of the message key? (copy and stdout plugins should be helpful)
from fluent-plugin-webhdfs.
JaeGeunBang
commented on June 4, 2024
Thank you for the reply.
I changed the .conf as below. Remove <format> @type single_value </format> formatter and add <store> @type stdout </store> for debugging.
<source>
@type kafka_group
brokers localhost:9092
consumer_group test
topics TEST
add_prefix fluentd
add_suffix log
format json
start_from_beginning false
</source>
<match fluentd.TEST.log>
@type copy
<store>
@type stdout
</store>
<store>
@type webhdfs
namenode localhost:50070
path /user/test/%Y/%m/%d/%H/%M/test-%Y%m%d%H%M.log
compress snappy
<buffer>
@type file
path /data/td-agent
flush_mode interval
flush_interval 1s
</buffer>
</store>
</match>
As a result of the test, it was confirmed that the output is as below.
2020-05-29 13:22:30.416036274 +0900 fluentd.TEST.log: {"date":"2020-05-29_13:12","topic":"test1","count":"0"}
2020-05-29 13:22:36.416878675 +0900 fluentd.TEST.log: {"date":"2020-05-29_13:12","topic":"test1","count":"3"}
2020-05-29 13:22:50.439255700 +0900 fluentd.TEST.log: {"date":"2020-05-29_13:12","topic":"test2","count":"3"}
2020-05-29 13:22:50.439408264 +0900 fluentd.TEST.log: {"date":"2020-05-29_13:12","topic":"test2","count":"0"}
2020-05-29 13:22:54 +0900 [warn]: #0 failed to communicate hdfs cluster, path: /user/test/2020/05/29/13/21/test-202005291321.log.sz
2020-05-29 13:22:54 +0900 [warn]: #0 got unrecoverable error in primary and no secondary error_class=TypeError error="no implicit conversion from nil to integer"
2020-05-29 13:22:54 +0900 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/plugin/buffer/chunk.rb:180:in `copy_stream'
2020-05-29 13:22:54 +0900 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/plugin/buffer/chunk.rb:180:in `block in write_to'
2020-05-29 13:22:54 +0900 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/plugin/buffer/file_chunk.rb:171:in `open'
2020-05-29 13:22:54 +0900 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/plugin/buffer/chunk.rb:179:in `write_to'
2020-05-29 13:22:54 +0900 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.2.4/lib/fluent/plugin/webhdfs_compressor_snappy.rb:20:in `compress'
2020-05-29 13:22:54 +0900 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.2.4/lib/fluent/plugin/out_webhdfs.rb:333:in `compress_context'
2020-05-29 13:22:54 +0900 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.2.4/lib/fluent/plugin/out_webhdfs.rb:388:in `write'
2020-05-29 13:22:54 +0900 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/plugin/output.rb:1133:in `try_flush'
2020-05-29 13:22:54 +0900 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/plugin/output.rb:1439:in `flush_thread_run'
2020-05-29 13:22:54 +0900 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
2020-05-29 13:22:54 +0900 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-05-29 13:22:54 +0900 [warn]: #0 bad chunk is moved to /tmp/fluent/backup/worker0/object_3f9075bed29c/5a6c1c8dcccdc1d38cbb85a0c4769670.log
When I change to a different format (gzip, bzip2, lzo_command), I can see that it is stored in hdfs successfully. But only Snappy does not work as above.
from fluent-plugin-webhdfs.
tagomoris
commented on June 4, 2024
Thanks for the follow-ups! But unfortunately, I can't find the clear cause of the failure now.
from fluent-plugin-webhdfs.
Related Issues (20)
- Plugin lost file buffer when setting buffer type to file and split file by hourly HOT 5
- forward error error=#<Encoding::UndefinedConversionError: "\xE6" from ASCII-8BIT to UTF-8> error_class=Encoding::UndefinedConversionError HOT 7
- Output by tag like out_file (Feature Request) HOT 3
- 根据日志中的字段指定hdfs上的路径 HOT 5
- httpFS - Do not create file if it does not exist HOT 5
- part of data missing
- output_format , just simple HOT 12
- out_webhdfs.rb's class changed from TimeSlicedOutput to Output HOT 2
- the webhdfs doesn't work HOT 6
- Kerberos Keytab example HOT 5
- Can hdfs path use ${tag} or ${record} ? just like: /data/hdfs/${tag}.#{Socket.gethostname}.log HOT 1
- Error installing fluent-plugin-webhdfs HOT 3
- HA configuration performs incorrectly HOT 4
- When try to append to file more often then default timekey, exception happens
- kerberos_keytab not authorize HOT 2
- use chunk_id in path HOT 3
- httpFS - Can not create file when it does not exist
- systemctl reload td-agent.service is not working properly with @type webhdfs
- Operation category READ is not supported in state standby
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluent-plugin-webhdfs.