Comments (6)
jimmidyson
commented on May 24, 2024
Can you check in the fluentd pod what version of this plugin the daemonset is using? Should be in the first lines of the logs.
from fluent-plugin-kubernetes_metadata_filter.
martindanko
commented on May 24, 2024
Hi @jimmidyson,
gem 'fluent-plugin-kubernetes_metadata_filter' version '0.26.2'
from fluent-plugin-kubernetes_metadata_filter.
jimmidyson
commented on May 24, 2024
I'm really sorry but I don't know what's happening with your error. I can't see where that would be coming from in this plugin. Is there any further stack trace?
from fluent-plugin-kubernetes_metadata_filter.
martindanko
commented on May 24, 2024
Hi @jimmidyson,
Here is a longer stack trace.
2017-02-28T11:22:13.231469564Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/filter_record_transformer.rb:100:in `block in filter_stream'
2017-02-28T11:22:13.231474625Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/event.rb:130:in `block in each'
2017-02-28T11:22:13.231477787Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/event.rb:129:in `each'
2017-02-28T11:22:13.231480891Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/event.rb:129:in `each'
2017-02-28T11:22:13.231484022Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/filter_record_transformer.rb:96:in `filter_stream'
2017-02-28T11:22:13.231487687Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/event_router.rb:152:in `block in emit'
2017-02-28T11:22:13.231490879Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/event_router.rb:151:in `each'
2017-02-28T11:22:13.231494051Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/event_router.rb:151:in `emit'
2017-02-28T11:22:13.231497143Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/event_router.rb:90:in `emit_stream'
2017-02-28T11:22:13.231500244Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:293:in `receive_lines'
2017-02-28T11:22:13.231503508Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:411:in `wrap_receive_lines'
2017-02-28T11:22:13.231506607Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:606:in `on_notify'
2017-02-28T11:22:13.231509680Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:437:in `on_notify'
2017-02-28T11:22:13.231512795Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:417:in `attach'
2017-02-28T11:22:13.231515825Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:190:in `setup_watcher'
2017-02-28T11:22:13.231518915Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:208:in `block in start_watchers'
2017-02-28T11:22:13.231521989Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:195:in `each'
2017-02-28T11:22:13.231524992Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:195:in `start_watchers'
2017-02-28T11:22:13.231528836Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:184:in `refresh_watchers'
2017-02-28T11:22:13.231532009Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:131:in `start'
2017-02-28T11:22:13.231537487Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/root_agent.rb:115:in `block in start'
2017-02-28T11:22:13.231540623Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/root_agent.rb:114:in `each'
2017-02-28T11:22:13.231543672Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/root_agent.rb:114:in `start'
2017-02-28T11:22:13.231546731Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/engine.rb:237:in `start'
2017-02-28T11:22:13.231740726Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/engine.rb:187:in `run'
2017-02-28T11:22:13.231749444Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/supervisor.rb:570:in `run_engine'
2017-02-28T11:22:13.231752843Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/supervisor.rb:162:in `block in start'
2017-02-28T11:22:13.231756021Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/supervisor.rb:366:in `main_process'
2017-02-28T11:22:13.231759075Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/supervisor.rb:339:in `block in supervise'
2017-02-28T11:22:13.231762196Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/supervisor.rb:338:in `fork'
2017-02-28T11:22:13.231765266Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/supervisor.rb:338:in `supervise'
2017-02-28T11:22:13.231768305Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/supervisor.rb:156:in `start'
2017-02-28T11:22:13.231771343Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
2017-02-28T11:22:13.231777380Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
2017-02-28T11:22:13.231780591Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
2017-02-28T11:22:13.231783703Z 2017-02-28 11:22:13 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/bin/fluentd:5:in `<top (required)>'
2017-02-28T11:22:13.231787077Z 2017-02-28 11:22:13 +0000 [warn]: /usr/bin/fluentd:22:in `load'
2017-02-28T11:22:13.231790138Z 2017-02-28 11:22:13 +0000 [warn]: /usr/bin/fluentd:22:in `<main>'
2017-02-28T11:22:13.232724603Z 2017-02-28 11:22:13 +0000 [warn]: failed to expand `%Q[#{kubernetes["labels"]}]` error_class=NameError error="undefined local variable or method `kubernetes'"
Below is my kubernetes specific configuration in my fluent.conf file (if that helps).
<filter kubernetes.var.log.containers.*.log>
type kubernetes_metadata
</filter>
<filter kubernetes.**>
@type record_transformer
enable_ruby
<record>
topic ${kubernetes["pod_labels"]}
</record>
</filter>
<match kubernetes.**>
@type cloudwatch_logs
log_group_name "#{ENV['CW_LOG_GROUP']}"
log_stream_name_key topic
remove_log_group_name_key true
auto_create_stream true
put_log_events_retry_limit 20
</match>
from fluent-plugin-kubernetes_metadata_filter.
martindanko
commented on May 24, 2024
Hi @jimmidyson,
Thanks for you help on this issue, I solved the problem by replacing
topic ${kubernetes["pod_labels"]}
with
kubehost ${record.fetch("kubernetes", Hash.new).fetch("labels", "unknown_label")
is it possible to filter via kubernetes services? Just wondering. Thanks!
from fluent-plugin-kubernetes_metadata_filter.
jimmidyson
commented on May 24, 2024
Really glad that you got it working :)
Sorry no this is raw pod logs, you would have to do your own correlation to services via labels in queries to your store (Elasticsearch?).
I'm going to close this now that you've got it working.
from fluent-plugin-kubernetes_metadata_filter.
Related Issues (20)
- Remove deprecated journal functionality
- Optimize cache to use event time
- Stop marshaling and unmarshaling meta
- fix dumping of stats and allow configuration to disable it
- [info]: [filter_kube_metadata] Exception encountered parsing namespace watch event. The connection might have been closed. Sleeping for 1 seconds and resetting the namespace watcher.error reading from socket: Connection reset by peer HOT 2
- watch_retry_interval & watch_retry_exponential_backoff_base are not documented HOT 1
- is a changelog maintained anywhere? HOT 2
- Failure while installing fluent-plugin-kubernetes_metadata_filter plugin version 2.10.0 HOT 4
- 410 Gone encountered. Restarting pod watch to reset resource versions HOT 7
- gemfile.lock fluentd version in v2.13.0 HOT 2
- Compatibility with Ruby >= 3 ? HOT 1
- Support Fluentd 1.16.0 HOT 1
- How to debug problem if kubernetes_metadata not add any metadata from k8s? HOT 1
- undefined method `get_namespaces' for k8s client HOT 2
- v2.x release with fluentd 1.16.x support HOT 3
- pod-id filled with null bytes HOT 1
- New version of the plugin HOT 1
- can add "Controlled By" field to meta HOT 1
- Elasticsearch k8s labels issue HOT 3
- Logs with exception connection to api server HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluent-plugin-kubernetes_metadata_filter.