Comments (11)
Yep. Use the record_transformer with the rewrite_tag_filter plugins like so:
# Query the API for extra metadata.
<filter kubernetes.**>
type kubernetes_metadata
# If the logs begin with '{' and end with '}' then it's JSON so merge
# the JSON log field into the log event
merge_json_log true
preserve_json_log true
</filter>
# rewrite_tag_filter does not support nested fields like
# kubernetes.container_name, so this exists to flatten the fields
# so we can use them in our rewrite_tag_filter
<filter kubernetes.**>
@type record_transformer
enable_ruby true
<record>
kubernetes_namespace_container_name ${record["kubernetes"]["namespace_name"]}.${record["kubernetes"]["container_name"]}
</record>
</filter>
# retag based on the namespace and container name of the log message
<match kubernetes.**>
@type rewrite_tag_filter
# Update the tag have a structure of kube.<namespace>.<containername>
rewriterule1 kubernetes_namespace_container_name ^(.+)$ kube.$1
</match>
# Remove the unnecessary field as the information is already available on
# other fields.
<filter kube.**>
@type record_transformer
remove_keys kubernetes_namespace_container_name
</filter>
# Parse logs in the kube-system namespace using the kubernetes formatter.
<filter kube.kube-system.**>
type parser
format kubernetes
reserve_data true
key_name log
suppress_parse_error_log true
</filter>
The filter at the bottom is an example of matching by namespace, you would match the same way with your output plugin.
from fluent-plugin-kubernetes_metadata_filter.
BOOM! Worked perfectly! Thanks so much for the quick and complete response!
from fluent-plugin-kubernetes_metadata_filter.
Thanks @chancez!
from fluent-plugin-kubernetes_metadata_filter.
I liked your approach and added some Go code to automate the boring stuff. You can have a look at it here: https://github.com/vmware/kube-fluentd-operator
I've taken the tag-rewriting to the extreme and at the namespace level you can now target a container in a pod based on container labels:
<match $labels(app=nginx)>
@type ...
</match>
I'd be happy to get your feedback.
Thanks,
Julian
from fluent-plugin-kubernetes_metadata_filter.
<match $labels(app=nginx)>
does this syntax work with fluentd 0.12? We still have to support that version of fluentd.
from fluent-plugin-kubernetes_metadata_filter.
$labels
is actually a macro: it gets translated to a couple of tag-rewriting <match>
directives internally.
As such, it will work with older versions of Fluentd but only in the context of kube-fluentd-operator.
from fluent-plugin-kubernetes_metadata_filter.
$labels is actually a macro: it gets translated to a couple of tag-rewriting directives internally.
Translated by whom? By fluentd? Do you run this through some sort of pre-processor?
from fluent-plugin-kubernetes_metadata_filter.
@richm If you read his first comment and most recent one he's specifically referring to the kube-fluentd-operator doing the preprocessing.
from fluent-plugin-kubernetes_metadata_filter.
is there any ways to restrict kube-system namespace logs in fluentd conf?
from fluent-plugin-kubernetes_metadata_filter.
Hi @chancez,
Our scenario does not have a fluentd interface for logs and we would like to create these in Cloud watch. What changes needs to be the done to the code mentioned above? Please advice. Thanks
from fluent-plugin-kubernetes_metadata_filter.
Hi @chancez,
Our scenario does not have a fluentd interface for logs and we would like to create these in Cloud watch. What changes needs to be the done to the code mentioned above? Please advice. Thanks
What does this mean? " fluentd interface for logs"
from fluent-plugin-kubernetes_metadata_filter.
Related Issues (20)
- Stop marshaling and unmarshaling meta
- fix dumping of stats and allow configuration to disable it
- [info]: [filter_kube_metadata] Exception encountered parsing namespace watch event. The connection might have been closed. Sleeping for 1 seconds and resetting the namespace watcher.error reading from socket: Connection reset by peer HOT 2
- watch_retry_interval & watch_retry_exponential_backoff_base are not documented HOT 1
- is a changelog maintained anywhere? HOT 2
- Failure while installing fluent-plugin-kubernetes_metadata_filter plugin version 2.10.0 HOT 4
- 410 Gone encountered. Restarting pod watch to reset resource versions HOT 7
- gemfile.lock fluentd version in v2.13.0 HOT 2
- Compatibility with Ruby >= 3 ? HOT 1
- Support Fluentd 1.16.0 HOT 1
- How to debug problem if kubernetes_metadata not add any metadata from k8s? HOT 1
- undefined method `get_namespaces' for k8s client HOT 2
- v2.x release with fluentd 1.16.x support HOT 3
- pod-id filled with null bytes HOT 1
- New version of the plugin HOT 1
- can add "Controlled By" field to meta HOT 1
- Elasticsearch k8s labels issue HOT 3
- Logs with exception connection to api server HOT 5
- Connection Reset By Peer HOT 1
- Dreaded "can't add a new key into hash during iteration" again
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluent-plugin-kubernetes_metadata_filter.