Support Docker fluentd plugin about fluent-plugin-kubernetes_metadata_filter HOT 5 CLOSED

@r4j4h Given the way this plugin derives metadata for log records, I don't see how it could properly support the Docker fluentd log driver. This plugin needs to understand the Kubernetes namespace in order to properly fetch and cache pod and namespace metadata. The current documentation indicate the driver is only providing container_name. This would require this plugin to jump through a lot of hoops to do the proper correlation if it is even possible to derive that information from the container_name and container_id.

@richm Do you see any way this could be supported in this plugin?

from fluent-plugin-kubernetes_metadata_filter.

What do the log records look like in fluentd when using docker with the fluentd log driver? Specifically, does the tag include the k8s metadata, or is there a CONTAINER_NAME field with the k8s metadata?
Do commands like docker logs and kubectl logs work when using the fluentd log driver?

from fluent-plugin-kubernetes_metadata_filter.

@jcantrill Great point!

@richm I'm not sure as the tailing approach works well enough I have not bothered to try the alternate logging driver. 😊

It does look like this plugin may need to handle the correlation anyway in light of #105 :/

from fluent-plugin-kubernetes_metadata_filter.

I'm using this fluentd conf to read logs directly from docker fluentd driver

<source>
  @type forward
  port 8888
  add_tag_prefix log
</source>

<filter log.*>
  @type record_transformer
  enable_ruby
  <record>
    custom_tag  ${record['container_name']}_${record['container_id']}
  </record>
</filter>

<match log.*>
  @type rewrite_tag_filter
  <rule>
    key custom_tag
    pattern /(?<name_prefix>[^_]+)_(?<container_name>[^\._]+)(\.(?<container_hash>[^_]+))?_(?<pod_name>[^_]+)_(?<namespace>[^_]+)_[^_]+_[^_]+_(?<docker_id>[^_]+)$/
    tag kubernetes.$4_$5_$2-$6
  </rule>
</match>

<filter kubernetes.**>
  @type kubernetes_metadata
  kubernetes_url "https://kube.api.url"
  tag_to_kubernetes_name_regexp "kubernetes\.(?<pod_name>[^_]+)_(?<namespace>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})"
  client_cert "/tmp/client.crt"
  client_key "/tmp/client.key"
  verify_ssl false
</filter>

<filter kubernetes.**>
  @type record_transformer
  remove_keys custom_tag
</filter>

<match kubernetes.**>
  @type stdout
</match>

It seems to work

from fluent-plugin-kubernetes_metadata_filter.

Closing per previous comment

from fluent-plugin-kubernetes_metadata_filter.