Giter Club home page Giter Club logo

fluentd-kubernetes-daemonset's Introduction

Fluentd Daemonset for Kubernetes

Caution

README.md is generated from templates/README.md.erb

Docker Stars Docker Pulls ImageLayers Size ImageLayers Layers

Supported tags and respective Dockerfile links

See also dockerhub tags page: https://hub.docker.com/r/fluent/fluentd-kubernetes-daemonset/tags

Debian

Current stable

Tip

Since v1.17.0, the container image build process has been migrated from automated builds on hub.docker.com to GitHub Actions. This is because there were limitation about the number of automated builds on hub.docker.com. Now, there is no limitation about the number of build pipelines.

Note that there were some restrictions to ship daemonset images for v1.16.5 or older ones before:

  • papertrail, syslog images (x86_64/arm64) won't be published anymore
  • logentries, loggly, logzio, s3 arm64 images won't be published anymore (x86_64 only supported) If you want to use above non published images, build it by yourself. Dockerfile itself is still maintained in this repository.
Multi-Arch images
  • Azureblob
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-azureblob-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-azureblob-1
  • Elasticsearch8
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-elasticsearch8-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-elasticsearch8-1
  • Elasticsearch7
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-elasticsearch7-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-elasticsearch7-1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
  • Opensearch
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-opensearch-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-opensearch-1
  • Cloudwatch
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-cloudwatch-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-cloudwatch-1
  • Forward
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-forward-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-forward-1
  • Gcs
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-gcs-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-gcs-1
  • Graylog
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-graylog-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-graylog-1
  • Kafka
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-kafka-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-kafka-1
  • Kafka2
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-kafka2-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-kafka2-1
  • Kinesis
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-kinesis-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-kinesis-1
x86_64 images
  • Azureblob Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-azureblob-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-azureblob-amd64-1
  • Elasticsearch8 Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-elasticsearch8-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-elasticsearch8-amd64-1
  • Elasticsearch7 Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-elasticsearch7-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-elasticsearch7-amd64-1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch-amd64
  • Opensearch Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-opensearch-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-opensearch-amd64-1
  • Loggly Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-loggly-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-loggly-amd64-1
  • Logentries Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-logentries-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-logentries-amd64-1
  • Cloudwatch Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-cloudwatch-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-cloudwatch-amd64-1
  • S3 Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-s3-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-s3-amd64-1
  • Syslog Dockerfile
  • Forward Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-forward-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-forward-amd64-1
  • Gcs Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-gcs-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-gcs-amd64-1
  • Graylog Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-graylog-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-graylog-amd64-1
  • Papertrail Dockerfile
  • Logzio Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-logzio-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-logzio-amd64-1
  • Kafka Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-kafka-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-kafka-amd64-1
  • Kafka2 Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-kafka2-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-kafka2-amd64-1
  • Kinesis Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-kinesis-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-kinesis-amd64-1
arm64 images
  • Azureblob Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-azureblob-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-azureblob-arm64-1
  • Elasticsearch8 Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-elasticsearch8-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-elasticsearch8-arm64-1
  • Elasticsearch7 Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-elasticsearch7-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-elasticsearch7-arm64-1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch-arm64
  • Opensearch Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-opensearch-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-opensearch-arm64-1
  • Loggly Dockerfile
  • Logentries Dockerfile
  • Cloudwatch Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-cloudwatch-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-cloudwatch-arm64-1
  • S3 Dockerfile
  • Syslog Dockerfile
  • Forward Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-forward-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-forward-arm64-1
  • Gcs Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-gcs-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-gcs-arm64-1
  • Graylog Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-graylog-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-graylog-arm64-1
  • Papertrail Dockerfile
  • Logzio Dockerfile
  • Kafka Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-kafka-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-kafka-arm64-1
  • Kafka2 Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-kafka2-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-kafka2-arm64-1
  • Kinesis Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17.0-debian-kinesis-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.17-debian-kinesis-arm64-1

You can also use v1-debian-PLUGIN tag to refer latest v1 image, e.g. v1-debian-elasticsearch. On production, strict tag is better to avoid unexpected update.

See dockerhub's tags page for older tags.

Old stable

v0.12 development has been ended. These images are never updated.

Alpine Linux (This is deprecated. Use Debian images instead)

What is Fluentd?

Fluentd Logo

Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.

www.fluentd.org

Image versions

Fluentd versioning is as follows:

Series Description
v1.x current stable
v0.12 Old stable, no longer updated

Settings

Default image version

Default YAML uses latest v1 images like fluent/fluentd-kubernetes-daemonset:v1-debian-kafka. If you want to avoid unexpected image update, specify exact version for image like fluent/fluentd-kubernetes-daemonset:v1.8.0-debian-kafka-1.0.

Run as root

This is for v0.12 images.

In Kubernetes and default setting, fluentd needs root permission to read logs in /var/log and write pos_file to /var/log. To avoid permission error, you need to set FLUENT_UID environment variable to 0 in your Kubernetes configuration.

Use your configuration

These images have default configuration and support some environment variables for parameters but it sometimes doesn't fit your case. If you want to use your configuration, use ConfigMap feature.

Each image has following configurations:

  • fluent.conf: Destination setting, Elaticsearch, kafka and etc.
  • kubernetes.conf: k8s specific setting. tail input for log files and kubernetes_metadata filter
  • tail_container_parse.conf: parser setting for /var/log/containers/*.log. See also "Use CRI parser for containerd/cri-o" logs section
  • prometheus.conf: prometheus plugin for fluentd monitoring
  • systemd.conf: systemd plugin for collecting systemd-journal log. See also "Disable systemd input" section.

Overwrite conf file via ConfigMap. See also several examples:

Use CRI parser for containerd/cri-o logs

This feature is available since v1.12.0-xxx-1.1.

By default, these images use json parser for /var/log/containers/ files because docker generates json formatted logs. On the other hand, containerd/cri-o use different log format. To parse such logs, you need to use cri parser instead.

You can use cri parser by overwriting tail_container_parse.conf via ConfigMap.

# configuration example
<parse>
  @type cri
</parse>

See also CRI parser README

Use FLUENT_CONTAINER_TAIL_PATH to change container logs folder

You can update the default path for the container logs i.e /var/log/container/*.log and also one can add multiple path as defined in this fluentd document https://docs.fluentd.org/input/tail#path

Use FLUENT_CONTAINER_TAIL_EXCLUDE_PATH to exclude specific container logs

Since v1.9.3 or later images.

You can exclude container logs from /var/log/containers/ with FLUENT_CONTAINER_TAIL_EXCLUDE_PATH. If you have a trouble with specific log, use this envvar, e.g. ["/var/log/containers/logname-*"].

Disable systemd input

If you don't setup systemd in the container, fluentd shows following messages by default configuration.

[warn]: #0 [in_systemd_bootkube] Systemd::JournalError: No such file or directory retrying in 1s
[warn]: #0 [in_systemd_kubelet] Systemd::JournalError: No such file or directory retrying in 1s
[warn]: #0 [in_systemd_docker] Systemd::JournalError: No such file or directory retrying in 1s

You can suppress these messages by setting disable to FLUENTD_SYSTEMD_CONF environment variable in your kubernetes configuration.

Disable prometheus input plugins

By default, latest images launch prometheus plugins to monitor fluentd. You can disable prometheus input plugin by setting disable to FLUENTD_PROMETHEUS_CONF environment variable in your kubernetes configuration.

Disable sed execution on elasticsearch image

This is for older images. Latest elasticsearch images don't use sed.

By historical reason, elasaticsearch image executes sed command during startup phase when FLUENT_ELASTICSEARCH_USER or FLUENT_ELASTICSEARCH_PASSWORD is specified. This sometimes causes a problem with read only mount. To avoid this problem, set "true" to FLUENT_ELASTICSEARCH_SED_DISABLE environment variable in your kubernetes configuration.

Running on OpenShift

This daemonset setting mounts /var/log as service account fluentd so you need to run containers as privileged container. Here is command example:

oc project kube-system
oc create -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch-rbac.yaml
oc adm policy add-scc-to-user privileged -z fluentd
oc patch ds fluentd -p "spec:
  template:
    spec:
      containers:
      - name: fluentd
        securityContext:
          privileged: true"
oc delete pod -l k8s-app=fluentd-logging

This is from nekop's japanese article.

Multiple fluentd

When you want to have multiple fluentd. For example push to multiple destination like: elsticsearch + S3. You need to use FLUENT_POS_EXTRA_DIR add additional directory for pos file. Otherwise they share same pos file. You may found some log only on one destination.

Note

kafka image doesn't support zookeeper parameters

zookeeper gem doesn't work on Debian 10, so kafka image doesn't include zookeeper gem.

Windows k8s daemonset not supported in this repository

Maintainers don't have k8s experience on Windows. Some users create k8s daemonset on Windows:

Please check them out.

kafka image suggestion

Using debian-kafka2/debian-kafka2-arm64 images are better than using debian-kafka/debian-kafka-arm64 images. Because debian-kafka2/debian-kafka2-arm64 images use out_kafka2 plugin but debian-kafka/debian-kafka-arm64 images use deprecated out_kafka_buffered plugin.

Maintainers

Some images are contributed by users. If you have a problem/question for following images, ask it to contributors.

  • azureblob : @elsesiy
  • papertrail : @alexouzounis
  • kafka : @erhudy
  • graylog : @rtnpro
  • gcs : @andor-pierdelacabeza
  • Amazon Kinesis : @shiftky
  • logz.io : @SaMnCo / @jamielennox
  • splunkhec: @FutureSharks

Currently, we don't accept new destination request without contribution. See #293

References

Kubernetes Logging with Fluentd

Issues

We can't notice comments in the DockerHub so don't use them for reporting issues or asking question.

If you have any problems with or questions about this image, please contact us through a GitHub issue.

Pull Request

Update templates files instead of docker-image files. docker-image files are automatically generated from templates.

Note: This file is generated from templates/README.md.erb

fluentd-kubernetes-daemonset's People

Contributors

aerostitch avatar alikhil avatar andor-pierdelacabeza avatar arvind-naidu avatar ashie avatar blaketastic2 avatar capitrium avatar carlosjgp avatar cosmo0920 avatar dependabot[bot] avatar edsiper avatar elsesiy avatar errm avatar gerkindev avatar keigohtr avatar kenhys avatar keperry avatar magnitus- avatar max-rocket-internet avatar nilsga avatar pyama86 avatar repeatedly avatar rtnpro avatar samnco avatar shinebayar-g avatar stuh84 avatar thiagophx avatar twalla avatar vigohe avatar yukirii avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

42company zhaosijun jrnt30 arvi3411301 augustgerro sportebois gandalfhq lizhede jmalouf69 tothandras chancez coreos kenden kz8s illuminateeducation dsapala dshmelev manfrid galacticfog innyso femiagbabiaka lonelyplanet while1eq1 acalephstorage alexbrand justicel juliusgawlas codesuki thiagophx zopanix cknowles leafly-com robertwol keidrych deitch smarsh ii ahawkins cleverlzc dileepfrog marcosamorim sbij marc-sensenich amitkumarj441 saidimu constantinevassil allanharris waeljammal loganbfisher renzroque vvakame ds17f charanm1985 withnale crowdflower iblockslimited msvbhat spinezhang alexouzounis jrottenberg iamplus madeden zacheddy inno-ron nnvema taylorshaulis vic0din ninadpage mitch000001 triduongtran rtnpro iterion stevenacoffman tmotabastos jimmyadepeju surfraz evaneos kivagant jyoungs cerilawley davide-talesco onedata invia-de vshimoda wangsz richmondwang denverwilliams aguedes-pivotal pirankashani stelminator andor-pierdelacabeza flexdrive yantang-msft lzbgt hpcloud flasheryu onlonely dubuc rishabhsverma93 proofpoint

fluentd-kubernetes-daemonset's Issues

401 Unauthorized on a cluster with X-Pack enabled

I'm experiencing following issue with this image...

2017-05-03T15:08:36.192683683Z 2017-05-03 15:08:36 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-05-03 15:09:06 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Unauthorized" error="[401] " plugin_id="object:2b1bd5d5c250"

I do have the following config in place

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v0.12-elasticsearch
        resources:
          requests:
            cpu: 100m
            memory: 200Mi
          limits:
            memory: 200Mi
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch"
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        - name: FLUENT_ELASTICSEARCH_USER
          value: "elastic"
        - name: FLUENT_ELASTICSEARCH_PASSWORD
          value: "changeme"
        volumeMounts:
        - name: fluentconfig
          mountPath: /home/fluent/fluentd/etc
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluentconfig
        configMap:
          name: fluent-config

In order to handle the FLUENT_ELASTICSEARCH_USER and FLUENT_ELASTICSEARCH_PASSWORD I have added the following configmap.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-config
  namespace: kube-system
data:
  fluent.conf: |
    @include kubernetes.conf

    <match **>
      type elasticsearch
      log_level info
      include_tag_key true
      host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
      port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
      user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
      password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
      logstash_format true
      buffer_chunk_limit 2M
      buffer_queue_limit 32
      flush_interval 5s
      max_retry_wait 30
      disable_retry_limit
      num_threads 8
    </match>
  kubernetes.conf: |
    <match fluent.**>
      type null
    </match>

    <source>
      type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      tag kubernetes.*
      format json
      read_from_head true
    </source>

    ..................
    ................
    .... remainder of kubernetes.conf left for brevity

Is there anyone else experiencing this issue, or knows how to handle the BASIC auth properly?
Is it actually supported by the elasticsearch plugin on this image?

Specify an Index for ES

Apologies if I missed this, but I've been trying to figure out how to specify an elasticsearch index value in the .yaml file. Is there an environment variable for that?

Permission problem access /var/log/containers/*

This issue must have come up before so apologies for not finding it.

My fluentd containers are now run as user '1000' and so don't have read access to /var/lib/docker/containers which is owned by root and has 0700 perms.

I get the log error mesg:

unreadable. It is excluded and would be examined next time.

I can't seem to figure out how to run my container as root or grant the necessary perms on the volume. Adding a securitycontext didn't work

I'm a kubernetes beginner so hand holding is much appreciated.

Here's my daemonset:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      securityContext:
        runAsNonRoot: false
        runAsUser: 0
      containers:
      - name: fluentd
        image: <privaterepo>/fluentd-syslog-ng
        securityContext:
          runAsNonRoot: false
          runAsUser: 0
        imagePullPolicy: Always
        env:
          - name: SYSLOG_SERVER
            value: "SYSLOGSERVER"
          - name: SYSLOG_PORT
            value: "516"
          - name: API_SERVER
            value: "https://FQDN"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: vartmp
          mountPath: /var/tmp
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: vartmp
        hostPath:
          path: /var/tmp
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Fluentd cloudwatch log forwarder for kubernetes - Problem

I have installed the cloudwatch log forwarder and set it up to forward logs from my kubernetes cluster to cloudwatch. I have setup a log group called "kubernetes" and a IAM user with CloudWatchFullAccess. I installled using helm install --name my-release --set awsRegion=eu-west-1
(eu-west-1 is the region os my cluster and I can access cloudwatch in this region eu(Ireland) )
However I am not getting any log into cloudwatch.
Is there a few steps I am missing. Do I need to login with the AWS user with a policy to access Cloudwatch ?

List of forks

Maybe create a list of Forks for different logging targets?

if they are all added to the Kubernetes Charts repo, end users can quickly get started using fluentd to ship to their logging service of preference.

my PR for logentries-fluentd Chart: helm/charts#779

I can do a PR with a Markdown document to list these (or add to the main README) @edsiper, what do you think?

Re-design how Fluentd should be deployed as a Daemonset

Current structure of this suggested way to deploy Fluentd as a daemonset is generating some complications to maintain different based distro, plugins and different setups.

Taking in count @chancez suggestions, I think this could work as:

  • One debian base image:
    • Deprecate Alpine because of: cannot work with systemd, hard to maintain with Jemalloc enabled (required to reduce memory usage because of high glibc memory fragmentation).
    • Debian image by Google (gcr.io/google-containers/debian-base-amd64:0.1) is really secure,
  • Use ConfigMaps instead of configuration files inside the images
  • Similar documentation to the one existing by CoreOS/Tectonic: Logging

Please share your thoughts and ideas.

container logs are not getting update continuously

Hi ,
i have configured kubernetes cluster in AWS environment using KOPS.
i have used the below git hub (fluentd-kubernetes-daemonset/docker-image/v0.12/debian-cloudwatch/).
i am trying to send application & kubernetes logs to cloud watch through fluentd. using fluent-plugin-cloudwatch-logs -v 0.4.0 plugin. however the application & cluster level logs stopped updating in cloudwatch within 10-15 mins.

it doesn't work even recreating the fluentd pod.
help me to fix this issue.

Docker image does not build due to lack of /bin/sh

Building the docker image results in errors:

# docker build .
Sending build context to Docker daemon 11.26 kB
Step 1/11 : FROM fluent/fluentd:v0.12-onbuild
# Executing 2 build triggers...
Step 1/1 : COPY fluent.conf /fluentd/etc/
 ---> Using cache
Step 1/1 : COPY plugins /fluentd/plugins/
 ---> Using cache
 ---> 14ec4026e25b
Step 2/11 : MAINTAINER Eduardo Silva <[email protected]>
 ---> Running in 03b78de3d801
 ---> fef1e854802f
Removing intermediate container 03b78de3d801
Step 3/11 : USER root
 ---> Running in 6e106da769a9
 ---> d5bbf8093c4a
Removing intermediate container 6e106da769a9
Step 4/11 : WORKDIR /home/fluent
 ---> 995a21b0438a
Removing intermediate container 244a814f84bf
Step 5/11 : ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH
 ---> Running in dee529338d96
 ---> 584670a50962
Removing intermediate container dee529338d96
Step 6/11 : RUN apk --no-cache --update add                             build-base                             ruby-dev &&     echo 'gem: --no-document' >> /etc/gemrc &&     gem install fluent-plugin-secure-forward &&     gem install fluent-plugin-record-reformer &&     gem install fluent-plugin-elasticsearch &&     gem install fluent-plugin-kubernetes_metadata_filter &&     apk del build-base ruby-dev &&     rm -rf /tmp/* /var/tmp/* /var/cache/apk/* /usr/lib/ruby/gems/*/cache/*.gem
 ---> Running in eb277b33fbd2
container_linux.go:247: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory"
oci runtime error: container_linux.go:247: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory"

It fails on the following line of the Dockerfile due to lack of /bin/sh in base image:

RUN apk --no-cache --update add \
							build-base \
							ruby-dev && \
	echo 'gem: --no-document' >> /etc/gemrc && \
	gem install fluent-plugin-secure-forward && \
	gem install fluent-plugin-record-reformer && \
	gem install fluent-plugin-elasticsearch && \
	gem install fluent-plugin-kubernetes_metadata_filter && \
	apk del build-base ruby-dev && \
	rm -rf /tmp/* /var/tmp/* /var/cache/apk/* /usr/lib/ruby/gems/*/cache/*.gem

Can't connect to AWS ES

I have an AWS ES instance protected by an IAM access policy. I added all my cluster nodes to said policy (by IP). Since I'm not using X-Pack, I figured I had to set the values for FLUENT_ELASTICSEARCH_USER and FLUENT_ELASTICSEARCH_PASSWORD to blank in the DaemonSet config file (fluentd-daemonset-elasticsearch.yml). Still, fluentd was unable to connect to ES:

2017-06-13 17:18:32 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-06-13 17:19:02 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:2ac3e37a42a8"

To confirm that my IAM policy was correctly set up, I curl'd my ES host from a fluentd pod and I got a successful response.

/home/fluent # curl https://<redacted>.us-west-2.es.amazonaws.com:443
{
  "name" : "Crimson Cavalier",
  "cluster_name" : "<redacted>",
  "version" : {
    "number" : "2.3.2",
    "build_hash" : "72aa8010df1a4fc849da359c9c58acba6c4d9518",
    "build_timestamp" : "2016-11-14T15:59:50Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

The solution I found was to manually edit the /fluentd/etc/fluent.conf file, remove the user and password lines and start the fluentd service again.

-   user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
-   password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"

I'm not sure what's the best way to get this into the code though.

ES Cluster value

Does the environment var FLUENT_ELASTICSEARCH_HOST support pointing to multiple ES instances?

example:
FLUENT_ELASTICSEARCH_HOST : es01, es02, es03

fluent-gem install doable?

I found a useful plugin (https://docs.fluentd.org/v0.12/articles/out_rewrite_tag_filter).
When trying to use the plugin, I get the following error: config error file="MY_FILENAME" error="Unknown output plugin 'rewrite_tag_filter'. Run 'gem search -rd fluent-plugin' to find plugins"

The github-documentation states that I (might) have to install it via fluent-gem install.

I don't know how to install the plugin in the container.
Can I use that plugin in combination with your docker-container in a simple way, or do I have to create an own docker image, where the plugin is installed?

Use "hosts" instead of "host" in config file template

References #7

The underlying plugin supports multiple targets, but the default config does not.

I'm not sure why they differentiate between singular and plural in a config-file, but simply switching the config option from «host "$blatti"» to «hosts "blatti"» is all that's needed to make this work as expected.

As for the use case, that's simple: Avoiding single point of failure with off-cluster elk installations.

Unique tags for docker images

I had a kubernetes cluster of two nodes.

fluent/fluentd-kubernetes-daemonset was collecting logs to elasticsearch.

      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:elasticsearch

Everything was fine.

Then I added another node to the cluster and on this new node fluentd didn't want to start. Error was:

2018-01-24 09:02:32 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="start_namespace_watch: Exception encountered setting up namespace watch from Kubernetes API v1 endpoint https://10.96.0.1:443/api: namespaces is forbidden: User "system:serviceaccount:kube-system:fluentd" cannot list namespaces at the cluster scope ({"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces is forbidden: User \"system:serviceaccount:kube-system:fluentd\" cannot list namespaces at the cluster scope","reason":"Forbidden","details":{"kind":"namespaces"},"code":403}\n)"

Output of kubectl get pods:

fluentd-4gmfg                 0/1       CrashLoopBackOff   31         1h        10.244.0.116      h5
fluentd-npst8                 1/1       Running            0          1h        10.244.1.43       h13
fluentd-pnkxh                 1/1       Running            0          1h        10.244.2.72       h11

After some digging and troubleshouting I found that the new node was running different version of fluent/fluentd-kubernetes-daemonset:elasticsearch image.

Image on the old nodes:

$ docker images | grep fluentd-kubernetes-daemonset
fluent/fluentd-kubernetes-daemonset                           elasticsearch                              c04d7276b4a1        3 months ago        60.3MB

Image on the new node:

fluent/fluentd-kubernetes-daemonset                        elasticsearch       f85edd2602f8        42 hours ago        67.9MB

After copying docker image from one of the old nodes to the new one, fluentd started successfully.

I want to suggest giving docker images you publish unique tags, so when you publish a new version:

  • the new version is uniquely identified
  • the old one is not overridden and still available on docker hub

Makefile syntax is incorrect for `ALL_IMAGES` var

The block of ALL_IMAGES currently looks like this:

ALL_IMAGES := \
	v0.12/alpine-elasticsearch:v0.12.33-elasticsearch,v0.12-elasticsearch,stable-elasticsearch,elasticsearch \
	v0.12/alpine-loggly:v0.12.33-loggly,v0.12-loggly,stable-loggly,loggly \
	v0.12/alpine-logentries:v0.12.33-logentries,v0.12-logentries,stable-logentries,logentries \
	v0.12/alpine-cloudwatch:v0.12.33-cloudwatch,v0.12-cloudwatch,stable-cloudwatch,cloudwatch \
	v0.12/alpine-s3:v0.12.33-s3,v0.12-s3,stable-s3,s3 \
	v0.12/alpine-papertrail:v0.12.33-papertrail,v0.12-papertrail,stable-papertrail,papertrail \
	v0.12/debian-elasticsearch:v0.12.33-debian-elasticsearch,v0.12-debian-elasticsearch,debian-elasticsearch \
	v0.12/alpine-kafka:v0.12.33-kafka,v0.12-kafka,stable-kafka,kafka \
	v0.12/debian-loggly:v0.12.33-debian-loggly,v0.12-debian-loggly,debian-loggly \
	v0.12/debian-logentries:v0.12.33-debian-logentries,v0.12-debian-logentries,debian-logentries \
	v0.12/debian-cloudwatch:v0.12.33-debian-cloudwatch,v0.12-debian-cloudwatch,debian-cloudwatch \
	v0.12/debian-stackdriver:v0.12.33-debian-stackdriver,v0.12-debian-stackdriver,debian-stackdriver \
	v0.12/debian-s3:v0.12.33-debian-s3,v0.12-debian-s3,debian-s3 \
	v0.12/debian-papertrail:v0.12.33-debian-papertrail,v0.12-debian-papertrail,debian-papertrail
	v0.12/debian-kafka:v0.12.33-debian-kafka,v0.12-debian-kafka,debian-kafka \

Look in particular at the last two lines... either that backslash should be removed or the last two lines need to be switched. Running a make command results in this:

➜ make image DOCKERFILE=v0.12/alpine-kafka                                                                
Makefile:29: *** recipe commences before first target.  Stop.

Add "usage" or installation section in readme

It would be nice to give more help in the README about how to use/install the docker images.
What parameters to pass to the image etc.

This could include a bit about installing with helm. With the cloudwatch image:

helm install incubator/fluentd-cloudwatch

Better document Logentries usage?

I'm not sure how to use this with Logentries. As far as I undertand I would need to mount a configmap at /etc/logentries/tokens.yaml but I'm not sure how this file should look like.

Typo in Dockerfile

ENV FLUENTD_CONF="fluentd.conf"

should be...

ENV FLUENTD_CONF="fluent.conf"

Get "403 Forbidden" message when running the pod

The pod does not start in my cluster, here is how I do that:

  • I run the pod with following command:
kubectl apply -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch.yaml
  • Wait for pod to run:
$ kubectl get pods | grep fluentd                      
fluentd-d7z2f                    0/1       CrashLoopBackOff   3          1m
  • Pod reports failure:
$ kubectl logs fluentd-d7z2f
2017-04-26 06:44:32 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2017-04-26 06:44:32 +0000 [info]: starting fluentd-0.12.32
2017-04-26 06:44:32 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.9.2'
2017-04-26 06:44:32 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '0.26.2'
2017-04-26 06:44:32 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.8.3'
2017-04-26 06:44:32 +0000 [info]: gem 'fluent-plugin-secure-forward' version '0.4.3'
2017-04-26 06:44:32 +0000 [info]: gem 'fluentd' version '0.12.32'
2017-04-26 06:44:32 +0000 [info]: adding match pattern="fluent.**" type="null"
2017-04-26 06:44:32 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2017-04-26 06:44:33 +0000 [info]: adding match pattern="**" type="elasticsearch"
2017-04-26 06:44:33 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error="Exception encountered fetching metadata from Kubernetes API endpoint: 403 Forbidden"
2017-04-26 06:44:33 +0000 [info]: process finished code=256
2017-04-26 06:44:33 +0000 [warn]: process died within 1 second. exit.
  • The error is:
config error file="/fluentd/etc/fluent.conf" error="Exception encountered fetching metadata from Kubernetes API endpoint: 403 Forbidden
  • After some digging I found out that server returns:
system:serviceaccount:kube-system:default cannot list pods at the cluster scope

How can I start fluentd with this issue? Thank you in advance!

Best practices for multiple targets?

I would like to target both elasticsearch and stackdriver from a kops cluster on AWS. My expected solution is to combine the fluent.conf files and rebuild the image with both plugins (and perhaps a couple more, shown below). Is there a better way?

Additionally, I expect to run the stackdriver logging and monitoring agents. The logging agent is a packaged version of fluentd, so, I'm wondering if it's worth it to leave it be or try to combine that config as well as that of fluentd-gcp just to grab everything I can.

Is there an existing solution for any of this? and/or recommendations on how I should proceed?

https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/b66ea52dce4d36c98d906518b724afd00ccf0dc1/fluentd-gcp-image/Gemfile#L7-L8

gem 'fluent-plugin-detect-exceptions', '~>0.0.5'
gem 'fluent-plugin-prometheus', '~>0.2.1'

Logz.io image not being published

I was looking through the repo, and while I see come configs for Logz.io, it seems that the image itself is not being built and published.

Is that an oversight, or is it intentionally being left unbuilt?

Elasticsearch images install fluentd-1.1.0

At least on debian.

# gem install fluent-plugin-elasticsearch
Building native extensions.  This could take a while...
Successfully installed strptime-0.2.3
Fetching: dig_rb-1.0.1.gem (100%)
Successfully installed dig_rb-1.0.1
Fetching: fluentd-1.1.0.gem (100%)
[...]

Images unable to start with Read-Only ConfigMaps

In response to kubernetes/kubernetes#60814, the Kubernetes team pushed out kubernetes/kubernetes#58720. This was also patched into the latest versions of 1.7.x, 1.8.x, and 1.9.x.

It makes mounted ConfigMaps ready only by default.

As a result, the sed commands in the default entrypoint for all images in this repo fail (combined with set -e causes the full entrypoint to fail, and thus enter a CrashLoopBackoff.

Entrypoint: https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/templates/entrypoint.sh

As a work-around, I've bypassed the entrypoint.sh file by specifying the command specifically, ie:

      containers:
      - name: fluentd
         image: fluent/fluentd-kubernetes-daemonset:v0.12-alpine-s3
         command:
           - fluentd
           - -c
           - /fluentd/etc/fluent.conf
         ...

This works for now, however it's definitely not the best, or an appropriate permanent, solution.

Any thoughts on how we can get this updated to avoid writes to mounted ConfigMaps?

Building images inside jenkins

Hello,

I'm trying to build the images inside a jenkins pipeline. That means using a docker image (running docker inside docker https://hub.docker.com/r/library/docker/ ) to build.

But I'm getting this error:

# make src-all
(set -e ;  make src DOCKERFILE=v0.12/alpine-elasticsearch VERSION=v0.12.33-elasticsearch TAGS=v0.12.33-elasticsearch,v0.12-elasticsearch,stable-elasticsearch,elasticsearch ;   make src DOCKERFILE=v0.12/alpine-loggly VERSION=v0.12.33-loggly TAGS=v0.12.33-loggly,v0.12-loggly,stable-loggly,loggly ;   make src DOCKERFILE=v0.12/alpine-logentries VERSION=v0.12.33-logentries TAGS=v0.12.33-logentries,v0.12-logentries,stable-logentries,logentries ;   make src DOCKERFILE=v0.12/alpine-cloudwatch VERSION=v0.12.33-cloudwatch TAGS=v0.12.33-cloudwatch,v0.12-cloudwatch,stable-cloudwatch,cloudwatch ;   make src DOCKERFILE=v0.12/alpine-s3 VERSION=v0.12.33-s3 TAGS=v0.12.33-s3,v0.12-s3,stable-s3,s3 ;   make src DOCKERFILE=v0.12/alpine-papertrail VERSION=v0.12.33-papertrail TAGS=v0.12.33-papertrail,v0.12-papertrail,stable-papertrail,papertrail ;   make src DOCKERFILE=v0.12/debian-elasticsearch VERSION=v0.12.33-debian-elasticsearch TAGS=v0.12.33-debian-elasticsearch,v0.12-debian-elasticsearch,debian-elasticsearch ;   make src DOCKERFILE=v0.12/alpine-kafka VERSION=v0.12.33-kafka TAGS=v0.12.33-kafka,v0.12-kafka,stable-kafka,kafka ;   make src DOCKERFILE=v0.12/debian-loggly VERSION=v0.12.33-debian-loggly TAGS=v0.12.33-debian-loggly,v0.12-debian-loggly,debian-loggly ;   make src DOCKERFILE=v0.12/debian-logentries VERSION=v0.12.33-debian-logentries TAGS=v0.12.33-debian-logentries,v0.12-debian-logentries,debian-logentries ;   make src DOCKERFILE=v0.12/debian-cloudwatch VERSION=v0.12.33-debian-cloudwatch TAGS=v0.12.33-debian-cloudwatch,v0.12-debian-cloudwatch,debian-cloudwatch ;   make src DOCKERFILE=v0.12/debian-stackdriver VERSION=v0.12.33-debian-stackdriver TAGS=v0.12.33-debian-stackdriver,v0.12-debian-stackdriver,debian-stackdriver ;   make src DOCKERFILE=v0.12/debian-s3 VERSION=v0.12.33-debian-s3 TAGS=v0.12.33-debian-s3,v0.12-debian-s3,debian-s3 ;   make src DOCKERFILE=v0.12/debian-papertrail VERSION=v0.12.33-debian-papertrail TAGS=v0.12.33-debian-papertrail,v0.12-debian-papertrail,debian-papertrail ;   make src DOCKERFILE=v0.12/debian-kafka VERSION=v0.12.33-debian-kafka TAGS=v0.12.33-debian-kafka,v0.12-debian-kafka,debian-kafka ; )
make[1]: Entering directory '/app'
mkdir -p docker-image/v0.12/alpine-elasticsearch
cp /app/templates/.dockerignore docker-image/v0.12/alpine-elasticsearch/.dockerignore
docker run --rm -i -v /app/templates/Dockerfile.erb:/Dockerfile.erb:ro \
	ruby:alpine erb -U -T 1 \
		dockerfile='v0.12/alpine-elasticsearch' \
		version='v0.12.33-elasticsearch' \
	/Dockerfile.erb > docker-image/v0.12/alpine-elasticsearch/Dockerfile
/usr/local/bin/erb:126:in `read': Is a directory @ io_fread - /Dockerfile.erb (Errno::EISDIR)
	from /usr/local/bin/erb:126:in `run'
	from /usr/local/bin/erb:170:in `<main>'
make[1]: *** [Makefile:145: dockerfile] Error 1
make[1]: Leaving directory '/app'
make: *** [Makefile:128: src-all] Error 2

Building locally, not using docker inside docker, it works.

Any tips?

Fluentd 1.0 stable support?

Hello
if I understand correctly, current fluentd stable relase is 1.0 but the README only talks about 0.12 and there's even a revert for 0.14 being announced as the new stable in Q3 2017 (I guess it got renamed to 1.0). So, do you have any plan to update the K8S daemonset to support fluentd 1.0? What are the alternatives if I need to build a custom fluentd image with a plugin from the fluent-plugins-nursery project?
Thank you very much!

Can't add multiple tags to Loggly logs

Hi there,

I'd like to add multiple tags to logs that are sent from Fluentd to Loggly. With the current Fluentd configuration, Loggly logs are only tagged as fluentd and I don't have the option to add my own tags.

It would be handy to pass a collection of tags tofluentd.conf.erb as an environment variable via my Kubernetes YAML file — maybe something like this:

<%# -- fluentd.conf.erb -- %>

<%# current Loggly configuration %>
loggly_url "https://logs-01.loggly.com/inputs/#{ENV['LOGGLY_TOKEN']}/tag/fluentd"
<%# proposed Loggly configuration %>
loggly_url "https://logs-01.loggly.com/inputs/#{ENV['LOGGLY_TOKEN']}/tag/#{ENV['LOGGLY_TAGS'] || 'fluentd'}"

This seems like a useful feature and I'm happy to set to work on a pull request. What do other people think?

If you're not familiar with Loggly, here's some info about tagging.

[Feature request] include HOSTNAME into static tag names

First of all, I have never used fluentd before so please be kind if I miss something obvious.

My setup is a multi-master setup on AWS and I want to use this DS to push all the logs to CoudWatch. The problem is that the sources defined in templates/conf/kubernetes.conf.erb include static tags like:

<source>
  type tail
  format none
  path /var/log/etcd.log
  pos_file /var/log/fluentd-etcd.log.pos
  tag etcd
</source>

This tag is used as logstream for CloudWatch and now all masters want to write to the same stream. Which leads to conflicts, provoking the error:

2017-11-16 16:18:23 +0000 [warn]: updating upload sequence token forcefully because unrecoverable error occured error=#<Aws::CloudWatchLogs::Errors::InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49576271730371018682039417059249827037114576942659553650> log_group="stage-cluster" log_stream="etcd" new_sequence_token="49576271730371018682039417229166768835589940760411130226"

Will a PR including the HOSTNAME env variable into all this static tags be accepted?

I thought about something like:

<source>
  type tail
  format none
  path /var/log/etcd.log
  pos_file /var/log/fluentd-etcd.log.pos
  tag etcd#{ENV['HOSTNAME']}
</source>

rpc error: code = 2 desc = "oci runtime error: write parent: broken pipe"

Hey guys,

I'm trying to deploy this daemonset (the cloudwatch version) to a kubernetes cluster.
First I tried with a cluster I have in AWS (Kube v 1.5.x) but it didn't work. I thought it was an issue with the version so now I've installed minikube (Kube v 1.6.4) but I keep getting the same error:

Failed to start container with id 4fe035a153fa874b6d6fc48e798ca4148d1675d6c54f04344bfb7cbbe48cba72 with error: rpc error: code = 2 desc = failed to start container "4fe035a153fa874b6d6fc48e798ca4148d1675d6c54f04344bfb7cbbe48cba72": Error response from daemon: rpc error: code = 2 desc = "oci runtime error: write parent: broken pipe"

My .yaml file looks like this:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: fluentd-cloudwatch
  name: prod-fluentd-cloudwatch
spec:
  selector:
    matchLabels:
      app: fluentd-cloudwatch
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: fluentd-cloudwatch
    spec:
      containers:
      - env:
        - name: AWS_REGION
          value: us-east-1
        - name: LOG_GROUP_NAME
          value: NAME
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              key: aws_access_key_id
              name: secret-fluentd-cloudwatch
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              key: aws_secret_access_key
              name: secret-fluentd-cloudwatch
        image: fluent/fluentd-kubernetes-daemonset:v0.12-debian-cloudwatch
        imagePullPolicy: IfNotPresent
        name: prod-fluentd-cloudwatch
        resources:
          limits:
            cpu: 100m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        terminationMessagePath: /dev/termination-log
        volumeMounts:
        - mountPath: /var/log
          name: varlog
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /var/log
        name: varlog
      - hostPath:
          path: /var/lib/docker/containers
        name: varlibdockercontainers
---
apiVersion: v1
data:
  aws_access_key_id: 'ACCESS_KEY'
  aws_secret_access_key: 'SECRET'
kind: Secret
metadata:
  labels:
    app: fluentd-cloudwatch
  name: secret-fluentd-cloudwatch
type: Opaque

I've done a ton of research and posted my question in several Slack groups but still no luck, has anyone stumbled upon this issue? I'm not sure if it's a bug or just something I'm missing.

Thanks in advance

Newest release cloudwatch containers no longer start

Following the merge of #44 yesterday and the subsequent image updates on docker hub, the cloudwatch docker containers fail to start up.

There is an existing issue for the aws sdk for ruby that details the problem: aws/aws-sdk-ruby#1590. Basically the latest sdk (3.x) is now modularised, so needs to be imported in a different way.

Unfortunately 3.0.0 of the aws sdk was released over a month after that merge request was created - so if it had been merged when the request had been opened, the new images would work correctly.

This re-release of existing tags on docker hub has completely broken our previously stable build. When a release is made, existing tags that reference a specific version (e.g. v0.12.33-cloudwatch) shouldn't be updated, but should be released using a brand new tag. For generic tags (such as cloudwatch) updating existing tags is OK, because people expect them to be updated.

RBAC policy example required

Hello,

I tried deploying fluent/fluentd-kubernetes-daemonset:v0.12-alpine-s3 into an RBAC enabled Kubernetes cluster and received this error:

2018-03-19 17:51:33 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error="Exception encountered fetching metadata from Kubernetes API endpoint: pods is forbidden: User \"system:serviceaccount:kube-system:default\" cannot list pods at the cluster scope ({\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"pods is forbidden: User \\\"system:serviceaccount:kube-system:default\\\" cannot list pods at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"pods\"},\"code\":403}\n)"

Unable to create the flunetd daemon on kubernetes

Trying to deploy the configuration for Flunetd Daemon set and it fails with the error:

https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch.yaml

kubectl version --short
Client Version: v1.6.1
Server Version: v1.9.3-2+0b1ffe38401940

Error:

kubectl create -f efk-fluentd.yaml
error: error validating "efk-fluentd.yaml": error validating data: found invalid field path for v1beta1.DaemonSet; if you choose to ignore these errors, turn validation off with --validate=false

I see that the error is already logged in a few places but the solutions are not helping resolve this my deployment issue.

Any leads is highly appreciated.

Thank you

Reduce fluentd container logging output

Currently the fluentd container is outputting info to stdout. There should be a way to increase the logging to warn or error as maybe an Environment variable for the container. This will reduce noise in production environments.

SSL configuration

I would love to be able to pass in SSL-related configuration to this image directly for connecting to an ElasticSearch that uses SSL.

I can imagine that I would put the certificates in a ConfigMap, mount it in, and have the certificate files available that way. But that is just part of the process, then I want to point to them via the configuration file, of course.

Or is there an easier way to accomplish this?

Wrong configuration details for S3

I am facing configuration related issue while trying to deploy S3 related changes.

Existing configuration for S3-
<% when "s3"%>
<match **>
type s3
s3_bucket "#{ENV['S3_BUCKET_NAME']}"

Required configuration for S3 should be as below I guess -
<% when "s3"%>
<match **>
type s3
log_level info
s3_bucket "#{ENV['S3_BUCKET_NAME']}"
s3_region "#{ENV['S3_BUCKET_REGION']}"
s3_object_key_format %{path}%{time_slice}/cluster-log-%{index}.%{file_extension}
path "#{ENV['S3_BUCKET_PATH']}"
buffer_path "#{ENV['S3_BUFFER_PATH']}"
aws_key_id "#{ENV['S3_BUCKET_KEY_ID']}"
aws_sec_key "#{ENV['S3_BUCKET_SEC_KEY']}"
time_slice_format %Y/%m/%d
time_slice_wait 10m
utc
include_time_key true
include_tag_key true
buffer_chunk_limit 256m

temporarily failed to flush the buffer Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8

Description

fluentd-kubernetes-daemonset does not send anything to the Cloudwatch group, it fails when trying to send with this error in the log:

2017-04-21 09:14:23 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:14:23 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:133:in `encode'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:133:in `to_json'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:133:in `block (2 levels) in write'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:126:in `each'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:126:in `block in write'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:104:in `each'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:104:in `write'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/buffer.rb:354:in `write_chunk'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/buffer.rb:333:in `pop'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/output.rb:342:in `try_flush'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/output.rb:149:in `run'

This is the complete log

$ kubectl logs dev-fluentd-cloudwatch-035t9 -n logging
2017-04-21 09:13:16 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2017-04-21 09:13:16 +0000 [info]: starting fluentd-0.12.33
2017-04-21 09:13:17 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2017-04-21 09:13:17 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.3.3'
2017-04-21 09:13:17 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '0.26.3'
2017-04-21 09:13:17 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.9.0'
2017-04-21 09:13:17 +0000 [info]: gem 'fluent-plugin-secure-forward' version '0.4.3'
2017-04-21 09:13:17 +0000 [info]: gem 'fluentd' version '0.12.33'
2017-04-21 09:13:17 +0000 [info]: adding match pattern="fluent.**" type="null"
2017-04-21 09:13:17 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2017-04-21 09:13:21 +0000 [info]: adding match pattern="**" type="cloudwatch_logs"
2017-04-21 09:13:21 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: using configuration file: <ROOT>
  <match fluent.**>
    type null
  </match>
  <source>
    type tail
    path /var/log/containers/*.log
    pos_file /var/log/fluentd-containers.log.pos
    time_format %Y-%m-%dT%H:%M:%S.%NZ
    tag kubernetes.*
    format json
    read_from_head true
  </source>
  <source>
    type tail
    format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
    time_format %Y-%m-%d %H:%M:%S
    path /var/log/salt/minion
    pos_file /var/log/fluentd-salt.pos
    tag salt
  </source>
  <source>
    type tail
    format syslog
    path /var/log/startupscript.log
    pos_file /var/log/fluentd-startupscript.log.pos
    tag startupscript
  </source>
  <source>
    type tail
    format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
    path /var/log/docker.log
    pos_file /var/log/fluentd-docker.log.pos
    tag docker
  </source>
  <source>
    type tail
    format none
    path /var/log/etcd.log
    pos_file /var/log/fluentd-etcd.log.pos
    tag etcd
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kubelet.log
    pos_file /var/log/fluentd-kubelet.log.pos
    tag kubelet
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kube-proxy.log
    pos_file /var/log/fluentd-kube-proxy.log.pos
    tag kube-proxy
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kube-apiserver.log
    pos_file /var/log/fluentd-kube-apiserver.log.pos
    tag kube-apiserver
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kube-controller-manager.log
    pos_file /var/log/fluentd-kube-controller-manager.log.pos
    tag kube-controller-manager
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kube-scheduler.log
    pos_file /var/log/fluentd-kube-scheduler.log.pos
    tag kube-scheduler
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/rescheduler.log
    pos_file /var/log/fluentd-rescheduler.log.pos
    tag rescheduler
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/glbc.log
    pos_file /var/log/fluentd-glbc.log.pos
    tag glbc
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/cluster-autoscaler.log
    pos_file /var/log/fluentd-cluster-autoscaler.log.pos
    tag cluster-autoscaler
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <filter kubernetes.**>
    type kubernetes_metadata
  </filter>
  <match **>
    type cloudwatch_logs
    log_group_name k8s-testq
    auto_create_stream true
    use_tag_as_stream true
  </match>
</ROOT>
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-apiserver-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_POD-8c84359f13adea39ed4a55a5125d1d0c0b7cd3a8d2cafa3bd28f693bafcd896a.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/etcd-server-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_etcd-container-53daba4bbd8bf05809af2b38de842804849a344f909da17a8b65592812c98f44.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/prometheus-prometheus-node-exporter-b3nkb_monitoring_prometheus-node-exporter-cd080c1c0153db79442f30f8203c650ece2cf461e903b63f75e5fb8549d6ba69.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-scheduler-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_kube-scheduler-fedcf148381b1376b40d02baab445a0cf6a9646fa864347b6d22662be6df585f.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/etcd-server-events-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_etcd-container-2b476b585a1fd76bd9aeb794ab768e7b517e8f19ef60683d4af55dcd17f9b678.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/dns-controller-3905064033-7dpzl_kube-system_POD-52d24c9afa094d70aff656bd86ab80520f3c55d461598749d10606c4cbc58769.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-proxy-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_POD-aab68b3db2ba54044fd14cca911c3a0c404816518d85897f8811b87dbe130a5b.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/dev-fluentd-cloudwatch-035t9_logging_dev-fluentd-cloudwatch-295565aca55dfe7fe9dfacb6c55e693de6eedfd045a8c276593730ccf437e704.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/etcd-server-events-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_POD-f4e74260b2e6060169bbd10a04cbae1d4a6537511033cd30ceb8c08a7b469517.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-controller-manager-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_kube-controller-manager-9b2ce2a7b180b03ddb6980dada16a75fd93e63791fddb710b902f51d8279310e.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/dev-fluentd-cloudwatch-0cf0x_logging_dev-fluentd-cloudwatch-5f7aeeffb3e6837a17d9fc44cdcbbd71025dc9e09b1be54d4b8cbc72e487f643.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/dns-controller-3905064033-7dpzl_kube-system_dns-controller-df93b047d0936b1cef30f3c4c7bb9004a0f51873279aeb25c7f31419c31f7f1f.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-controller-manager-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_kube-controller-manager-e33d89c54cec783c7b1c93a1e1c71851373a4331c25496351747a59369cbbe24.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-apiserver-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_kube-apiserver-c9d15e2dbbd44265d369cf26ecef754eb6fdf9ba26f7aa3648eacc8b0036735c.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/dev-fluentd-cloudwatch-0cf0x_logging_POD-6788ad180875f93ced6ea831b2c32e3159219f6807de0b42184b66130554cd23.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/calico-node-8s1nt_kube-system_install-cni-4ef4fd7464bcd03261eb8a92089c773ba12bfed7fb4b8e616ad4aa58890f9046.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/etcd-server-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_POD-fcb6afeb1431e326f918e8d007a3804f92030721fe0270487e86ea79899e2825.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-scheduler-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_POD-912beb2045e51c78644cb32508f2da0f6186f82575b3f5457afbb8f7fcb0b046.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/prometheus-prometheus-node-exporter-b3nkb_monitoring_POD-ca353f808c94b2151b6ea49ddc77544612cf7eabbd185676eba3b41bf20f3261.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-proxy-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_kube-proxy-984aa615033c397dfe53a7d946f4aa340d3e9befc921be0165a6500ac611cf62.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/calico-node-8s1nt_kube-system_calico-node-1858f09a9872eb301f48df6a6854ed119adee7b26f042c3fd4f53f0c2a27da21.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/calico-node-8s1nt_kube-system_POD-595063d9b99efb2b6bb24289467e73350985f0e3c64e561a24cab3aac45fae51.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/dev-fluentd-cloudwatch-035t9_logging_POD-58a8d71f2511784ed483503cbaea39502de9257cefe9807d95f763ddb295750c.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-controller-manager-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_POD-dbe4ddf33c9c294d1497bb5ce5859d66a9903f87d5ca110b2d34768e00862f73.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/etcd.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/kube-proxy.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/kube-apiserver.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/kube-controller-manager.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/kube-scheduler.log
2017-04-21 09:13:52 +0000 [info]: detected rotation of /var/log/containers/dev-fluentd-cloudwatch-0cf0x_logging_dev-fluentd-cloudwatch-5f7aeeffb3e6837a17d9fc44cdcbbd71025dc9e09b1be54d4b8cbc72e487f643.log; waiting 5 seconds
2017-04-21 09:13:52 +0000 [info]: detected rotation of /var/log/containers/dev-fluentd-cloudwatch-0cf0x_logging_POD-6788ad180875f93ced6ea831b2c32e3159219f6807de0b42184b66130554cd23.log; waiting 5 seconds
2017-04-21 09:14:23 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:14:23 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:133:in `encode'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:133:in `to_json'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:133:in `block (2 levels) in write'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:126:in `each'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:126:in `block in write'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:104:in `each'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:104:in `write'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/buffer.rb:354:in `write_chunk'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/buffer.rb:333:in `pop'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/output.rb:342:in `try_flush'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/output.rb:149:in `run'
2017-04-21 09:14:24 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:14:26 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:14:24 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:14:26 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:14:30 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:14:26 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:14:30 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:14:37 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:14:30 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:14:38 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:14:53 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:14:38 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:14:54 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:15:21 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:14:54 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:15:22 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:16:19 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:15:22 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:16:19 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:18:37 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:16:19 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:18:38 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:23:09 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:18:38 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:23:09 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:32:17 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:23:09 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:32:17 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:49:24 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:32:17 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:49:24 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 10:20:37 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:49:24 +0000 [warn]: suppressed same stacktrace
2017-04-21 10:20:37 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 11:29:40 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 10:20:37 +0000 [warn]: suppressed same stacktrace
2017-04-21 11:29:41 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 13:57:55 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 11:29:41 +0000 [warn]: suppressed same stacktrace

Repro steps:

  • Create a namespace so you can clean up easilly afterwards:
    kubectl create namespace encodingbug

  • Put the following into fluentd-cloudwatch.yaml

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: fluentd-cloudwatch
    release: dev
  name: dev-fluentd-cloudwatch
spec:
  selector:
    matchLabels:
      app: fluentd-cloudwatch
      release: dev
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: fluentd-cloudwatch
        release: dev
    spec:
      containers:
      - env:
        - name: AWS_REGION
          value: eu-west-1
        - name: LOG_GROUP_NAME
          value: k8s-testq
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              key: aws_access_key_id
              name: dev-fluentd-cloudwatch
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              key: aws_secret_access_key
              name: dev-fluentd-cloudwatch
        image: fluent/fluentd-kubernetes-daemonset:v0.12.33-cloudwatch
        imagePullPolicy: IfNotPresent
        name: dev-fluentd-cloudwatch
        resources:
          limits:
            cpu: 100m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        terminationMessagePath: /dev/termination-log
        volumeMounts:
        - mountPath: /var/log
          name: varlog
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /var/log
        name: varlog
      - hostPath:
          path: /var/lib/docker/containers
        name: varlibdockercontainers
---
apiVersion: v1
data:
  aws_access_key_id: REDACTED_AWS_ACCESS_KEY_ID
  aws_secret_access_key: REDACTED_AWS_SECRET_ACCESS_KEY
kind: Secret
metadata:
  labels:
    app: fluentd-cloudwatch
    release: dev
  name: dev-fluentd-cloudwatch
type: Opaque

  • Replace REDACTED_AWS_ACCESS_KEY_ID and REDACTED_AWS_SECRET_ACCESS_KEY by a valid AWS user with proper permissions in file fluentd-cloudwatch.yaml

  • Run kubectl create fluentd-cloudwatch.yaml --namespace encodingbug

  • Check the pods got created:
    kubectl get pods --namespace encodingbug -l "app=fluentd-cloudwatch,release=dev"

  • Check the logs:
    This checks all pods:
    kubectl logs --namespace encodingbug -l "app=fluentd-cloudwatch,release=dev"
    You can (should) specify a specific pod:
    kubectl logs dev-fluentd-cloudwatch-035t9 --namespace encodingbug

  • Clean up step:
    kubectl delete namespace encodingbug

Suggestion:

Check this Stackoverflow entry

Versions:

Docker image: fluent/fluentd-kubernetes-daemonset:v0.12.33-cloudwatch
Kubernetes: 1.5.2

Helm Package

It would be nice to get the daemonsets into helm at github.com/kubernetes/charts.

This means it would be as easy to install as:
helm install stable/fluentd-kubernetes-daemonset

fluent can not parse multiline correctly

image version: fluent/fluentd-kubernetes-daemonset:v0.12-debian
docker version: 1.12.5
plugin:

# gem list|grep fluent-plugin
fluent-plugin-kubernetes_metadata_filter (0.27.0)
fluent-plugin-multiline-parser (0.1.1)
fluent-plugin-record-reformer (0.9.0)
fluent-plugin-redis (0.2.3)
fluent-plugin-secure-forward (0.4.5)
fluent-plugin-systemd (0.0.8)

after deploying this image via kubernetes daemonsets,i found parser cannot deal with multiline log.

java trancestack log as follows:

2017-08-10 15:12:11.994 [com.pdcss.sfjc.db.DBPool:55] ERROR com.pdcss.sfjc.db.DBPool - Network error IOException: Connection timed out: connect
  java.sql.SQLException: Network error IOException: Connection timed out: connect
   at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:421)
   at net.sourceforge.jtds.jdbc.ConnectionJDBC3.<init>(ConnectionJDBC3.java:50)
   at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:185)

the filter conf like this:

<filter container.error>
      @type parser
      format multiline
      format_firstline /\d{4}-\d{1,2}-\d{1,2}/
      format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}\.\d{0,3}) \[(?<thread>.*)\] (?<level>[^\s]+) (?<classmathod>[^\s]+) - (?<message>[\s\S]*)/
      key_name all
</filter>

docker log:

2017-08-23 15:29:00 +0800 [warn]: pattern not match with data '  java.sql.SQLException: Network error IOException: Connection timed out: connect'
2017-08-23 15:29:00 +0800 [warn]: pattern not match with data '   at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:421)'
2017-08-23 15:29:00 +0800 [warn]: pattern not match with data '   at net.sourceforge.jtds.jdbc.ConnectionJDBC3.<init>(ConnectionJDBC3.java:50)'
2017-08-23 15:29:00 +0800 [warn]: pattern not match with data '   at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:185)'
2017-08-10 15:28:11 +0800 container.error: {"thread":"com.pdcss.sfjc.db.DBPool:55","level":"ERROR","classmathod":"com.pdcss.sfjc.db.DBPool","message":"Network error IOException: Connection timed out: connect"}

as context log said,filter can parser single line log correctly,but incorrectly when parsing multiline such as tracestack.
i guess the format param is wrong with the line break(\n) but no idea about this config.
by the way,as documentation fluent multiline parser mentioned,format (?<message>.*) also not work.
Any solution on this issue,please?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.