teamhephy / fluentd Goto Github PK

License: MIT License

Makefile 4.74% Ruby 42.73% Shell 47.79% Dockerfile 4.75%

fluentd's Introduction

Description

Deis (pronounced DAY-iss) is an open source PaaS that makes it easy to deploy and manage applications on your own servers. Deis builds on Kubernetes to provide a lightweight, Heroku-inspired workflow.

About

This is an centos7 based image for running fluentd. It is built for the purpose of running on a kubernetes cluster.

This work is based on the docker-fluentd and docker-fluentd-kubernetes images by the fabric8 team. This image is in with deis v2 to send all log data to the logger component.

Configuration

Support containerd log format

By default, fluent parses logs in docker json format. If you use containerd, you only need to set the following environment variables

CONTAINER_TAIL_PARSER_TYPE="/^(?<time>.+) (?<stream>stdout|stderr)( (?<tags>.))? (?<log>.*)$/"

Enable more verbose logging

By default we do not capture kubernetes system logs. However, it is possible to tell fluentd to capture those logs just by specifying a few new environment variables.

CAPTURE_START_SCRIPT
CAPTURE_DOCKER_LOG
CAPTURE_ETCD_LOG
CAPTURE_KUBELET_LOG
CAPTURE_KUBE_API_LOG
CAPTURE_CONTROLLER_LOG
CAPTURE_SCHEDULER_LOG

Set a variable's value to a non-empty string such as "true" to capture that log. Make these changes to the tpl/deis-logger-fluentd-daemon.yaml file in the Workflow chart directory.

Drop Fluentd Logs

To turn off log collection of fluentd's own logs to avoid infinite loops set the following environment variable to a non-empty string value

DROP_FLUENTD_LOGS

Disable Deis Output

To turn off the deis output plugin set the following environment variable to a non-empty string value

DISABLE_DEIS_OUTPUT

Disable sending log or metrics data to nsq

To turn off sending log or metrics data to nsq set the following environment variable to "false"

SEND_LOGS_TO_NSQ
SEND_METRICS_TO_NSQ

This means we will not capture data from the log stream and send it to NSQ for processing. This means you will disable application logs (deis logs) and metrics generated from deis router.

Plugins

fluent-plugin-kubernetes_metadata_filter

This plugin is used to decorate all log entries with kubernetes metadata.

fluent-plugin-elasticsearch

Allows fluentd to send log data to an elastic search cluster. You must specify an ELASTICSEARCH_HOST environment variable for this plugin to work.

ELASTICSEARCH_HOST="some.host"
ELASTICSEARCH_SCHEME="http/https"
ELASTICSEARCH_PORT="9200"
ELASTICSEARCH_USER="username"
ELASTICSEARCH_PASSWORD="password"
ELASTICSEARCH_LOGSTASH_FORMAT="true/false" - Creates indexes in the format index_prefix-YYYY.MM.DD
ELASTICSEARCH_TARGET_INDEX_KEY="kubernetes.namespace_name" - Allows the index name to come from within the log message map. See example message format below. This allows the user to have an index per namespace, container name, or other dynamic value.
ELASTICSEARCH_TARGET_TYPE_KEY="some.key" - Allows the user to set _type to a custom value found in the map.
ELASTICSEARCH_INCLUDE_TAG_KEY="true/false" - Merge the fluentd tag back into the log message map.
ELASTICSEARCH_INDEX_NAME="fluentd" - Set the index name where all events will be sent.
ELASTICSEARCH_LOGSTASH_PREFIX="logstash" - Set the logstash prefix variable which is used when you want to use logstash format without specifying ELASTICSEARCH_TARGET_INDEX_KEY.
ELASTICSEARCH_TIME_KEY="" - specify where the plugin can find the timestamp used for the @timestamp field
ELASTICSEARCH_TIME_KEY_FORMAT="" - specify the format of ELASTICSEARCH_TIME_KEY
ELASTICSEARCH_TIME_KEY_EXCLUDE_TIMESTAMP="" - If ELASTICSEARCH_TIME_KEY specified dont set ``@timestamp

fluent-plugin-remote_syslog

This plugin allows fluentd to send data to a remote syslog endpoint like papertrail. You can configure fluentd to talk to multiple remote syslog endpoints by using the following scheme:

SYSLOG_HOST_1=some.host
SYSLOG_PORT_1=514
SYSLOG_HOST_2=some.other.host
SYSLOG_PORT_2=52232

You can also set SYSLOG_HOST and SYSLOG_PORT.

fluent-plugin-sumologic

This plugin allows for fluentd to send all log data to a sumologic endpoint. You can configure it using the following environment variables:

SUMOLOGIC_COLLECTOR_URL
SUMOLOGIC_ENDPOINT
SUMOLOGIC_HOST
SUMOLOGIC_PORT : defaults to 80 (unless IS_HTTPS is set and then its 443)
IS_HTTPS

fluent-plugin-gelf-hs

This plugin allows for fluentd to send all log data to a remote graylog endpoint. You can configure it using the following environment variables:

GELF_HOST=some.host
GELF_PORT=12201
GELF_PROTOCOL="udp/tcp"
GELF_TLS="true/false"
GELF_TLS_OPTIONS_CERT="-----BEGIN CERTIFICATE-----\n[...]\n-----END CERTIFICATE-----"
GELF_TLS_OPTIONS_KEY="-----BEGIN PRIVATE KEY-----\n[...]\n-----END PRIVATE KEY-----"
GELF_TLS_OPTIONS_ALL_CIPHERS="true/false"
GELF_TLS_OPTIONS_TLS_VERSION=":TLSv1/:TLSv1_1/:TLSv1_2"
GELF_TLS_OPTIONS_NO_DEFAULT_CA="true/false"

Deis Output

Deis output is a custom fluentd plugin that was written to forward data directly to deis components while filtering out data that we did not care about. We have 2 pieces of information we care about currently.

Logs from applications that are written to stdout within the container and the controller logs that represent actions against those applications. These logs are sent to an internal messaging system (NSQ) on a configurable topic. The logger component then reads those messages and stores the data in an ring buffer.
Metric data from the nginx based router. We take the log and parse out request_time, response_time, and bytes_sent. Each one of these metrics makes up a series that we will ultimately send to our InfluxDB system. Attached to each series is the host the data came from (where router is running) and the status code for that request.

The topics these messages are put on are configurable via environment variables.

NSQ_LOG_TOPIC
NSQ_METRIC_TOPIC

Custom Plugins

If you need something beyond the plugins that come pre-installed in the image, it is possible to set some environment variables to install and configure custom plugins as well.

To install a custom plugin, simply set a FLUENTD_PLUGIN_# environment variable. For multiple plugins simply increment the trailing number. FLUENTD_PLUGIN_1=some-fluentd-plugin

To configure your custom plugins, use either the CUSTOM_STORE_# or CUSTOM_FILTER_# environment variables

CUSTOM_STORE_1="configuration text"
CUSTOM_FILTER_1="configuration text"

If you need the build tools available for installing your plugin, this can be enabled with another environment variable INSTALL_BUILD_TOOLS="true"

fluentd's People

Contributors

Stargazers

Watchers

Forkers

n0n0x yebyen kingdonb cryptophobia dmcnaught pdomagala bookingsync

fluentd's Issues

Use build args to capture build time data

From @jchauncey on November 10, 2016 17:53

Acceptance Criteria:

Use the --build-args flag when running docker build to pass in the following items (and more if needed)
BUILD_DATE
VERSION

You will need to do the following in the dockerfile to persist the data into the image:

ARG VERSION
ARG BUILD_DATE
ENV VERSION $VERSION
ENV BUILD_DATE $BUILD_DATE

Copied from original issue: deis/fluentd#71

Restarting container causes fluentd.conf to duplicate config

From @jschneiderhan on October 26, 2016 18:51

Logs of a deis fluentd container during it's first run:

ubuntu@ip-172-20-12-148:~$ sudo docker logs cf1ae5af2bc1
2016-10-24 18:45:58 +0000 [info]: reading config file path="/opt/fluentd/conf/fluentd.conf"
2016-10-24 18:45:58 +0000 [info]: starting fluentd-0.14.4
2016-10-24 18:45:58 +0000 [info]: spawn command to main: /usr/bin/ruby2.3 -Eascii-8bit:ascii-8bit /usr/local/bin/fluentd -c /opt/fluentd/conf/fluentd.conf --under-supervisor
2016-10-24 18:45:58 +0000 [info]: reading config file path="/opt/fluentd/conf/fluentd.conf"
2016-10-24 18:45:58 +0000 [info]: starting fluentd-0.14.4 without supervision
2016-10-24 18:45:58 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2016-10-24 18:45:58 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-10-24 18:45:58 +0000 [info]: gem 'fluent-mixin-rewrite-tag-name' version '0.1.0'
2016-10-24 18:45:58 +0000 [info]: gem 'fluent-plugin-deis_output' version '0.1.0'
2016-10-24 18:45:58 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.6.0'
2016-10-24 18:45:58 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '0.25.3'
2016-10-24 18:45:58 +0000 [info]: gem 'fluent-plugin-remote_syslog' version '0.3.2'
2016-10-24 18:45:58 +0000 [info]: gem 'fluent-plugin-sumologic-mattk42' version '0.0.4'
2016-10-24 18:45:58 +0000 [info]: gem 'fluentd' version '0.14.4'
2016-10-24 18:45:58 +0000 [info]: gem 'fluentd' version '0.14.0'
2016-10-24 18:45:58 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2016-10-24 18:45:59 +0000 [info]: adding match pattern="**" type="copy"
2016-10-24 18:45:59 +0000 [info]: adding source type="tail"
2016-10-24 18:45:59 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path "/var/log/containers/*.log"
    pos_file "/var/log/containers.log.pos"
    tag "kubernetes.*"
    format json
    read_from_head false
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
    kubernetes_url "https://172.28.0.1:443"
    bearer_token_file "/var/run/secrets/kubernetes.io/serviceaccount/token"
    verify_ssl false
  </filter>
  <match **>
    @type copy
    <store>
      @type "deis"
    </store>
  </match>
</ROOT>
[... and so on ...]

If I restart the container, the config is duplicated:

ubuntu@ip-172-20-12-148:~$ sudo docker restart cf1ae5af2bc1
cf1ae5af2bc1
ubuntu@ip-172-20-12-148:~$ sudo docker logs cf1ae5af2bc1
[... previous logs ...]
2016-10-26 18:46:46 +0000 [info]: reading config file path="/opt/fluentd/conf/fluentd.conf"
2016-10-26 18:46:46 +0000 [info]: starting fluentd-0.14.4 without supervision
2016-10-26 18:46:46 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2016-10-26 18:46:46 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-10-26 18:46:46 +0000 [info]: gem 'fluent-mixin-rewrite-tag-name' version '0.1.0'
2016-10-26 18:46:46 +0000 [info]: gem 'fluent-plugin-deis_output' version '0.1.0'
2016-10-26 18:46:46 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.6.0'
2016-10-26 18:46:46 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '0.25.3'
2016-10-26 18:46:46 +0000 [info]: gem 'fluent-plugin-remote_syslog' version '0.3.2'
2016-10-26 18:46:46 +0000 [info]: gem 'fluent-plugin-sumologic-mattk42' version '0.0.4'
2016-10-26 18:46:46 +0000 [info]: gem 'fluentd' version '0.14.4'
2016-10-26 18:46:46 +0000 [info]: gem 'fluentd' version '0.14.0'
2016-10-26 18:46:46 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2016-10-26 18:46:47 +0000 [info]: adding match pattern="**" type="copy"
2016-10-26 18:46:47 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2016-10-26 18:46:47 +0000 [info]: adding match pattern="**" type="copy"
2016-10-26 18:46:47 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2016-10-26 18:46:47 +0000 [info]: adding match pattern="**" type="copy"
2016-10-26 18:46:47 +0000 [info]: adding source type="tail"
2016-10-26 18:46:47 +0000 [info]: adding source type="tail"
2016-10-26 18:46:47 +0000 [info]: adding source type="tail"
2016-10-26 18:46:47 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path "/var/log/containers/*.log"
    pos_file "/var/log/containers.log.pos"
    tag "kubernetes.*"
    format json
    read_from_head false
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
    kubernetes_url "https://172.28.0.1:443"
    bearer_token_file "/var/run/secrets/kubernetes.io/serviceaccount/token"
    verify_ssl false
  </filter>
  <match **>
    @type copy
    <store>
      @type "deis"
    </store>
  </match>
  <source>
    @type tail
    path "/var/log/containers/*.log"
    pos_file "/var/log/containers.log.pos"
    tag "kubernetes.*"
    format json
    read_from_head false
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
    kubernetes_url "https://172.28.0.1:443"
    bearer_token_file "/var/run/secrets/kubernetes.io/serviceaccount/token"
    verify_ssl false
  </filter>
  <match **>
    @type copy
    <store>
      @type "deis"
    </store>
  </match>
  <source>
    @type tail
    path "/var/log/containers/*.log"
    pos_file "/var/log/containers.log.pos"
    tag "kubernetes.*"
    format json
    read_from_head false
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
    kubernetes_url "https://172.28.0.1:443"
    bearer_token_file "/var/run/secrets/kubernetes.io/serviceaccount/token"
    verify_ssl false
  </filter>
  <match **>
    @type copy
    <store>
      @type "deis"
    </store>
  </match>
</ROOT>
[... and so on ...]

I'm not sure what's up yet. Looking into it now. Apologies in advance if this is already known or intentional behavior.

Copied from original issue: deis/fluentd#68

feature request - ship custom topics only

From @darshanime on January 18, 2017 9:56

Currently, when we enable shipping logs to a remote syslog server, all the logs are shipped.

Internally, if the message is from the Workflow Controller or from an application deployed via workflow, it is sent to the logs topic on the local NSQD instance. If from the Workflow Router, to the metrics topic.

Similarly, there should be an environment variable that takes in logs from which apps to ship to syslog.
something like:

SHIP_LOGS_1=controller
SHIP_LOGS_2=my-app-foo
SHIP_LOGS_3=my-app-bar

Copied from original issue: deis/fluentd#82

Fluentd only ships logs from it's own namespace.

From @FrederikNS on October 14, 2016 11:57

Hi Deis,

I have been trying to run the deis/fluentd:v2.4.2 image as part of my Kubernetes cluster, as I wanted better tagging than the official fluentd container provided. The container seems to ship the logs to elasticsearch just fine, but unfortunately it only grabs the logs for the namespace which it's started up in.

I decided to start it up in the kube-system namespace, as it seemed like a "system" service, so now I can only see the logs from containers in the kube-system namespace. Is this how deis/fluentd works, or do I need to configure something? I can see that the configs in the repository uses the deis namespace.

The daemonset I created looks like this:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd-elasticsearch
  namespace: kube-system
  labels:
    k8s-app: fluentd-elasticsearch
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-elasticsearch
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: fluentd-elasticsearch
        image: deis/fluentd:v2.4.2
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        env:
          - name: DISABLE_DEIS_OUTPUT
            value: "true"
          - name: ELASTICSEARCH_HOST
            value: elasticsearch-logging
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

The fluentd logs mention my containers, for example:

2016-10-14 11:29:42 +0000 [info]: following tail of /var/log/containers/hello-kube-1544476892-ygjnd_default_hello-kube-de80e8bb19cebf6ca31935d4e9a692076212f82cd5a54c794c26b3ed6450a845.log
2016-10-14 11:29:42 +0000 [info]: following tail of /var/log/containers/hello-kube-1544476892-ygjnd_default_POD-59f14681b2e35862a76d92ccc7bd5f4639f465a41cef5d185a9faf85822c691b.log

Copied from original issue: deis/fluentd#59

If fluentd container is restarted configuration is duplicated

If container is restarted on the node (docker restart <id>) fluentd.conf is not built from scratch, configuration is duplicated (appended once more), with multiple restarts it may lead to significant log multiplication.

I think it is this line fault (and any other using cat << EOF >>).

I think we could start with empty file on each fluentd boot, any objections? or better ideas how to handle this? If not I can prepare PR.

spikes in requests/sec reporting under high number of requests

From @felipejfc on June 6, 2017 21:41

Deleting all deis-fluentd pods solves the problem for like 10 minutes or so...

My guess is that this is related to the number of metrics/sec that is being sent from each host to nsq, it's like 2000 metrics/sec.

Or maybe the number of logs fluentd is parsing?

Copied from original issue: deis/fluentd#99