callstats-io / fluentd-kubernetes-cloudwatch Goto Github PK

View Code? Open in Web Editor NEW

26.0 20.0 16.0 13 KB

Fluentd Docker image to send Kuberntes logs to CloudWatch

Shell 100.00%

fluentd-kubernetes-cloudwatch's Introduction

Fluentd image to send Kubernetes logs to CloudWatch

Deployment

Create AWS IAM user which has permission to store logs to CloudWatch
Optionally set environment variable to change the default resource names and labels

NAMESPACE defaults to 'kube-system'
APP_NAME defaults to 'fluentd-cloudwatch'
SECRET_NAME,CONFIG_NAME default to APP_NAME
ENV_NAME defaults to 'system'
CW_LOG_GROUP defaults to 'kubernetes-cluster'

Set environment variables for the AWS IAM user and CloudWatch region or set them just when running deploy.sh in the next step

export LOGGING_AWS_ACCESS_KEY_ID=<your key>
export LOGGING_AWS_SECRET_ACCESS_KEY=<your secret>
export LOGGING_AWS_REGION=<your region>

Run deploy.sh to create the Secret, ConfigMap, and DaemonSet, setting the AWS IAM user and CloudWatch region environments variables if you did not export them in the previous step.

LOGGING_AWS_ACCESS_KEY_ID=<your key> LOGGING_AWS_SECRET_ACCESS_KEY=<your secret> LOGGING_AWS_REGION=<your region> ./deploy.sh

Run display.sh to check everything is running

Removal

If you set custom values for the namespace or resource name environment variables, ensure they are still set to your values (NAMESPACE,APP_NAME,SECRET_NAME,CONFIG_NAME)
Run delete.sh
Run display.sh to check nothing is left

fluentd-kubernetes-cloudwatch's People

Contributors

Stargazers

Watchers

Forkers

tuananh bhegazy whereisaaron reliefsoftware jansel lc0 frontapp brymaven surfraz kivagant wlan0 ankitforcode troetz talentinc sasidhardw aramkarapetian

fluentd-kubernetes-cloudwatch's Issues

What would it take to log pod_name as the keyname instead of the node name

Include LICENSE.md

Would love to use this at our company. However, because there is no license on this repository, legally I can't fork or contribute. IMO Apache 2.0 or MIT may make the most sense.

What would it take to modify this to use IAM roles of the node?

Rather than specifying the AWS access keys in the k8s secrets repo, it would be great to simply leverage the IAM roles that are already assigned to the node

Configuration references non existant image

Default config points at, ${IMAGE_NAME:=callstats/fluentd-kubernetes-cloudwatch:v1.1.2}

Last versioned image tag pushed to docker hub is v1.0.1

CloudWatchLog Error Incomplete Signature Exception

First I created IAM policy then I followed the steps . The result is as follows:

Kubernetes container has environment variables:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_REGION
with correct values.

Container is started with success but can't flush the buffer. Error message looks as follows

2016-12-30 15:51:39 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-12-30 15:52:11 +0000 error_class="Aws::CloudWatchLogs::Errors::IncompleteSignatureException" error="Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter.

Region I am trying to push the logs to is eu-west-1 if that is important info

Add helm chart to deploy more easilly

Helm charts are quite easy to deploy.

It would be nice to be able to deploy this app with a simple command like:
helm upgrade --install v1 stable/fluentd-kubernetes-cloudwatch -f values.yaml
The package would be downloaded and installed, without us having to create/maintain the Kubernetes config files.
Also, it would give the app more visibility and the community could help improve the chart.

Is it possible to only collect logs from certain deployments?

There are some daemon set / deployments which i do not want to collect logs from. They produce quite a lots of noise.

UnrecognizedClientException

I was able to send my logs multiple times and now I'm getting this error.

error_class="Aws::CloudWatchLogs::Errors::UnrecognizedClientException" error="The security token included in the request is invalid." plugin_id="@out_cloudwatch_logs

My AWS credentials are correct. Any ideas on what is causing this?

AWS client error: Authorization header requires 'Signature' parameter

hello, i'm trying to deploy this fluentd build in a kubernetes 1.7 cluster

i have the following yaml config:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cloudwatch-logging-agent
  namespace: kube-system
  labels:
    app: fluentd
    env: dev
data:
  AWS_ACCESS_KEY_ID: {AWS_ACCESS_KEY_ID}
  AWS_SECRET_ACCESS_KEY: {AWS_SECRET_ACCESS_KEY}

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    env: dev
data:
  AWS_REGION: eu-west-1
  CW_LOG_GROUP: kubernetes-cluster

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    env: dev
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        env: dev
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: callstats/fluentd-kubernetes-cloudwatch # fluent/fluentd-kubernetes-daemonset:v0.12-debian-cloudwatch
        env:
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: cloudwatch-logging-agent
                key: AWS_ACCESS_KEY_ID
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: cloudwatch-logging-agent
                key: AWS_SECRET_ACCESS_KEY
          - name: AWS_REGION
            valueFrom:
              configMapKeyRef:
                name: fluentd
                key: AWS_REGION
          - name: CW_LOG_GROUP
            valueFrom:
              configMapKeyRef:
                name: fluentd
                key: CW_LOG_GROUP
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

and the AWS user i use for this has the following IAM policies attached

AmazonAPIGatewayPushToCloudWatchLogs
CloudWatchFullAccess
CloudWatchLogsReadOnlyAccess
CloudWatchLogsFullAccess
AmazonDMSCloudWatchLogsRole
AWSOpsWorksCloudWatchLogs

unfortunately, nothing shows up in cloudwatch logs, and when looking at the pod logs, i get these warnings from the aws client:

2017-10-11 09:15:31 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-10-11 09:15:31 +0000 error_class="Aws::CloudWatchLogs::Errors::IncompleteSignatureException" error="Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization=AWS4-HMAC-SHA256 Credential=AKIAIIFLVF4YM3PVXQUA" plugin_id="object:b0cb3c9ac"
  2017-10-11 09:15:31 +0000 [warn]: /home/fluent/.gem/ruby/2.3.0/gems/aws-sdk-core-2.7.15/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
  2017-10-11 09:15:31 +0000 [warn]: /home/fluent/.gem/ruby/2.3.0/gems/aws-sdk-core-2.7.15/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
  2017-10-11 09:15:31 +0000 [warn]: /home/fluent/.gem/ruby/2.3.0/gems/aws-sdk-core-2.7.15/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
  2017-10-11 09:15:31 +0000 [warn]: /home/fluent/.gem/ruby/2.3.0/gems/aws-sdk-core-2.7.15/lib/aws-sdk-core/plugins/response_paging.rb:26:in `call'
  2017-10-11 09:15:31 +0000 [warn]: /home/fluent/.gem/ruby/2.3.0/gems/aws-sdk-core-2.7.15/lib/seahorse/client/plugins/response_target.rb:21:in `call'
  2017-10-11 09:15:31 +0000 [warn]: /home/fluent/.gem/ruby/2.3.0/gems/aws-sdk-core-2.7.15/lib/seahorse/client/request.rb:70:in `send_request'
  2017-10-11 09:15:31 +0000 [warn]: /home/fluent/.gem/ruby/2.3.0/gems/aws-sdk-core-2.7.15/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
  2017-10-11 09:15:31 +0000 [warn]: /home/fluent/.gem/ruby/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.0/lib/fluent/plugin/out_cloudwatch_logs.rb:298:in `log_group_exists?'
  2017-10-11 09:15:31 +0000 [warn]: /home/fluent/.gem/ruby/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.0/lib/fluent/plugin/out_cloudwatch_logs.rb:121:in `block in write'
  2017-10-11 09:15:31 +0000 [warn]: /home/fluent/.gem/ruby/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.0/lib/fluent/plugin/out_cloudwatch_logs.rb:113:in `each'
  2017-10-11 09:15:31 +0000 [warn]: /home/fluent/.gem/ruby/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.0/lib/fluent/plugin/out_cloudwatch_logs.rb:113:in `write'
  2017-10-11 09:15:31 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/buffer.rb:354:in `write_chunk'
  2017-10-11 09:15:31 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/buffer.rb:333:in `pop'
  2017-10-11 09:15:31 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/output.rb:342:in `try_flush'
  2017-10-11 09:15:31 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/output.rb:149:in `run'
2017-10-11 09:15:32 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-10-11 09:15:34 +0000 error_class="Aws::CloudWatchLogs::Errors::IncompleteSignatureException" error="Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization=AWS4-HMAC-SHA256 Credential=AKIAIIFLVF4YM3PVXQUA" plugin_id="object:b0cb3c9ac"
  2017-10-11 09:15:32 +0000 [warn]: suppressed same stacktrace

is there something i can do to fix this ?

thank you

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.