kiyoto / fluent-plugin-docker-metrics Goto Github PK

Fluentd plugin to collect Docker metrics

Ruby 100.00%

fluent-plugin-docker-metrics's Introduction

Fluentd Docker Metrics Input Plugin

This is a Fluentd plugin to collect Docker metrics periodically.

How it works

It's assumed to run on the host server. It periodically runs Docker Remote API calls to fetch container IDs and looks at /sys/fs/cgroups/<metric_type>/docker/<container_id>/ for relevant stats. You can say this is an implementation of the metric collection strategy outlined in this blog post.

Installing

to be uploaded on Rubygems

Example config

<source>
  type docker_metrics
  stats_interval 1m
</source>

Parameters

stats_interval: how often to poll Docker containers for stats. The default is every minute.
cgroup_path: The path to cgroups pseudofiles. The default is sys/fs/cgroup.
tag_prefix: The tag prefix. The default value is "docker"

Example output

2014-11-22 17:48:26 +0000 docker.blkio.io_queued: {"key":"blkio_io_queued_total","value":0,"type":"counter","hostname":"precise64","id":"24f5fb3bfc429e88aa3dbacd704667899dc496067cedcfa58dd84da42e7cb3cf","name":"/world"}
2014-11-22 17:48:26 +0000 docker.blkio.sectors: {"key":"blkio_sectors","value":136,"type":"counter","hostname":"precise64","id":"24f5fb3bfc429e88aa3dbacd704667899dc496067cedcfa58dd84da42e7cb3cf","name":"/world"}

In particular, each event is a key-value pair of individual metrics. Also, it has

hostname is the hostname of the Docker host
id is the ID of the container
name is the descriptive name of the container (a la docker inspect --format '{{ .Names }}')

fluent-plugin-docker-metrics's People

Contributors

Stargazers

Watchers

Forkers

philalex doragon ixixi funkymonkeymonk shingo78 keihino dimafujitsu nathanpower hi-king gavinwhyte cosmo0920 esselius chenlonggang hasseio rexroof charygao jamesmawm

fluent-plugin-docker-metrics's Issues

Value for memory_stat_hierarchical_memory_limit too high

memory_stat_hierarchical_memory_limit causes ES to crash on import because the value is out of range. I simply capped the value for now with [v.to_i, 9223372036854775807].min. Maybe drop it altogether?

Permission denied for /var/run/docker.sock

plugin/in_docker_metrics.rb:103:rescue in on_timer: Permission denied - connect(2) for /var/run/docker.sock (Errno::EACCES)

My docker run:

docker run -d \
    --name fluentd \
    -e DEV_UID=$UID -e DEV_GID=$GID \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v "$(pwd)/data/logs:/fluentd/log" \
    -p 24224:24224 \
    fluentd

Is this plugin still maintained ?

Hi i would just like to know if this plugin is still maintained by either you or funkeymonkeymonk ?

As cadvisor is currently in a mess i was planning on using this for gaining access to metrics ?

Kind Regards

k.s-dean

Modification for running in pod

Hi.
I'm trying to modify the plugin so it can be ran with fluentd, which runs as daemonset. I'd like to create a pull request once I make it working.
But now I've got a question - where should it log metrics? I expected it to be logged in the fluentd's pod but I saw there only Ruby's error messages. Once I've fixed mentioned errors it logs nothing more.

Optionally add container name to output

Something like this:

data["name"] = `docker inspect --format "{{ .Name }}" #{id}`
mes.add(time, data) `

Add gauge metric type

The code only sets the 'type' field for counter-based metrics, it should also add the type field for 'gauge' metrics.

Make plugin compatible with fluent-plugin-rewrite-tag-filter

The record emitted by fluent-plugin-docker-metrics contains field that cannot be used by the rewritre-tag-filter plugin.

for ex:

<source>
  type docker_metrics
  stats_interval 10s
  cgroup_path /sys/fs/cgroup
</source>

<match docker.memory.stat>
  type rewrite_tag_filter
  rewriterule1 type .* ${tag}.counter
  rewriterule2 source .* all
</match>

rewrite rule 1 is never applied, despite the 'catch all' regex.
The rewrite rules seems to work only on the 'source' field, but not on the 'key', 'value' and 'type' fields