fluent-plugin-prometheus, a plugin for Fluentd
A fluent plugin that instruments metrics from records and exposes them via web interface. Intended to be used together with a Prometheus server.
| fluent-plugin-prometheus | fluentd | ruby |
|---|---|---|
| 1.x.y | >= v1.9.1 | >= 2.4 |
| 1.[0-7].y | >= v0.14.8 | >= 2.1 |
| 0.x.y | >= v0.12.0 | >= 1.9 |
Since v1.8.0, fluent-plugin-prometheus uses http_server helper to launch HTTP server.
If you want to handle lots of connections, install async-http gem.
Add this line to your application's Gemfile:
gem 'fluent-plugin-prometheus'And then execute:
$ bundle
Or install it yourself as:
$ gem install fluent-plugin-prometheus
fluentd-plugin-prometheus includes 6 plugins.
prometheusinput pluginprometheus_monitorinput pluginprometheus_output_monitorinput pluginprometheus_tail_monitorinput pluginprometheusoutput pluginprometheusfilter plugin
See sample configuration, or try tutorial.
You have to configure this plugin to expose metrics collected by other Prometheus plugins. This plugin provides a metrics HTTP endpoint to be scraped by a Prometheus server on 24231/tcp(default).
With following configuration, you can access http://localhost:24231/metrics on a server where fluentd running.
<source>
@type prometheus
</source>
More configuration parameters:
bind: binding interface (default: '0.0.0.0')port: listen port (default: 24231)metrics_path: metrics HTTP endpoint (default: /metrics)aggregated_metrics_path: metrics HTTP endpoint (default: /aggregated_metrics)content_encoding: encoding format for the exposed metrics (default: identity). Supported formats are {identity, gzip}
When using multiple workers, each worker binds to port + fluent_worker_id.
To scrape metrics from all workers at once, you can access http://localhost:24231/aggregated_metrics.
Use <trasnport tls>. See transport config article for more details.
<source>
@type prometheus
<transport tls>
# TLS parameters...
</transport
</source>
This plugin collects internal metrics in Fluentd. The metrics are similar to/part of monitor_agent.
fluentd_status_buffer_queue_lengthfluentd_status_buffer_total_bytesfluentd_status_retry_countfluentd_status_buffer_newest_timekeyfrom fluentd v1.4.2fluentd_status_buffer_oldest_timekeyfrom fluentd v1.4.2
With following configuration, those metrics are collected.
<source>
@type prometheus_monitor
</source>
More configuration parameters:
<labels>: additional labels for this metric (optional). See Labelsinterval: interval to update monitor_agent information in seconds (default: 5)
This plugin collects internal metrics for output plugin in Fluentd. This is similar to prometheus_monitor plugin, but specialized for output plugin. There are Many metrics prometheus_monitor does not include, such as num_errors, retry_wait and so on.
Metrics for output
fluentd_output_status_retry_countfluentd_output_status_num_errorsfluentd_output_status_emit_countfluentd_output_status_retry_wait- current retry_wait computed from last retry time and next retry time
fluentd_output_status_emit_recordsfluentd_output_status_write_countfluentd_output_status_rollback_countfluentd_output_status_flush_time_countin milliseconds from fluentd v1.6.0fluentd_output_status_slow_flush_countfrom fluentd v1.6.0
Metrics for buffer
fluentd_output_status_buffer_total_bytesfluentd_output_status_buffer_stage_lengthfrom fluentd v1.6.0fluentd_output_status_buffer_stage_byte_sizefrom fluentd v1.6.0fluentd_output_status_buffer_queue_lengthfluentd_output_status_buffer_queue_byte_sizefrom fluentd v1.6.0fluentd_output_status_buffer_newest_timekeyfrom fluentd v1.6.0fluentd_output_status_buffer_oldest_timekeyfrom fluentd v1.6.0fluentd_output_status_buffer_available_space_ratiofrom fluentd v1.6.0
With following configuration, those metrics are collected.
<source>
@type prometheus_output_monitor
</source>
More configuration parameters:
<labels>: additional labels for this metric (optional). See Labelsinterval: interval to update monitor_agent information in seconds (default: 5)gauge_all: Specify metric type. Iftrue, usegaugetype. Iffalse, usecountertype. Since v2, this parameter will be removed and usecountertype.
This plugin collects internal metrics for in_tail plugin in Fluentd. in_tail plugin holds internal state for files that the plugin is watching. The state is sometimes important to monitor plugins work correctly.
This plugin uses internal class of Fluentd, so it's easy to break.
fluentd_tail_file_position: Current bytes which plugin reads from the filefluentd_tail_file_inode: inode of the filefluentd_tail_file_closed: Number of closed filesfluentd_tail_file_opened: Number of opened filesfluentd_tail_file_rotated: Number of rotated filesfluentd_tail_file_throttled: Number of times files got throttled (only with fluentd version > 1.17)
Default labels:
plugin_id: a value set for a plugin in configuration.type: plugin name.in_tailonly for now.path: file path
With following configuration, those metrics are collected.
<source>
@type prometheus_tail_monitor
</source>
More configuration parameters:
<labels>: additional labels for this metric (optional). See Labelsinterval: interval to update monitor_agent information in seconds (default: 5)
Both output/filter plugins instrument metrics from records. Both plugins have no impact against values of each records, just read.
Assuming you have following configuration and receiving message,
<match message>
@type stdout
</match>
message {
"foo": 100,
"bar": 200,
"baz": 300
}
In filter plugin style,
<filter message>
@type prometheus
<metric>
name message_foo_counter
type counter
desc The total number of foo in message.
key foo
</metric>
</filter>
<match message>
@type stdout
</match>
In output plugin style:
<filter message>
@type prometheus
<metric>
name message_foo_counter
type counter
desc The total number of foo in message.
key foo
</metric>
</filter>
<match message>
@type copy
<store>
@type prometheus
<metric>
name message_foo_counter
type counter
desc The total number of foo in message.
key foo
</metric>
</store>
<store>
@type stdout
</store>
</match>
With above configuration, the plugin collects a metric named message_foo_counter from key foo of each records.
You can access nested keys in records via dot or bracket notation (https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax), for example: $.kubernetes.namespace, $['key1'][0]['key2']. The record accessor is enable only if the value starts with $. or $[.
See Supported Metric Type and Labels for more configuration parameters.
For details of each metric type, see Prometheus documentation. Also see metric name guide.
<metric>
name message_foo_counter
type counter
desc The total number of foo in message.
key foo
<labels>
tag ${tag}
host ${hostname}
foo bar
</labels>
</metric>
name: metric name (required)type: metric type (required)desc: description of this metric (required)key: key name of record for instrumentation (optional)initialized: boolean controlling initilization of metric (optional). See Metric initialization<labels>: additional labels for this metric (optional). See Labels<initlabels>: labels to use for initialization of ReccordAccessors/Placeholder labels (optional). See Metric initialization
If key is empty, the metric values is treated as 1, so the counter increments by 1 on each record regardless of contents of the record.
<metric>
name message_foo_gauge
type gauge
desc The total number of foo in message.
key foo
<labels>
tag ${tag}
host ${hostname}
foo bar
</labels>
</metric>
name: metric name (required)type: metric type (required)desc: description of metric (required)key: key name of record for instrumentation (required)initialized: boolean controlling initilization of metric (optional). See Metric initialization<labels>: additional labels for this metric (optional). See Labels<initlabels>: labels to use for initialization of ReccordAccessors/Placeholder labels (optional). See Metric initialization
<metric>
name message_foo
type summary
desc The summary of foo in message.
key foo
<labels>
tag ${tag}
host ${hostname}
foo bar
</labels>
</metric>
name: metric name (required)type: metric type (required)desc: description of metric (required)key: key name of record for instrumentation (required)initialized: boolean controlling initilization of metric (optional). See Metric initialization<labels>: additional labels for this metric (optional). See Labels<initlabels>: labels to use for initialization of ReccordAccessors/Placeholder labels (optional). See Metric initialization
<metric>
name message_foo
type histogram
desc The histogram of foo in message.
key foo
buckets 0.1, 1, 5, 10
<labels>
tag ${tag}
host ${hostname}
foo bar
</labels>
</metric>
name: metric name (required)type: metric type (required)desc: description of metric (required)key: key name of record for instrumentation (required)initialized: boolean controlling initilization of metric (optional). See Metric initializationbuckets: buckets of record for instrumentation (optional)<labels>: additional labels for this metric (optional). See Labels<initlabels>: labels to use for initialization of ReccordAccessors/Placeholder labels (optional). See Metric initialization
See Prometheus Data Model first.
You can add labels with static value or dynamic value from records. In prometheus_monitor input plugin, you can't use label value from records.
<labels>
key1 value1
key2 value2
</labels>
All labels sections has same format. Each lines have key/value for label.
You can access nested fields in records via dot or bracket notation (https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax), for example: $.kubernetes.namespace, $['key1'][0]['key2']. The record accessor is enable only if the value starts with $. or $[. Other values are handled as raw string as is and may be expanded by placeholder described later.
You can use placeholder for label values. The placeholders will be expanded from reserved values and records.
If you specify ${hostname}, it will be expanded by value of a hostname where fluentd runs.
The placeholder for records is deprecated. Use record accessor syntax instead.
Reserved placeholders are:
${hostname}: hostname${worker_id}: fluent worker id${tag}: tag name- only available in Prometheus output/filter plugin
${tag_parts[N]}refers to the Nth part of the tag.- only available in Prometheus output/filter plugin
${tag_prefix[N]}refers to the [0..N] part of the tag.- only available in Prometheus output/filter plugin
${tag_suffix[N]}refers to the [tagsize-1-N..] part of the tag.- where
tagsizeis the size of tag which is splitted with.(when tag is1.2.3, thentagsizeis 3) - only available in Prometheus output/filter plugin
- where
You can configure if a metric should be initialized to its zero value before receiving any event. To do so you just need to specify initialized true.
<metric>
name message_bar_counter
type counter
desc The total number of bar in message.
key bar
initialized true
<labels>
foo bar
</labels>
</metric>
If your labels contains ReccordAccessors or Placeholders, you must use <initlabels> to specify the values your ReccordAccessors/Placeholders will take. This feature is useful only if your Placeholders/ReccordAccessors contain deterministic values. Initialization will create as many zero value metrics as <initlabels> blocks you defined.
Potential reserved placeholders ${hostname} and ${worker_id}, as well as static labels, are automatically added and should not be specified in <initlabels> configuration.
<metric>
name message_bar_counter
type counter
desc The total number of bar in message.
key bar
initialized true
<labels>
key $.foo
tag ${tag}
foo bar
worker_id ${worker_id}
</labels>
<initlabels>
key foo1
tag tag1
</initlabels>
<initlabels>
key foo2
tag tag2
</initlabels>
</metric>
<labels>
hostname ${hostname}
</labels>
Prometheus output/filter plugin can have multiple metric section. Top-level labels section specifies labels for all metrics. Labels section inside metric section specifies labels for the metric. Both are specified, labels are merged.
<filter message>
@type prometheus
<metric>
name message_foo_counter
type counter
desc The total number of foo in message.
key foo
<labels>
key foo
data_type ${type}
</labels>
</metric>
<metric>
name message_bar_counter
type counter
desc The total number of bar in message.
key bar
<labels>
key bar
</labels>
</metric>
<labels>
tag ${tag}
hostname ${hostname}
</labels>
</filter>
In this case, message_foo_counter has tag, hostname, key and data_type labels.
Checkout repository and setup.
$ git clone git://github.com/fluent/fluent-plugin-prometheus.git
$ cd fluent-plugin-prometheus
$ bundle install --path vendor/bundle
Download pre-compiled Prometheus binary and start it. It listens on 9090.
$ wget https://github.com/prometheus/prometheus/releases/download/v1.5.2/prometheus-1.5.2.linux-amd64.tar.gz -O - | tar zxf -
$ ./prometheus-1.5.2.linux-amd64/prometheus -config.file=./misc/prometheus.yaml -storage.local.path=./prometheus/metrics
Install Nginx for sample metrics. It listens on 80 and 9999.
$ sudo apt-get install -y nginx
$ sudo cp misc/nginx_proxy.conf /etc/nginx/sites-enabled/proxy
$ sudo chmod 777 /var/log/nginx && sudo chmod +r /var/log/nginx/*.log
$ sudo service nginx restart
Start fluentd with sample configuration. It listens on 24231.
$ bundle exec fluentd -c misc/fluentd_sample.conf -v
Generate some records by accessing nginx.
$ curl http://localhost/
$ curl http://localhost:9999/
Confirm that some metrics are exported via Fluentd.
$ curl http://localhost:24231/metrics
Then, make a graph on Prometheus UI. http://localhost:9090/
- Fork it ( https://github.com/fluent/fluent-plugin-prometheus/fork )
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create a new Pull Request
| Author | Masahiro Sano |
| Copyright | Copyright (c) 2015- Masahiro Sano |
| License | Apache License, Version 2.0 |
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent