takus / fluent-plugin-ec2-metadata Goto Github PK

I am attempting to use version 0.0.12 of this gem. I installed it and am able to use it successfully for most AWS tag key values. However....

We would like to have our custom tag tracking-id and the AWS standard aws:autoscaling:groupName values brought into fluent. Sadly, instead of getting the value, it is set to ${tagset_aws:autoscaling:groupname}. That is the value is ES. See screenshot below. It seems like the code is getting hung up on the included - and :. I thought it might be in https://github.com/takus/fluent-plugin-ec2-metadata/blob/master/lib/fluent/plugin/ec2_metadata.rb#L147 because I was struggling to parse the regex, but the str seems unchanged after going through the gsub.

A second question, what is the purpose of this line ? Since you are enumerating the pairs, how can that key not be present ?
https://github.com/takus/fluent-plugin-ec2-metadata/blob/master/lib/fluent/plugin/ec2_metadata.rb#L18

Thanks for your help, and this useful gem.
Kevin

The AWS EC2 APIs are not needed if you only want to use the metadata that is already available in the EC2-metadata service locally on in the instance.

You don't want to give the instances permissions to access APIs unless absolutely necessary as a large deployment can cause throttling issues and the like in the account.

Only the ${tagset_xxx} place holder seem to need access to the API.

For a record that does not contain a simple value (i.e. starting with the tagset phrase), the plugin returns no such attribute.
Example of configuration fragment:

   <record>
         my_json {"hostname":"${tagset_name}","instance_id":"${instance_id}","instance_type":"${instance_type}"}
    </record>

So they want to get JSON format for the record, it is not possible to use tagset_ (the others like instance_id - not tag - work).

Warn in log:

[warn]: #0 ec2-metadata: unknown placeholder ${tagset_name} found in a tag {...}

I've made a simple proposal to change to #50

with 0.0.4 the following worked fine:

  <match fluentd.all>
    type ec2_metadata
    output_tag ${tag}.ec2_metadata
    <record>
      fd_meta.ec2_az ${availability_zone}
      fd_meta.ec2_instance_id ${instance_id}
    </record>
  </match>

with 0.0.5:

2015-04-27 12:08:45 +0000 [error]: unexpected error error="\nMissing Credentials.\n\nUnable to find AWS credentials.  You can configure your AWS credentials\na few different ways:\n\n* Call AWS.config with :access_key_id and :secret_access_key\n\n* Export AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to ENV\n\n* On EC2 you can run instances with an IAM instance profile and credentials\n  will be auto loaded from the instance metadata service on those\n  instances.\n\n* Call AWS.config with :credential_provider.  A credential provider should\n  either include AWS::Core::CredentialProviders::Provider or respond to\n  the same public methods.\n\n= Ruby on Rails\n\nIn a Ruby on Rails application you may also specify your credentials in\nthe following ways:\n\n* Via a config initializer script using any of the methods mentioned above\n  (e.g. RAILS_ROOT/config/initializers/aws-sdk.rb).\n\n* Via a yaml configuration file located at RAILS_ROOT/config/aws.yml.\n  This file should be formated like the default RAILS_ROOT/config/database.yml\n  file.\n\n"

the two used parameters do not require API access according to the documentation

Would it be possible to support metadata endpoint version 3?

The old, HTTP based endpoint from v2 is not available to AWS Fargate tasks using platform version 1.3.0 and above, which was released last September.

I'm facing issue when use this plugin with google-fluentd 1.10.0 on CentOS7
This plugin work on google-fluentd 1.9.12 but when I upgrade to 1.10.0 it does not work and I don't know why.
Below are plugins I'm using:
gem 'fluentd' version '1.13.3'
gem 'fluent-plugin-ec2-metadata' version '0.1.3'

ruby version: ruby 2.0.0p648 (2015-12-16) [x86_64-linux]

Hi Takus,

I am getting subjected error while, adding metadata using ec2-metadata plugin. Its working fine for me on the instances, expose to internet (member of public subnet).

On my instance which where i am getting error, i am able to access ec2 metadata services using curl, also during fluentd startup i have captured network dump, where also i was able to see its successfully able to fetch matadata from metadata service. However still giving me above error.

Kindly help let me know, what other services it uses while fetching ec2-metadata.

I am attaching config/error as attachment.

ec2-metadata_issue.txt

-Rohit

http://docs.fluentd.org/articles/filter-plugin-overview

Because 0.0.16 contains breaking changes which cause runtime error to use with Fluentd v0.12.
Fluentd plugin upgrade guide says that fluentd plugins which are updating for v0.14 from v0.12 should bump major or minor version up.
ref: http://docs.fluentd.org/v0.14/articles/plugin-update-from-v12#2-update-dependency

Expected

• yank 0.0.16 and release 0.1.0 once again
• Fluentd v0.12 users can continue to use v0.12 series of Fluentd with ~> 0.0.15 syntax in their plugin management Gemfile.

Actual

• only patch level version up (0.0.15 -> 0.0.16)

It looks like this plugin doesn't work with the latest version of fluentd

'fluent-plugin-ec2-metadata' version '0.1.3'
'fluentd' version '1.9.1'

Using the following filter policy:

  <filter **>
    @type ec2_metadata

    <record>
      hostname    ${tagset_Name}
      instance_id ${instance_id}
      private_ip  ${private_ip}
      account_id  ${account_id}
      ami_id      ${image_id}
    </record>
  </filter>

My logs show all of the above fields as being empty. The keys do exist though so the plugin does seem to do something

Making calls to the following on the instance returns expected results

• http://169.254.169.254/latest/dynamic/instance-identity/document
• http://169.254.169.254/latest/meta-data/local-ipv4
• ...

Hello,
I just noticed that fluentd keep complaining about configurations for this module:
2018-03-12 13:36:48 +0000 [warn]: section <record> is not used in <filter ***> of ec2_metadata plugin
or
parameter 'output_tag' in <filter ***> is not used.

but it is still needed. This is pretty confusing.

Config example:

<filter ***>
  @type ec2_metadata
  metadata_refresh_seconds 300
  output_tag ${tag}
  <record>
    instance_id ${instance_id}
    instance_type ${instance_type}
    private_ip ${private_ip}
  </record>
</filter>

Basically I want to do something like this:

<match httpd_access_log>
  @type s3

  s3_bucket my-bucket
  path {instance_id}/httpd_access_log/
  acl bucket-owner-full-control

  <buffer time>
    @type memory
    timekey 60s
    timekey_wait 10s
  </buffer>

  time_slice_format %Y%m%d%H%M
</match>

The above isn't substituting {instance_id}.

I don't know much about fluentd and filters/matchers, but do I have to pass through your logic first then send to the S3 matcher?

Hey,
as far as I see from code AWS IAM permissions ec2:DescribeInstances only to read instance tags.

But even if I don't use tags in the message plugin still needs the IAM Permission, would be much better if it is required only when you use the tags in messages.

Can this plugin create nested fields, so the ec2 metadata is not spread through the _source, but everything is created inside of a nested object?

This:

{
   "ec2": {
        "ip": "1.2.3.4",
        "az": "eu-west-1"
    }
}

Instead of this:

{
   "ec2_ip":  "1.2.3.4",
   "ec2_az": "eu-west-1"
}

This is the current configuration we have, but it's not nesting the output:

<filter **>
    @type ec2_metadata

    <record>
        ec2.instance_id   ${instance_id}
        ec2.private_ip    ${private_ip}
        ec2.az            ${availability_zone}
    </record>
</filter>

Thanks!

Services provided by Gemnasium are no longer available. Need to migrate GitLab.
https://docs.gitlab.com/ee/user/project/import/gemnasium.html

With this configuration:

    <match **>
      @type ec2_metadata

      # aws_key_id  YOUR_AWS_KEY_ID
      # aws_sec_key YOUR_AWS_SECRET/KEY

      metadata_refresh_seconds 300 # Optional, default 300 seconds
      imdsv2 false                  # Optional, default false

      output_tag ${availability_zone}.${tag}
    </match>

I get this error:

#<Thread:0x00007fb5331ed648@event_loop@/var/lib/gems/2.5.0/gems/fluentd-1.9.2/lib/fluent/plugin_helper/thread.rb:70 run> terminated with exception (report_on_exception is true):
/var/lib/gems/2.5.0/gems/fluent-plugin-ec2-metadata-0.1.3/lib/fluent/plugin/ec2_metadata.rb:156:in `each': stack level too deep (SystemStackError)
	from /var/lib/gems/2.5.0/gems/fluent-plugin-ec2-metadata-0.1.3/lib/fluent/plugin/ec2_metadata.rb:156:in `each_with_index'
	from /var/lib/gems/2.5.0/gems/fluent-plugin-ec2-metadata-0.1.3/lib/fluent/plugin/ec2_metadata.rb:156:in `prepare_placeholders'
	from /var/lib/gems/2.5.0/gems/fluent-plugin-ec2-metadata-0.1.3/lib/fluent/plugin/ec2_metadata.rb:131:in `modify'
	from /var/lib/gems/2.5.0/gems/fluent-plugin-ec2-metadata-0.1.3/lib/fluent/plugin/out_ec2_metadata.rb:23:in `block in process'
	from /var/lib/gems/2.5.0/gems/fluentd-1.9.2/lib/fluent/event.rb:110:in `each'
	from /var/lib/gems/2.5.0/gems/fluent-plugin-ec2-metadata-0.1.3/lib/fluent/plugin/out_ec2_metadata.rb:22:in `process'
	from /var/lib/gems/2.5.0/gems/fluentd-1.9.2/lib/fluent/plugin/output.rb:797:in `emit_sync'
	from /var/lib/gems/2.5.0/gems/fluentd-1.9.2/lib/fluent/event_router.rb:97:in `emit_stream'
	 ... 2617 levels...
	from /var/lib/gems/2.5.0/gems/cool.io-1.6.0/lib/cool.io/loop.rb:88:in `run_once'
	from /var/lib/gems/2.5.0/gems/cool.io-1.6.0/lib/cool.io/loop.rb:88:in `run'
	from /var/lib/gems/2.5.0/gems/fluentd-1.9.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
	from /var/lib/gems/2.5.0/gems/fluentd-1.9.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-02-21 14:46:35 +0000 [error]: unexpected error error_class=SystemStackError error="stack level too deep"

gem 'activesupport', '6.0.2.1'
gem 'fluentd', '1.8.1'
gem 'fluent-plugin-concat', '2.4.0'
gem 'fluent-plugin-detect-exceptions', '0.0.13'
gem 'fluent-plugin-elasticsearch', '3.8.0'
gem 'fluent-plugin-kubernetes_metadata_filter', '2.4.1'
gem 'fluent-plugin-multi-format-parser', '1.0.0'
gem 'fluent-plugin-prometheus', '1.7.0'
gem 'fluent-plugin-systemd', '1.0.2'
gem 'fluent-plugin-grok-parser', '2.6.1'
gem 'fluent-plugin-rewrite-tag-filter', '2.2.0'
gem 'fluent-plugin-kinesis', '3.2.0'
gem 'oj', '3.10.1'

Regards

Updating @ec2_metadata instance variable should be atomic in thread.

https://github.com/takus/fluent-plugin-ec2-metadata/blob/master/lib/fluent/plugin/ec2_metadata.rb#L31-L32