inputs, filters and outputs attrs can't handle "duplicates" about chef-logstash HOT 13 CLOSED

I might misunderstand this issue, but I am able to get e.g. multiple grok filters. I think this has to do with the fact that Ruby allows to omit {} for hashes if they are e.g. the last argument to a function or last element in an array. Thus something like

"filters" => [
  "grok" => { ... },
  "multiline" => { ... },
  "grok" => { ... }
]

is not an array of three hashes but an array with one hash with three keys (well, two really since the latter grok overrides the former), whereas

"filters" => [{
    "grok" => { ... }
  }, {
    "multiline" => { ... },
  }, {
    "grok" => { ... }
  }
]

is an array with three hashes, which get output correctly to the logstash config file.
Though the call to sort (https://github.com/lusis/chef-logstash/blob/master/libraries/logstash_conf.rb#L52) is probably wrong at least for the filters, as logstash processes the filters in order and this might change the order.

from chef-logstash.

i had this issue a while back too, and solved it in a similar way to the above suggestion. here is my workaround with the syslog example...(yes it is horrible)

override['logstash']['agent']['filters'] = [
  {"grok" => {
      "type" => "syslog",
      "pattern" => [ "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ],
      "add_field => [ \"received_at\", \"%{@timestamp}\" ]" => nil,
      "add_field" => [ "received_from", "%{@source_host}" ]}
      },
  {"syslog_pri" => {
      "type" => "syslog"}
  },
  {"date" => {
      "type" => "syslog",
      "match" => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]}
  },
  {"mutate" => {
      "type" => "syslog",
      "exclude_tags" => "_grokparsefailure",
      "replace => [ \"@source_host\", \"%{syslog_hostname}\" ]" => nil,
      "replace" => [ "@message", "%{syslog_message}" ]}
  },
  {"mutate" => {
      "type" => "syslog",
      "remove" => [ "syslog_hostname", "syslog_message", "syslog_timestamp" ]}
  }
  ]

from chef-logstash.

I tried williejs's work around and found that the quoting of the fields in the conf file messes this up. The entire key will get single quoted, so the configuration for the extra add_field, for example, becomes:

'add_field => [ "received_at", "%{@timestamp}" ]'

instead of
'add_field' => [ "received_at", "%{@timestamp}" ]

I still haven't figured out a good kluge for this yet.

from chef-logstash.

@willejs @sfiggins The docs say that the value type of add_field is Hash. Have you tried specifying a hash instead of arrays , e.g.

"add_field" => {
  "received_at" => "%{@timestamp}",
  "received_from" => "%{@source_host}"
}

?

from chef-logstash.

The configuration script currently flattens out hashes. It would translate that to:

'add_field' => ['received_at', '%{@timestamp}', 'received_from', '%{@source_host}']

I don't think there is really a good way to get these settings into the configuration using node attributes and the template in the cookbook.

Maybe that's okay though. The recipe does allow you to override the cookbook and template file name, so you don't have to use node attributes to configure logstash. Adding a caveat to the documents that points out the limitation and recommends to override the template might be good enough.

from chef-logstash.

@sfiggins That might be fine depending on how logstash handles the field's value, since there is a sort-of duality between arrays and hashes in Ruby where the array has the form [key1, value1, key2, value2]. All the docs' examples have an array of that form even though they state that the value type is Hash. Worth trying out I guess :)

from chef-logstash.

Reviewing the docs I see that hashes like this get converted to arrays of pairs by logstash anyway, so for add_field it looks like this would indeed work fine, and in my testing it seems to work.

Looks like there still might be a problem if you wanted multiple grok filters though. I think I'm happier with just overriding the template in a wrapper cookbook rather than trying to define all of this in node attributes.

from chef-logstash.

The whole approach of defining one array of hashes for inputs limits the usefulness imho.
I think something like the following makes more sense:

  node.set['logstash']['agent']['inputs']['myservice'] = {
    :file => {...}
  }

This way not only would we be able to specify multiple inputs of the same type without hassle, but we could also register logstash-agent inputs from other recipes, which would be awesome.

Or we enable proper etc/conf.d support. Either way, I'm afraid this needs a breaking api change. See also #51

from chef-logstash.

Would like to get this solved before the 0.7.0 branch goes live - since there are other breaking changes required for Logstash 1.2 support anyway, it seems like a pretty good time. Would love some design preference input from @lusis or @paulczar

See also #151

from chef-logstash.

in 0.7.0 branch I put in the workings to just override the configs to be template driven rather than the current big fat object. I figure the current system is useful for basic configs, but for anything complicated people would probably just prefer to create config files from templates.

from chef-logstash.

@paulczar that's probably not an awful solution - it might actually be worth emphasizing the reality of the state of attribute-driven configs in the README, since as it stands, there are some pretty serious deficiencies, particularly for 1.2

from chef-logstash.

Here's the commit with the code for template based configs.

646e9bb

from chef-logstash.

Closing out due to inactivity.

from chef-logstash.