saltstack-formulas / logstash-formula Goto Github PK

View Code? Open in Web Editor NEW

23.0 38.0 66.0 23 KB

Home Page: http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html

License: MIT License

SaltStack 67.56% Jinja 32.44%

logstash-formula's People

Contributors

Stargazers

Watchers

logstash-formula's Issues

Allow versions > 1.4

At the time of writing, versions 1.5 and 2.0 are out. Users should be able to choose their Logstash version.

Pillar input always parsed to string

It seems that the generated logstash configs are always generated with the pillar input being put into strings ("").

For example

  outputs:
    -
      plugin_name: elasticsearch
      cond: 'if [type] == "test"'
      workers: 8
      index: 'test-%{+YYYY.MM.dd}'
      flush_size: 5000
      hosts:
        - 127.0.0.1:9200

Is parsed into:

output {
    if [type] == "test" {
        elasticsearch {
            index => "test-%{+YYYY.MM.dd}"
            workers => "8" 
            hosts => [
                "127.0.0.1:9200"
            ] 
            flush_size => "5000"
        }
    }
}

In this case logstash will fail overwriting the default of the "workers" setting:

:message=>"Pipeline aborted due to error", :exception=>#<ArgumentError: comparison of String with 1 failed>

This is on logstash-2.3.3; I don't know if logstash previously has been able to overwrite int setting inputs with string.

Unable to manage map file: Jinja cannot concatenate 'str' and 'list' objects

I am getting an error when trying to use a list in my pillar file. The use case is that one of the fields I want to add to my logstash record is a tag, which can have multiple values.

When I run Logstash locally, I can use the following filter:

filter {
  if [path] =~ "nginx" {
    grok {
      match => { "message" => "%{DATESTAMP} \[%{LOGLEVEL:loglevel}\] %{GREEDYDATA:message}" }
    }
    mutate {
      add_field => { "my_string_field" => "this-works" }
      add_field => { "my_list_field" => ['this-also-works','so-does-this'] }
    }
  }
}

My corresponding pillar file for doing this with this salt formula:

-
  plugin_name: grok
  cond: 'else if [type] == "nginx"'
  match:
    message: '%{DATESTAMP} \[%{LOGLEVEL:loglevel}\] %{GREEDYDATA:message}'
  add_field:
    my_string_field: 'this-works'
    my_list_field: ['this-does-not-work','neither-does-this']

The error I receive in Salt:

Comment: Unable to manage file: Jinja error: cannot concatenate 'str' and 'list' objects
          /var/cache/salt/minion/files/base/logstash/map.jinja(43):
          ---
          [...]
                {{- output_indented(col, (key + ' => "' + value|string + '"')) }}
              {%- elif value is mapping %}
                {{- output_indented(col, (key + ' => {')) }}
                {%- set col = col + logstash.indent %}
                {%- for attr_key, attr_value in value.items() %}
                  {{- output_indented(col, (attr_key + ' => "' + attr_value + '"')) }}    <======================
                {%- endfor %}
                {%- set col = col - logstash.indent %}
                {{- output_indented(col, '}') }}
              {%- elif value is iterable %}
                {{- output_indented(col, key + ' => [') }}
          [...]
          ---
          Traceback (most recent call last):
            File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 368, in render_jinja_tmpl
              output = template.render(**decoded_context)
            File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1008, in render
              return self.environment.handle_exception(exc_info, True)
            File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 780, in handle_exception
              reraise(exc_type, exc_value, tb)
            File "<template>", line 2, in top-level template code
            File "/usr/lib/python2.7/dist-packages/jinja2/runtime.py", line 551, in _invoke
              rv = self._func(*arguments)
            File "/var/cache/salt/minion/files/base/logstash/map.jinja", line 43, in template
              {{- output_indented(col, (attr_key + ' => "' + attr_value + '"')) }}
          TypeError: cannot concatenate 'str' and 'list' objects

It seems like the config file only expects strings, not list values. Ideally it should accept any input that works in Logstash. I'm not totally sure how the formatting works, but perhaps it could do something like this:

 {%- for attr_key, attr_value in value.items() %}
      {% if attr_value is iterable %}{{- output_indented(col, (attr_key + ' => ' + attr_value + '')) }} 
      {% else %}{{- output_indented(col, (attr_key + ' => "' + attr_value + '"')) }} {% endif %}
 {%- endfor %}

Deprecated iteritems() method not forward-compatible with Python 3

Formulas iterating over dictionaries should use the items() method instead of iteritems() because the latter method has been removed from Python 3. While the items() method in Python 2 uses more memory than iteritems(), these dictionaries are typically small (e.g., sets of settings from Pillar).

Dealing with "big" update

In the current status, the formula results to be a bit outdated (e.g. old DEB repository) and uses a monolithic approach (a single init state that installs and configures everything).

Recently I had to install the latest version of logstash (5.4.x) and this formula resulted to be not usable, so I had to rewrite it in a "modern" way. Here the repo

https://github.com/psmiraglia/logstash-formula

In this case, where a massive update could be usefull for the community, how to deal with PR? I mean, is considered licit a PR that totally rewrite the formula?

Error in repo.sls

I get the below error when trying to apply the logstash-formula in a CentOS 7 (RedHat) VM.

In repo.sls, should the tilde's and quotes be removed on line 56?:

- humanname: "Elastic repository for " ~ {{ version }} ~ ".x packages"

Error:

Data failed to compile:
----------
    Rendering SLS 'base:logstash.repo' failed: while parsing a block mapping
  in "<unicode string>", line 13, column 11:
            - humanname: "Elastic repository f ...
              ^
expected <block end>, but found '<scalar>'
  in "<unicode string>", line 13, column 48:
     ... name: "Elastic repository for " ~ 5 ~ ".x packages"
                                         ^

Strange TemplateNotFound Error

Hi All,

I have the formula cloned as it comes, where I have specified the config in a pillar file, with no filters, and output pointing to elasticsearch as shown:

logstash:
  java: openjdk-8-jre-headless
  use_upstream_repo: True
  repo:
      version: 6
      old_repo: False
  inputs:
    -
      plugin_name: file
      path:
        - /var/log/syslog
        - /var/log/auth.log
      type: syslog
  outputs:
    -
      plugin_name: elasticsearch
      hosts:
        - "elastic7.myserver.com"
      port: 9200

Nothing else has been changed from the cloned repo. However, I get the following error upon applying the state:

          ID: logstash-config-inputs
    Function: file.managed
        Name: /etc/logstash/conf.d/01-inputs.conf
      Result: False
     Comment: Unable to manage file: Jinja error: logstash/map.jinja
              Traceback (most recent call last):
                File "/var/tmp/.salt_67a7f8_salt/py2/salt/utils/templates.py", line 389, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/var/tmp/.salt_67a7f8_salt/py2/jinja2/environment.py", line 1008, in render
                  return self.environment.handle_exception(exc_info, True)
                File "/var/tmp/.salt_67a7f8_salt/py2/jinja2/environment.py", line 780, in handle_exception
                  reraise(exc_type, exc_value, tb)
                File "<template>", line 1, in top-level template code
                File "/var/tmp/.salt_67a7f8_salt/py2/salt/utils/jinja.py", line 140, in get_source
                  raise TemplateNotFound(template)
              TemplateNotFound: logstash/map.jinja

              ; line 1

              ---
              {%- from 'logstash/map.jinja' import logstash, format_logstash_config with context -%}    <======================
              {{ format_logstash_config('input', logstash.inputs) }}
              Traceback (most recent call last):
                File "/var/tmp/.salt_67a7f8_salt/py2/salt/utils/templates.py", line 389, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/var/tmp/.salt_67a7f8_salt/py2/jinja2/environment.py", line 1008, in render
              [...]
              ---
     Started: 11:56:16.988834
    Duration: 45.645 ms
     Changes:

I'm unsure what the issue may be because as far as I can see on my side, everything is where it should be. Perhaps the formula does not support ES? Please let me know if this is a legitimate bug, or if there is something outstanding in my configuration.

Many Thanks!

Reload service instead of restart

Should have - reload: True and a fix for elastic/logstash#10500.

Quotes not escaped in config

This YAML incorrectly generates this config. Note the unescaped "s in the output.

- plugin_name: ruby
  code: |-
    event.set("foo", "bar")
    return [event]

ruby {
    code => "event.set("foo", "bar")
    return [event]"
}

install logstash plugins

Currently you can install logstash, configure logstash, but some of the configurations in logstash require additional plugins installed that are not there by default. It would be nice to add a plugins.sls that is refrenced in init.sls if logstash:plugins pillar is defined. The plugins.sls would then run the command referenced in documentation to install the plugin.

The Logstash Documentation refrences a relative path of bin/logstash-plugin. In the Debian based repo this maps to /usr/share/logstash/bin/logstash-plugin. I'm guessing it is similar on Redhat but I am unsure of the Redhat based installation path.

Config did not apply (Unable to manage file: Jinja error: logstash/map.jinja)

Hello,

The formula does not woark under salt-ssh (or something wrong with my settings).

Pillar's top:

$ cat salt/pillar/top.sls 
base:
  'elk-ls-*':
    - logstash

Logstash pillar (only inputs for the testing purpose)

$ cat salt/pillar/logstash.sls 
---
logstash:
  inputs:
    - 
      plugin_name: beat
      port: 5044

The following error message has appeared:

$ salt-ssh  'elk-ls-*' state.apply logstash
----------
          ID: logstash-config-inputs
    Function: file.managed
        Name: /etc/logstash/conf.d/01-inputs.conf
      Result: False
     Comment: Unable to manage file: Jinja error: logstash/map.jinja
              Traceback (most recent call last):
                File "/var/tmp/.sadm_2296c1_salt/pyall/salt/utils/templates.py", line 392, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/var/tmp/.sadm_2296c1_salt/pyall/jinja2/asyncsupport.py", line 76, in render
                  return original_render(self, *args, **kwargs)
                File "/var/tmp/.sadm_2296c1_salt/pyall/jinja2/environment.py", line 1008, in render
                  return self.environment.handle_exception(exc_info, True)
                File "/var/tmp/.sadm_2296c1_salt/pyall/jinja2/environment.py", line 780, in handle_exception
                  reraise(exc_type, exc_value, tb)
                File "/var/tmp/.sadm_2296c1_salt/pyall/jinja2/_compat.py", line 37, in reraise
                  raise value.with_traceback(tb)
                File "<template>", line 1, in top-level template code
                File "/var/tmp/.sadm_2296c1_salt/pyall/salt/utils/jinja.py", line 171, in get_source
                  raise TemplateNotFound(template)
              jinja2.exceptions.TemplateNotFound: logstash/map.jinja
              
              ; line 1
              
              ---
              {%- from 'logstash/map.jinja' import logstash,format_logstash_config with context -%}    <======================
              {{ format_logstash_config('input', logstash.inputs) }}
              Traceback (most recent call last):
                File "/var/tmp/.sadm_2296c1_salt/pyall/salt/utils/templates.py", line 392, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/var/tmp/.sadm_2296c1_salt/pyall/jinja2/asyncsupport.py", line 76, in render
              [...]
              ---
     Started: 06:06:19.322577
    Duration: 108.909 ms
     Changes:

Versions:

$ salt-ssh --versions-report
Salt Version:
           Salt: 2019.2.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.4 (default, Jul 16 2019, 07:12:58)
   python-gnupg: Not Installed
         PyYAML: 5.1.1
          PyZMQ: Not Installed
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: Not Installed
 
System Versions:
           dist: arch Manjaro Linux
         locale: UTF-8
        machine: x86_64
        release: 4.19.69-1-MANJARO
         system: Linux
        version: arch Manjaro Linux

[FEATURE] Validate rendered config files

Is your feature request related to a problem?

No.

Describe the solution you'd like

Validate config files generated by the macro using the command mentioned in the following reference.

https://stackoverflow.com/a/45991263

bin/logstash --config.test_and_exit --path.config config.conf

Describe alternatives you've considered

N/a.

Additional context

Multiple Message Lines?

Is it possible to add multiple message lines? I've tried multiple different configs but I've not been able to get anything to work.

Things I've tried:

        -
            plugin_name: grok
            match:
                message: '["pattern1","pattern2"]'
            add_field:
                received_at: '%{@timestamp}'
                received_from: '%{host}'

        -
            plugin_name: grok
            match:
                message: "pattern1"
                message: "pattern2"
            add_field:
                received_at: '%{@timestamp}'
                received_from: '%{host}'

And a bunch of other random things.

New feature: Plugins installation

This formula is great, I would like to have also, let's say a plugin.sls state to install every input/output plugin listed on pillars.
something like (for an output plugin):

logstash-output-{{logstash.outputs.plugin_name}}:
  gem.installed:
    - ruby: jruby@jgemset

Since plugins are now available as gem:

./gem search logstash-output-s3

*** REMOTE GEMS ***

logstash-output-s3 (0.1.6)

What do you think?

Add mutate examples to example pillar file

It's hard (for me at least) to see how I would go about mutating a field using the pillar field.

Examples of field mutation in the output json are shown here in the Logstash Config examples: https://www.elastic.co/guide/en/logstash/current/config-examples.html

Would it be possible for you to add a very basic usage example to the example pillar file in this repo for reference?

[FEATURE] Support bundled java

Newer logstash versions come with their own bundled JRE, and shouldn't use an older one from the system.

[BUG] salt-ssh python 3.7 contextvars issue

Hello,

The following error has appeared when I try run the test.ping:

salt-ssh 'ops*' test.ping
ops-c-admin:
    ----------
    retcode:
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.sadm_4f1a73_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.sadm_4f1a73_salt/pyall/salt/scripts.py", line 437, in salt_call
            import salt.cli.call
          File "/var/tmp/.sadm_4f1a73_salt/pyall/salt/cli/call.py", line 6, in <module>
            import salt.cli.caller
          File "/var/tmp/.sadm_4f1a73_salt/pyall/salt/cli/caller.py", line 14, in <module>
            import salt.loader
          File "/var/tmp/.sadm_4f1a73_salt/pyall/salt/loader.py", line 7, in <module>
            import contextvars
          File "/var/tmp/.sadm_4f1a73_salt/py3/contextvars.py", line 1, in <module>
            from _contextvars import Context, ContextVar, Token, copy_context

Seems that I have to apply the following fix but I didn't find how to apply it.
Could you help me?

Your setup

Initial salt-ssh setup :

virtualenv venv
source venv/bin/activate
pip install salt-ssh

Versions reports (master & minion)

 Salt Version:
          Salt: 3003
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.7.10 (default, Jun  3 2021, 07:40:26)
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 22.1.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: solus 4.2 fortitude
        locale: UTF-8
       machine: x86_64
       release: 5.11.22-180.current
        system: Linux
       version: Solus 4.2 fortitude

Just in case:

venv  ~  salt-ssh  pip list
Package       Version
------------- ---------
certifi       2021.5.30
chardet       4.0.0
contextvars   2.4
distro        1.5.0
idna          2.10
immutables    0.15
Jinja2        3.0.1
MarkupSafe    2.0.1
msgpack       1.0.2
pip           21.1.2
pycryptodomex 3.10.1
PyYAML        5.4.1
pyzmq         22.1.0
requests      2.25.1
salt-ssh      3003
setuptools    57.0.0
urllib3       1.26.5
wheel         0.36.2

Keys not quoted in filter config

This YAML

 - plugin_name: mutate
   copy: { host.name: hostname }

should generate this config

mutate {
    copy => {
        "host.name" => "hostname"
    }
}

but instead it generates

mutate {
    copy => {
        host.name => "hostname"
    }
}

which causes

FailedAction: Expected one of #, =>

apt-transport-https not supported in Flourine

Salt 2018.3.3
logstash:repo:version: 6

      ID: logstash-repo
Function: pkg.installed
    Name: apt-transport-https
  Result: True
 Comment: All specified packages are already installed
 Started: 17:33:39.722449
Duration: 482.545 ms
 Changes:
Warnings: The following package(s) are "virtual package" names: apt-
          transport-https. These will no longer be supported as of the
          Fluorine release. Please update your SLS file(s) to use the actual
          package name.