Giter Club home page Giter Club logo

puppet-logstashforwarder's Introduction

puppet-logstashforwarder

A puppet module for managing logstash-forwarder

Build Status

Requirements

  • Puppet 2.7.x or better.
  • The stdlib Puppet library.

Optional:

  • The apt Puppet library when using repo management on Debian/Ubuntu.

Usage

Installation, make sure service is running and will be started at boot time:

 class { 'logstashforwarder': }

Install a certain version:

 class { 'logstashforwarder':
   version => '1.0'
 }

In the absense of an appropriate package for your environment it is possible to install from other sources as well.

http/https/ftp source:

 class { 'logstashforwarder':
   package_url => 'http://download.website.com/packages/logstashforwarder.rpm'
 }

puppet:// source:

 class { 'logstashforwarder':
   package_url => 'puppet:///path/to/logstashforwarder.rpm'
 }

Local file source:

 class { 'logstashforwarder':
   package_url => 'file:/path/to/logstashforwarder.rpm'
 }

Attempt to upgrade logstashforwarder if a newer package is detected (false by default):

 class { 'logstashforwarder':
   autoupgrade => true
 }

Install everything but disable the service (useful for pre-configuring systems):

 class { 'logstashforwarder':
   status => 'disabled'
 }

Under normal circumstances a modification to the logstashforwarder configuration will trigger a restart of the service. This behaviour can be disabled:

 class { 'logstashforwarder':
   restart_on_change => false
 }

Disable and remove logstashforwarder entirely:

 class { 'logstashforwarder':
   ensure => 'absent'
 }     

Configuration

Network and SSL

For the network part of the configuration you need to set the servers and ssl information.

 class { 'logstashforwarder':
   servers  => [ 'logstash.yourdomain.com' ],
   ssl_key  => 'puppet:///path/to/your/ssl.key',
   ssl_ca   => 'puppet:///path/to/your/ssl.ca',
   ssl_cert => 'puppet:///path/to/your/ssl.cert'
 }

If you already manage the SSL files you can also specify them as the full path.

 class { 'logstashforwarder':
   servers  => [ 'logstash.yourdomain.com' ],
   ssl_key  => '/path/to/your/ssl.key',
   ssl_ca   => '/path/to/your/ssl.ca',
   ssl_cert => '/path/to/your/ssl.cert'
 }

Files

For configuring the files you want to process you can use the 'file' define:

 logstashforwarder::file { 'apache':
   paths  => [ '/var/log/apache/access.log' ],
   fields => { 'type' => 'apache' },
 }

The 'fields' hash allows you to set custom fields which you can use in Logstash.

Repository management

Most sites will manage repositories seperately; however, this module can manage the repository for you.

 class { 'logstashforwarder':
   manage_repo  => true
 }

Note: When using this on Debian/Ubuntu you will need to add the Puppetlabs/apt module to your modules.

Service Management

Currently only the basic SysV-style init service provider is supported but others could be implemented relatively easily (pull requests welcome).

init

Defaults File

The defaults file (/etc/defaults/logstashforwarder or /etc/sysconfig/logstashforwarder) for the logstashforwarder service can be populated as necessary. This can either be a static file resource or a simple key value-style hash object, the latter being particularly well-suited to pulling out of a data source such as Hiera.

file source
 class { 'logstashforwarder':
   init_defaults_file => 'puppet:///path/to/defaults'
 }
hash representation
 $config_hash = {
   'logstashforwarder_USER' => 'logstashforwarder',
   'logstashforwarder_GROUP' => 'logstashforwarder',
 }

 class { 'logstashforwarder':
   init_defaults => $config_hash
 }

Support

Need help? Join us in #logstash on Freenode IRC or on the https://discuss.elastic.co/c/logstash discussion forum.

puppet-logstashforwarder's People

Contributors

camelloj avatar electrical avatar jskarpe avatar ycombinator avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

puppet-logstashforwarder's Issues

Problem with the RPM

When puppet tries to install the RPM I get the following error:
Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install logstash-forwarder' returned 1: Error in PREIN scriptlet in rpm package logstash-forwarder-0.4.0-1.x86_64
/var/tmp/rpm-tmp.LDCPp9: line 10: 14553 Segmentation fault (core dumped) useradd -r -g logstash-forwarder -d /opt/logstash-forwarder -s /sbin/nologin -c "logstash-forwarder" logstash-forwarder
error: %pre(logstash-forwarder-0.4.0-1.x86_64) scriptlet failed, exit status 139
error: install: %pre scriptlet failed (2), skipping logstash-forwarder-0.4.0-1

Any idea what the problem might be?

logstash-forwarder service is not notified when ssl_key or ssl_ca is updated

If you update the ssl ca or key files that logstash-forwarder uses, the puppet module does not notify the logstash-forwarder service and so it is not restarted.

It's possible we just need something like this in manifests/config.pp:

105       file { "${ssldir}/${basefilename_ssl_key}":
106         source  => $ssl_key,
107         mode    => '0440',
108         require => File[$ssldir],
+++         notify    => $notify_service,
109       }

And the same for the 'file { "${ssldir}/${basefilename_ssl_ca}"' declaration.

Logstashforwarder uses deprecated apt::source parameters

(Scope(Apt::Source[logstashforwarder])) $include_src is deprecated and will be removed in the next major release, please use $include => { 'src' => false } instead
(Scope(Apt::Source[logstashforwarder])) $key_source is deprecated and will be removed in the next major release, please use $key => { 'source' => http://packages.elasticsearch.org/GPG-KEY-elasticsearch } instead.
(Scope(Apt::Key[Add key: 46095ACC8548582C1A2699A9D27D666CD88E42B4 from Apt::Source logstashforwarder])) $key_source is deprecated and will be removed in the next major release. Please use $source instead.

How to set spool-size?

Isn't there a way to set the spool-size? If not, then support for this will be handy.

not able to set logstashforwarder::file parameters through yaml

I am able to define the servers through yaml as
logstashforwarder::servers: ["localhost:5043"]

but this was not possible for the logstashforwarder::file

I created the following wrapper class to enable setting the logstashforwarder::file through yaml as
logstashforwarder_wrapper::files:
'messages':
'paths': ["/var/log/messages"]
'fields':
'type': 'syslog'

class logstashforwarder_wrapper (
$files = $logstashforwarder_wrappper::params::files,
) inherits xlrelease_template::params {
create_resources('logstashforwarder::file', $files)
}

maybe you would like add this functionality to the master. I would not need an extra wrapper class and the parameters can be set through yaml

Problem with Debian packages

Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install logstash-forwarder' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package logstash-forwarder
Error: /Stage[main]/Logstashforwarder::Package/Package[logstash-forwarder]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install logstash-forwarder' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package logstash-forwarder

Logstashforwarder::File : dependency cycle

I define in my manifest this:

  logstashforwarder::file { 'syslog':
    paths  => [ "/var/log/syslog" ],
    fields => { "type" => "syslog" },
    require => Class['::logstashforwarder'],
  }

Here is the error message:

Error: Could not apply complete catalog: Found 1 dependency cycle:
(Anchor[logstashforwarder::end] => Class[Logstashforwarder] => Logstashforwarder::File[syslog] => Logstashforwarder_fragment[syslog] => Logstashforwarder_config[lsf-config] => Class[Logstashforwarder::Service] => Logstashforwarder::Service::Init[logstash-forwarder] => Service[logstash-forwarder] => Logstashforwarder::Service::Init[logstash-forwarder] => Class[Logstashforwarder::Service] => Anchor[logstashforwarder::end])

Of course I tried without this declaration and it works fine, and I tried with other things than syslog, same error result.

8 character apt key in repo.pp

The puppetlabs/apt module displays a warning if a key id is not specified with a full fingerprint(40 characters).
In repo.pp the apt::source key parameter defines an 8 character id. Should be replaced with the full 40 characters.

Remove log directory from module uploaded on forge.puppetlabs.com

I installed this module using the command:

puppet module install elasticsearch-logstashforwarder --environment=nehal --modulepath /home/nehal/puppet/modules/

And now I have a log directory inside the module mentioning some docker stuff in it.

nehal@tst-puppet1 ~/puppet ❯❯❯ find modules/logstashforwarder/log
modules/logstashforwarder/log
modules/logstashforwarder/log/centos-6-x64
modules/logstashforwarder/log/centos-6-x64/2015-03-30_13_12_19
modules/logstashforwarder/log/centos-6-x64/2015-03-30_13_12_19/sut.log
...
...
modules/logstashforwarder/log/centos-6-x64/2015-03-30_12_17_19
modules/logstashforwarder/log/centos-6-x64/2015-03-30_12_17_19/sut.log

each puppet run will try to enable the service

I wanted to mention the following behaviour.
elasticsearch-logstashforwarder (v0.1.1)
PE2015.2
CentOS 7

Each puppet run prints the following notice information:
Notice: /Stage[main]/Logstashforwarder::Service/Logstashforwarder::Service::Init[logstash-forwarder]/Service[logstash-forwarder]/enable: enable changed 'false' to 'true'

Debug: Caching connection for https://puppetmaster.home:8140
Debug: Executing: '/bin/systemctl is-active logstash-forwarder'
Debug: Executing: '/bin/systemctl show logstash-forwarder --property LoadState --property UnitFileState --no-pager'
Debug: Executing: '/bin/systemctl unmask logstash-forwarder'
Debug: Executing: '/bin/systemctl enable logstash-forwarder'
Notice: /Stage[main]/Logstashforwarder::Service/Logstashforwarder::Service::Init[logstash-forwarder]/Service[logstash-forwarder]/enable: enable changed 'false' to 'true'
Debug: /Stage[main]/Logstashforwarder::Service/Logstashforwarder::Service::Init[logstash-forwarder]/Service[logstash-forwarder]: The container Logstashforwarder::Service::Init[logstash-forwarder] will propagate my refresh event
Debug: Logstashforwarder::Service::Init[logstash-forwarder]: The container Class[Logstashforwarder::Service] will propagate my refresh event
Debug: Class[Logstashforwarder::Service]: The container Stage[main] will propagate my refresh event

Setting ensure => absent deletes /etc when used as configdir

I am deploying logstash-forwarder to a number of nodes but wanted to exclude the logstash master.

I have done this through Hiera and the following settings:

classes:

  • logstashforwarder

logstashforwarder::servers:

  • ip.ad.d.re.ss
    logstashforwarder::package_url: http://packages.elasticsearch.org/logstashforwar
    der/debian/pool/main/l/logstashforwarder/logstash-forwarder_0.4.0_amd64.deb
    logstashforwarder::configdir: /etc
    logstashforwarder::ssl_ca: /etc/pki/tls/certs/logstash-forwarder.crt

When the logstash master is set to exclude the forwarder with,

logstashforwarder::ensure: absent

Running,
puppet agent --test
notice: /Stage[main]/Logstashforwarder::Config/File[/etc]/ensure: removed

And /etc is no more. Package removal scripts typically take a more forgiving approach to directories that have other data in them.

Support for logstash-forwarder 0.4.0

Presently these manifests do not work with logstash-forwarder 0.4.0 (on Debian at least, haven't tried the RPM, but I suspect the same issues would be present), because the /etc/defaults file it writes is missing several settings required for the new init script

Port setting

Hi, thanks for this puppet module. Two documentation-related questions:

  1. What is the default port setting for the logstash forwarder?
  2. How do you explicitly set the port? I'm assuming something like:
 class { 'logstashforwarder':
   servers  => [ 'logstash.yourdomain.com:5000' ],
...

RHEL5 / CentOS5 compatible repo/rpm?

I found a signed CentOS6/RHEL6 logstash forwarder RPM for installation here:
http://packages.elasticsearch.org/logstashforwarder/centos/logstash-forwarder-0.3.1-1.x86_64.rpm

However, using puppet-logstashforwarder, when i try to install it on RHEL5, it gives the following error because it's using v4 signatures, whereas CentOS5/RHEL5 only support v3 signatures:
Error: Execution of '/bin/rpm -i /opt/logstashforwarder/swdl/logstash-forwarder-0.3.1-1.x86_64.rpm' returned 1: error: /opt/logstashforwarder/swdl/logstash-forwarder-0.3.1-1.x86_64.rpm: Header V4 RSA/SHA1 signature: BAD, key ID d88e42b4

Is there an unsigned, or v3 signature compatible rpm/repo somewhere that I'm missing?

Thanks,

Matt.

Monitor files with foreman

I have a foreman instance setup and am having problems telling log stash what files to monitor. I am able to set the server and certificate using overrides within foreman, but for the life of me I can not figure out how to set what log files to monitor. Has that been encountered before?

undefined method `sort' for nil:NilClass

I am using PE 2015.2, it comes with
puppetlabs-stdlib (v4.9.0)

After setting the ssl parameters to empty string in yaml file,

logstash::package_url: 'file:/vagrant/salt/states/puppetmaster/files/elk-external-resources/logstash-1.5.4-1.noarch.rpm'
elasticsearch::java_install: true
elasticsearch::package_url: 'file:/vagrant/salt/states/puppetmaster/files/elk-external-resources/elasticsearch-1.7.3.noarch.rpm'
elasticsearch::instances:
'es-01':
'config':
'node.name': 'es-01'

logstashforwarder::package_url: 'file:/vagrant/salt/states/puppetmaster/files/elk-external-resources/logstash-forwarder-0.4.0-1.x86_64.rpm'
logstashforwarder::ssl_key: ''
logstashforwarder::ssl_cert: ''
logstashforwarder::ssl_ca: ''

I get the following error:

[root@elkagent ~]# puppet agent -t
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Evaluation Error: Error while evaluating a Function Call, Failed to parse inline template: undefined method `sort' for nil:NilClass at /etc/puppetlabs/code/environments/production/modules/logstashforwarder/manifests/config.pp:117:20 on node elkagent.home
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
[root@elkagent ~]#

config.json is not created

Hey, some more feedback for your module! 👍

On a clean system, it seems like the config.json file is not created; it seems the logstashforwarder_config type is responsible for this. I don't know ruby, but maybe it is a matter of adding the following line to the top of lib/puppet/type/logstashforwarder_config.rb

require 'puppet/type/file/ensure'

For now I was able to work around it by adding this to the config.pp of the module:

    file { "${logstashforwarder::configdir}/config.json":
        ensure  => present,
        require => File[ $logstashforwarder::configdir ],
        before  => Logstashforwarder_config [ 'lsf-config' ]
    }

init scripts are for logstash

the init scripts provided are just a straight copy of the Java-based LogStash script, they don't run logstashforwarder.

$configdir is ignored for logstash-forwarder.conf

The path to the config file is hardcoded in config.pp

logstashforwarder_config { 'lsf-config':
  ensure  => 'present',
  config  => $main_config,
  path    => '/etc/logstash-forwarder.conf',
  tag     => "LSF_CONFIG_${::fqdn}",
  owner   => $logstashforwarder::logstashforwarder_user,
  group   => $logstashforwarder::logstashforwarder_group,
  notify  => $notify_service,
  require => File[$logstashforwarder::configdir]
}

Beaker publickey - test fails

When we pull request, there's a beaker error. And I cannot find out in the web how to deal with this message by myseilf. Maybe an administrator should have a look at this.

Fetching [email protected]:puppetlabs/beaker.git
Warning: Permanently added the RSA host key for IP address '192.30.252.131' to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

init script

Hey Richard, thanks for the module!

At the moment supplied init script is pointing to the wrong config file location and fails silently.

DAEMON_ARGS="-config /etc/logstash-forwarder -spool-size 100 -log-to-syslog"

  1. Logstashforwarder gets installed in /etc/logstashforwarder (on Ubuntu anyway)

  2. Looks like Logstashforwarder requires the config file and not the directory

Cheers,

Simon

Could not autoload puppet/type/logstashforwarder_config

I'm busy testing the logstashforwarder puppet module in my Puppet Enterprise (3.7.1) setup but I'm encountering this error:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not autoload puppet/type/logstashforwarder_config: uninitialized constant Puppet::Type::File::Owner on node xxx.yyy.zz

Any help is appreciated to resolve this bug.

Maintenance?

I was using the module to start integrating logstash at one of my sites.

The module seems a bit unmaintained when looking on the PRs...

Whats the status here?

modify timeout feature

This module is explicitly setting Puppet's default value of 300 seconds on the exec timeouts, and this takes precedence over a global Exec timeout setting. It would be better if it didn't touch the Exec timeout setting unless someone asks it to. That would enable people to have:

Exec {
  timeout => 0,
}

include ::logstashforwarder

If no objections, I will submit a PR to change this behaviour.

Failed to parse inline template

Hi,

I am just using the logstashforwarder as described in the example:

class { 'logstashforwarder': }

The result is the following line:
Failed to parse inline template: undefined method `sort' for nil:NilClass at /tmp/vagrant-puppet/modules-5fb4a587328a1815a7de96967c07c0f5/logstashforwarder/manifests/config.pp:117

Can somebody tell me what I am doing wrong?
Puppet Version 3.8.2

Greetings
Jan

running twice: package is already installed errors

With the following configuration, the second run of puppet errors out - it seems to try to install the RPM twice. Obviously, this is on RedHat. I wasn't able to get the repo working because of CentOS/RHEL signing issues (reported elsewhere).

Configuration -

    $config_hash = {
            'logstashforwarder_USER' => 'root',
            'logstashforwarder_GROUP' => 'root',
    }

    # https://github.com/elasticsearch/puppet-logstashforwarder
    class { 'logstashforwarder':
            init_defaults => $config_hash,
            #manage_repo => true,
            package_url => 'http://download.elasticsearch.org/logstash/logstash/packages/centos/logstash-1.3.3-1_centos.noarch.rpm',
            servers => ['logs:12345'],
            #ssl_key => 'puppet:///modules/test-logstashforwarder/files/logstash-forwarder.key'
            #ssl_ca => 'puppet:///modules/test-logstashforwarder/files/logstash-forwarder.crt'
            #ssl_cert => 'puppet:///modules/test-logstashforwarder/files/logstash-forwarder.crt'
            logstashforwarder_user => 'root',
            logstashforwarder_group => 'root',
            timeout => 15,
    }

Errors -

Error: Execution of '/bin/rpm -i /opt/logstashforwarder/swdl/logstash-1.3.3-1_centos.noarch.rpm' returned 1: package logstash-1.3.3-1_centos.noarch is already installed
Error: /Stage[main]/Logstashforwarder::Package/Package[logstash-forwarder]/ensure: change from absent to present failed: Execution of '/bin/rpm -i /opt/logstashforwarder/swdl/logstash-1.3.3-1_centos.noarch.rpm' returned 1: package logstash-1.3.3-1_centos.noarch is already installed

[Discussion] Multi instance support?

Do we need multi instance support for this module?
This could be instances that are managed by a service manager ( init ) or not.
Some of these instances can be used then to be run from the command line via stdin or other things.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.