Giter Club home page Giter Club logo

puppet-graylog_api's Introduction

graylog_api

Table of Contents

  1. Description
  2. Setup - The basics of getting started with graylog_api
  3. Usage - Configuration options and additional functionality
  4. Limitations - OS compatibility, etc.
  5. Development - Guide for contributing to the module

Description

This module allows you to use Graylog's REST API to adjust its configuration. It picks up where the official graylog module leaves off.

Setup

What graylog_api affects

This module manages configuration aspects of Graylog that can only be adjusted via the REST API. This includes:

  • LDAP Authentication
  • SSO Authentication
  • Static users
  • User roles
  • Inputs
  • Streams
  • Pipelines and Pipeline rules
  • Extractors
  • Lookup Tables, Data Adapters and Caches
  • Grok Patterns
  • Dashboards
  • Index Sets

More components of Graylog configuration are in scope for this module, but they have not been implemented yet.

Setup Requirements

The Ruby installation used by the Puppet agent on the Graylog server will need to have the httparty and retries gems installed. The easiest way to manage this is to use a package resource with the puppet_gem provider:

package { ['httparty','retries']:
  ensure   => present,
  provider => 'puppet_gem',
}

The server will also need to have Graylog installed, of course. For this we recommend the official graylog-graylog module.

Beginning with graylog_api

In order to use any of the resources contained in this module, you first need to supply the credentials the module should use to access the REST API. In general, this should be the root credentials. Provide these through the graylog_api resource.

graylog_api { 'api':
  username => 'admin',
  password => $password,
  port     => 9000,
  tls      => false,
  server   => 'localhost'
}

The resource title here must be 'api'.

Supplying the password

This module requires the graylog root password in cleartext in order to be able to authenticate to the API. The official graylog module, on the other hand, only needs the password hash. Rather than storing both the password and the hash, we recommend storing the password in Hiera using EYAML, and computing the hash using Puppet's built-in sha256 function.

Usage

Configure the default index set

graylog_index_set { 'graylog':
  description                => 'The Graylog default index set',
  display_name               => 'Default index set',
  shards                     => 1,
  replicas                   => 0,
  rotation_strategy          => 'size',
  rotation_strategy_details  => {
    max_size => '10 GB'.to_bytes,
  },
  retention_strategy         => 'delete',
  retention_strategy_details => {
    max_number_of_indices => 10,
  },
}

Configure inputs

# Default properties are often acceptable
graylog_api::input::gelf_tcp { 'A GELF TCP Input': }

# But you can customize if you want
graylog_api::input::gelf_tcp { 'A GELF TCP Input with TLS':
  port          => 12202,
  tls_cert_file => '/etc/graylog/server/tls/cert.pem',
  tls_enable    => true,
  tls_key_file  => '/etc/graylog/server/tls/key.pem',
}

Load Grok Patterns

# Load a single pattern
graylog_grok_pattern { 'SOMEFORMAT':
  pattern => '%{WORD:username} %{IP:ipaddress} %{GREEDYDATA:message}',
}

# Or load a bunch of patterns from a pattern file
graylog_api::grok::pattern_file { 'common patterns':
  content => file('profile/graylog/patterns/common),
}

Set up processing pipelines

# First set up some rules
graylog_api::pipeline::rule { 'copy message to full_message':
  description => 'Copy the message field to the full_message field before performing extraction',
  condition   => 'has_field("message") && has_field("log_format") && !has_field("full_message")',
  action      => 'set_field("full_message",$message.message);',
}

graylog_api::pipeline::rule { 'parse log format':
  description => 'Parse log via GROK if log_format field is provided',
  condition   => 'has_field("log_format")',
  action      => @(END_OF_ACTION),
                 let format_name = uppercase(to_string($message.log_format));
                 let pattern = concat(concat("%{",format_name),"}");
                 let map = grok(pattern: pattern, value: to_string($message.message), only_named_captures: true);
                 remove_field("log_format");
                 set_fields(map);
                 |-END_OF_ACTION
}

# Then put those rules in a pipeline
graylog_api::pipeline { 'custom log formats':
  description       => "Parse custom log formats",
  stages            => [
    'copy message to full_message',
    'parse log format'
  ],
  connected_streams => ['All messages'],
}

Limitations

There are a lot of different settings in Graylog that this module cannot yet manage. Essentially it only manages those settings that we've needed so far ourselves.

This module aims for compatibility with Graylog 3.x, and specifically has been tested with Graylog 3.1.x. It probably works on later versions of Graylog 3.x but may not work with 4.x or 2.x.

If you discover any issues, please report them at https://github.com/magicmemories/puppet-graylog_api/issues

License and Authorship

This module was authored by Adam Gardner, and is Copyright (c) 2019 Magic Memories (USA) LLC.

It is distributed under the terms of the Apache-2.0 license; see the LICENSE file for details.

puppet-graylog_api's People

Contributors

bastelfreak avatar natlibfi-arlehiko avatar philomory avatar vstone avatar witjoh avatar zilchms avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

vstone witjoh

puppet-graylog_api's Issues

Add support for changing the order of processors

Hi,

Changing the order of processors allows to hook up pipelines to streams, instead of going through pipelines first.
I did some digging in the API for Graylog.

API endpoint: /api/system/cluster_config/org.graylog2.messageprocessors.MessageProcessorsConfig
To get the current config: GET
To update the config: PUT

The response data (for both methods) is the same as the data to send for an update:
Example

{
  "processor_order": [
    "org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor",
    "org.graylog.plugins.map.geoip.processor.GeoIpProcessor",
    "org.graylog2.messageprocessors.MessageFilterChainProcessor",
    "org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter"
  ],
  "disabled_processors": []
}

graylog_ldap_settings api call changed in version +4.x

Debug: Prefetching graylog_api resources for graylog_ldap_settings
Debug: GET request for https://localhost.:9000/api/system/ldap/settings with params {}
Error: Got error response {"type":"ApiError","message":"HTTP 404 Not Found"}
Error: Could not prefetch graylog_ldap_settings provider 'graylog_api': 

Looking into it preparing PR to solve this.

This is now done using

https://localhost:9000/api/system/authentication/services/backends
https://localhost:9000/api/system/authentication/services/configuration

api calls

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.