40a / bitfan Goto Github PK

View Code? Open in Web Editor NEW

This project forked from vjeantet/bitfan

0.0 2.0 0.0 2.56 MB

rewrite logstash with go

License: Apache License 2.0

Go 100.00%

bitfan's Introduction

Bitfan

Logstash like implementation with GO.

Install

2 ways to get bitfan : download a released version or compile it from source.

Download binary

linux, windows, osx available here : https://github.com/veino/bitfan/releases

Get source and compile

$ go get -u github.com/veino/bitfan

Run

$ bitfan run https://raw.githubusercontent.com/veino/bitfan/master/examples.d/simple.conf

logstash flags works as well `-f`, `-e`, `--configtest`, ...

copy/paste this in your console

127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"

Other command

type bitfan help to display usage information

Usage:
  bitfan [flags]
  bitfan [command]

Available Commands:
  doc         Display documentation about plugins
  list        List running pipelines
  run         Run bitfan
  service     Install and manage bitfan service
  start       Start a new pipeline in a running bitfan
  stop        Stop a running pipeline
  test        Test configurations (files, url, directories)
  version     Display version informations

Flags:
  -f, --config string       Load the Logstash config from a file a directory or a url
  -t, --configtest          Test config file or directory
      --debug               Increase verbosity to the last level (trace), more verbose.
  -e, --eval string         Use the given string as the configuration data.
  -w, --filterworkers int   number of workers (default 4)
  -h, --help                help for bitfan
  -l, --log string          Log to a given path. Default is to log to stdout.
      --settings string     Set the directory containing the bitfan.toml settings (default "current dir, then ~/.bitfan/ then /etc/bitfan/")
      --verbose             Increase verbosity to the first level (info), less verbose.
  -V, --version             Display version info.

Use "bitfan [command] --help" for more information about a command.

include configuration from other configuration

include configuration from an URL

$ bitfan run "input{stdin{}} filter{use{path=>'https://raw.githubusercontent.com/veino/bitfan/master/examples.d/use/lol/test.conf'}} output{stdout{codec=>rubydebug}}"

use configuration file on local filesystem

$ bitfan run "input{stdin{}} filter{use{path=>'apachelogs.conf'}} output{stdout{codec=>rubydebug}}"

See examples in examples.d/use/ folder

TODO

Supported inputs, filters and outputs in config file

can be found here : https://github.com/veino/veino/tree/master/processors

type bitfan doc to list all available plugins

INPUT

PLUGIN	DESCRIPTION
twitter
exec
unix
stdin	Reads events from standard input
file
beats
rabbitmq
udp
syslog
readfile

type bitfan doc pluginname to get more information about plugin configuration and usage

FILTER

PLUGIN	DESCRIPTION
date	Parses dates from fields to use as the Bitfan timestamp for an event
grok
split	Splits multi-line messages into distinct events
json	Parses JSON events
uuid	Adds a UUID to events
drop	Drops all events
geoip	Adds geographical information about an IP address
kv	Parses key-value pairs
html
mutate

type bitfan doc pluginname to get more information about plugin configuration and usage

OUTPUT

PLUGIN	DESCRIPTION
stdout	Prints events to the standard output
null	Drops everything received
file
glusterfs
statsd
mongodb
elasticsearch
elasticsearch2
rabbitmq
email	Sends email when output is received