parse-nl / systemdjournal2gelf Goto Github PK
View Code? Open in Web Editor NEWExport entries from systemds' journal and send them to a graylog server using gelf
License: BSD 2-Clause "Simplified" License
Export entries from systemds' journal and send them to a graylog server using gelf
License: BSD 2-Clause "Simplified" License
This Go-Binary currently doesn't support sending the logs via TCP.
For a more secure Graylog deployment sending logs via TCP with TLS is recommended.
Is there a way to implement the option of using TCP as transport instead of UDP?
Trying this for the first time:
$ ~/go/bin/SystemdJournal2Gelf 127.0.0.1:12201 --merge
panic: could not parse journal output: json: cannot unmarshal array into Go struct field entryAlias.SYSLOG_FACILITY of type string
goroutine 1 [running]:
main.main()
/home/arno/go/src/github.com/parse-nl/SystemdJournal2Gelf/SystemdJournal2Gelf.go:207 +0x5b3
$ ~/go/bin/SystemdJournal2Gelf 127.0.0.1:12201 _TRANSPORT=kernel
panic: runtime error: slice bounds out of range
goroutine 1 [running]:
main.(*SystemdJournalEntry).isJsonMessage(...)
/home/arno/go/src/github.com/parse-nl/SystemdJournal2Gelf/SystemdJournal2Gelf.go:128
main.(*SystemdJournalEntry).toGelf(0xc4200e86e0, 0x7f2110535d90)
/home/arno/go/src/github.com/parse-nl/SystemdJournal2Gelf/SystemdJournal2Gelf.go:57 +0x78c
main.(*SystemdJournalEntry).send(0xc4200e86e0)
/home/arno/go/src/github.com/parse-nl/SystemdJournal2Gelf/SystemdJournal2Gelf.go:115 +0x2f
main.(*pendingEntry).Push(0xc42009e160, 0xc4200a4000, 0x7b, 0x571c27d944a13, 0xc4200a60d0, 0xb, 0xc4200c2000, 0x20, 0xc4200a6008, 0x6, ...)
/home/arno/go/src/github.com/parse-nl/SystemdJournal2Gelf/SystemdJournal2Gelf.go:140 +0xc9
main.main()
/home/arno/go/src/github.com/parse-nl/SystemdJournal2Gelf/SystemdJournal2Gelf.go:210 +0x396
It looks like that only the --follow
works well:
$ ~/go/bin/SystemdJournal2Gelf 127.0.0.1:12201 --follow
$ echo 'test message to journald' | systemd-cat
Probably people are configuring journald write to files and feed them to filebeats, since filebeats isn't reading the systemd journald directly yet elastic/beats#7955
FYI, I created an Arch Linux package for your software:
https://aur.archlinux.org/packages/systemdjournal2gelf/
It has a configuration file to set the Gelf-Endpoint-Url and installs the systemd service from your repo.
Maybe you would like to add it to your readme for your users :)
Thanks for your good-working piece of software :)))
$ go version
go version go1.13.8 linux/amd64
$ cd ./go/bin/ && ./SystemdJournal2Gelf --help
panic: usage: SystemdJournal2Gelf SERVER:12201 [JOURNALCTL PARAMETERS]
goroutine 1 [running]:
main.main()
~/go/src/github.com/parse-nl/SystemdJournal2Gelf/SystemdJournal2Gelf.go:164 +0x64d
$ ./SystemdJournal2Gelf localhost:12201 -- --version
Failed to add match '--version': Invalid argument
# ^^ I would expect it to be passed to journald...
$ ./SystemdJournal2Gelf localhost:12201 --version
panic: could not parse journal output: invalid character 's' looking for beginning of value
# ^^ Why?
OVH hosted Graylog instance (https://www.ovh.com/fr/data-platforms/logs/) works on authentication. The TCP payload must include a specific string X-OVH-TOKEN: authentication
(could be a header too).
README.md reads This script supports a special syntax to send additional properties; when you log a JSON encoded object in the Message field it Unmarshalls it for you
but I don't see how to make a use of this to solve above issue.
the message cleanup feature was introduced when graylog didn't support any similar functionality. It provides message extractors now - so this client doesn't necessarily need to keep providing this
Recieved another issue when merging:
Error from Scanner: bufio.Scanner: token too long
remove json
parsing from SystemdJournal2Gelf
- let users configure this themselves as a pipeline. Steps to recreate this as a custom pipeline:
Go to /system/pipelines
in your graylog install and create a new pipeline
Edit the new pipeline and add a stage:
In the first stage, add a rule to interpret the message as json:
rule "interpret message as JSON"
when
has_field("message") && starts_with(to_string($message.message), "{\"")
then
let json = parse_json(to_string($message.message));
let map = to_map(json);
set_fields(map);
rename_field("Message", "message");
rename_field("FullMessage", "full_message");
end
php-fpm
:rule "strip fpm pool prefix"
when
has_field("message") && starts_with(to_string($message.message), "pool ")
then
set_field("message", regex_replace("^pool [a-zA-Z_\\[\\d\\]]+: ", to_string($message.message), ""));
end
Pipeline Processor
is configured after Message Filter Chain
in system/configurations
I get the following error on a merge:
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x402d76]
goroutine 1 [running]:
panic(0x602900, 0xc820010130)
/usr/lib/go/src/runtime/panic.go:481 +0x3e6
main.main()
message can be an array of numbers [ 116, 104, 105, 115, 32, 105, 115, 32, 97, 32, 98, 105, 110, 97, 114, 121, 32, 118, 97, 108, 117, 101, 32, 7 ],
- we could use a custom json parser to handle this
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.