Comments (3)
jedisct1
commented on August 24, 2024
Hi,
And thanks for using Flowgger!
I would recommend using a structured format such as ltsv, which works really well with Nginx, instead of syslog messages, whose format vary according to the syslog daemon and its configuration.
What does a line of log look like?
from flowgger.
jedisct1
commented on August 24, 2024
The system syslog daemon on macOS still uses the very old RFC3164 format, that got obsolete with RFC5424.
The old format has limitations. Timestamps cannot be reliably parsed, and payloads are limited to strings. Key/value pairs are not supported.
Flowgger doesn't support the old format, only the RFC5424 one, which is supported by common logging daemons such as rsyslogd.
Still, even RFC5424 is terrible. Slow, complicated and limited. Use LTSV.
from flowgger.
progamer71
commented on August 24, 2024
Thanks you for quick response.
This is the example of error.log
2016/10/02 17:05:36 [emerg] 1395#0: open() "./log/error.log" failed (2: No such file or directory)
2016/10/02 17:05:53 [emerg] 1399#0: open() "./logs/nginx.pid" failed (2: No such file or directory)
2016/10/02 17:06:46 [alert] 1407#0: setrlimit(RLIMIT_NOFILE, 100000) failed (1: Operation not permitted)
2016/10/02 17:06:46 [alert] 1408#0: setrlimit(RLIMIT_NOFILE, 100000) failed (1: Operation not permitted)
2016/10/02 17:06:46 [alert] 1409#0: setrlimit(RLIMIT_NOFILE, 100000) failed (1: Operation not permitted)
2016/10/02 17:06:46 [alert] 1410#0: setrlimit(RLIMIT_NOFILE, 100000) failed (1: Operation not permitted)
2016/10/04 14:55:35 [emerg] 4414#0: open() "./conf/nginx.conf" failed (2: No such file or directory)
after search for a while, i found that nginx generate 2 log files
1 access_log: the format can be customized in configuration file
2 error_log: the format is hard coded in src/core/ngx_log.c
YYYY/MM/DD HH:MM:SS [LEVEL] PID#TID: *CID MESSAGE
(reference http://stackoverflow.com/questions/16711573/nginx-error-log-format-documentation)
My use case is to create a centralized logging system from many nginx servers.
The connection need TLS and compression.
So flowgger seem to fit my use case.
Right now my solution is
nginx.conf
error_log logs/error.log;
...
access_log logs/access.log;
flowgger.toml
[input]
### Standard input
type = "stdin"
[output]
### TLS output
type = "tls"
connect = [ "172.16.205.128:6514", "172.16.205.129:6514" ]
timeout = 3600
tls_threads = 1
tls_cert = "flowgger.pem"
tls_key = "flowgger.pem"
tls_ca_file = "flowgger.pem"
# tls_compatibility_level = "intermediate"
# tls_verify_peer = false
tls_compression = true
# tls_ciphers = "EECDH+AES128:EECDH+CHACHA20:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3$
# tls_async = false
# tls_recovery_delay_init = 1
# tls_recovery_delay_max = 10000
# tls_recovery_probe_time = 30000
run command
tail -f -n 0 logs/error.log | flowgger flowgger.toml &
tail -f -n 0 logs/access.log | flowgger flowgger.toml &
it is not the best solution but good enough for my use case
Thanks you for your great work
from flowgger.
Related Issues (20)
- Publish to crates.io HOT 10
- Update dependencies version HOT 2
- Add file size/time based rotation
- Add log pass-through feature HOT 1
- Increase unit test coverage and add performance testing
- GELF decoder doesn't respect output.gelf_extra add or replace fields
- LTSV decoder doesn't respect output.ltsv_extra add or replace fields
- coio is not build in cargo.io migrate to tokyo the tls module HOT 6
- Lua Parsing HOT 1
- Timezone ? HOT 1
- Comparision to Vector (https://vector.dev/) HOT 5
- Decoder RFC 5424 does not correctly parse structured data HOT 2
- Log file naming hardcode the time and extensions separators
- Enhancement request: NATS Output HOT 1
- Release a new version HOT 2
- Build fail with redis feature HOT 1
- Send and receive utf-8 messages HOT 1
- Add timestamps HOT 2
- Can I create a simple effective filter using config file flowgger.toml?
- Cross build workflow fails again HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flowgger.