Status (master branch)
log_proxy is a tiny C utility for log rotation for apps that write their logs to stdout.
This is very useful, specially for 12-factor apps that write their logs to stdout.
It can be used to avoid loosing some logs if you use logrotate with copytruncate feature or to prevent a log file from filling your hard disk.
- usable as a pipe (
myapp myapp_arg1 myapp_arg2 |log_proxy /log/myapp.log) - configurable log rotation suffix with
strftimeplaceholders (for example:.%Y%m%d%H%M%S) - can limit the number of rotated files (and delete oldest)
- can rotate files depending on their size (in bytes)
- can rotate files depending on their age (in seconds)
- does not need a specific log directory for a given app (you can have one directory with plenty of different log files from different apps)
- several instances of the same app can log to the same file without issue (example:
myapp arg1 |log_proxy --use-locks /log/myapp.logandmyapp arg2 |log_proxy --use-locks /log/myapp.logcan run at the same time) - configurable action (a command to execute) to run after each log rotation
- rock solid (it's perfectly stable in our use case but we are waiting for other success stories to check this feature)
- option to add a timestamp before each log line, thanks to mk-fg
- really fast
- do not eat a lot of memory
- configurable with CLI options as well with env variables
- usable as a wrapper to capture stdout and stderr (
log_proxy_wrapper --stdout=/log/myapp.stdout --stderr=/log/myapp.stderr -- myapp myapp_arg1 myapp_arg2) - usable as a wrapper to capture stdout and stderr in the same file (
log_proxy_wrapper --stdout=/log/myapp.log --stderr=STDOUT -- myapp myapp_arg1 myapp_arg2) - very few dependencies (only
glib2is required) - very easy to build (event on old distributions like
CentOS 6)
We provide Linux 64 bits binaries in releases section. There is virtually no requirement (you just need a Linux 64 bits distribution more recent than CentOS 6 (2011!)).
Of course, you can also build the tool by yourself (see at the end of this document).
To install the binary distribution:
# As root user (or with sudo)
bash -c "$(curl -fsSLk https://raw.githubusercontent.com/metwork-framework/log_proxy/master/installer/install.sh)"
Notes:
- if you are very concerned about the security of your system and if you don't want to execute
a remote
rootscript on your system, please review the very small install script (it's just about downloading and installing two statically compiled binaries in/usr/local/bin/) - our binary distribution won't work on Alpine Linux because of
glibcreplacement but @tomalok is maintaining a log_proxy Alpine Linux package.
# As root user (or with sudo)
rm -f /usr/local/bin/log_proxy
rm -f /usr/local/bin/log_proxy_wrapper
If you use myapp myapp_arg1 myapp_arg2 >/log/myapp.log 2>&1for example and if you can't stop easily your app (because it's a critical thing), you can configure logrotate with copytruncate feature to do the log rotation of /log/myapp.log but:
- you may loose a few lines during log rotation (1)
- the rotation is mainly time-based, so you can fill your storage if your app suddently start to be very very verbose
(1), see https://unix.stackexchange.com/questions/475524/how-copytruncate-actually-works
Please note also that copyrotate has an inherent race condition, in that it's possible that the writer will append a line to the logfile just after logrotate finished the copy and before it has issued the truncate operation. That race condition would cause it to lose those lines of log forever. That's why rotating logs using copytruncate is usually not recommended, unless it's the only possible way to do it.
After reading: https://superuser.com/questions/291368/log-rotation-of-stdout and http://zpz.github.io/blog/log-rotation-of-stdout/, we reviewed plenty of existing tools (multilog, rotatelogs, piper...).
But none of them was ok with our needed features:
- configurable log rotation on size AND age
- no dedicated log directory for an app
- (and) several instances of the same app can log to the same log file without issue
The piper tool was the more close but does not support the last feature (several instances to the same log file).
your_app your_app_arg1 your_app_arg2 2>&1 |log_proxy --rotation-size=1000000 my_log_file_max_1MB.logFull help:
$ ./log_proxy --help
Usage:
log_proxy [OPTION?] LOGNAME - log proxy
Help Options:
-h, --help Show help options
Application Options:
-s, --rotation-size maximum size (in bytes) for a log file before rotation (0 => no maximum, default: content of environment variable LOGPROXY_ROTATION_SIZE or 104857600 (100MB))
-t, --rotation-time maximum lifetime (in seconds) for a log file before rotation (0 => no maximum, default: content of environment variable LOGPROXY_ROTATION_TIME or 86400 (24H))
-S, --rotation-suffix strftime based suffix to append to rotated log files (default: content of environment variable LOGPROXY_ROTATION_SUFFIX or .%%Y%%m%%d%%H%%M%%S)
-d, --log-directory directory to store log files (default: content of environment variable LOGPROXY_LOG_DIRECTORY or current directory), directory is created if missing
-n, --rotated-files maximum number of rotated files to keep including main one (0 => no cleaning, default: content of environment variable LOGPROXY_ROTATED_FILES or 5)
-T, --timestamps strftime prefix to prepend to every output line (default: content of environment variable LOGPROXY_TIMESTAMPS or none)
-c, --chmod if set, chmod the logfile to this value, '0700' for example (default: content of environment variable LOGPROXY_CHMOD or NULL)
-o, --chown if set, try (if you don't have sufficient privileges, it will fail silently) to change the owner of the logfile to the given user value
-g, --chgrp if set, try (if you don't have sufficient privileges, it will fail silently) to change the group of the logfile to the given group value
-m, --use-locks use locks to append to main log file (useful if several process writes to the same file)
-f, --fifo if set, read lines on this fifo instead of stdin
-r, --rm-fifo-at-exit if set, drop fifo at then end of the program (you have to use --fifo option of course)
Optional environment variables to override defaults:
LOGPROXY_ROTATION_SIZE
LOGPROXY_ROTATION_TIME
LOGPROXY_ROTATION_SUFFIX
LOGPROXY_LOG_DIRECTORY
LOGPROXY_ROTATED_FILES
LOGPROXY_TIMESTAMPS
Example for rotation-size option :
- If log_proxy is run with the option --rotation-size on command line, rotation-size will take the provided value
- If the option --rotation-size is not provided on command line :
- If the environment variable LOGPROXY_ROTATION_SIZE is set, rotation-size will take this value
- If the environment variable LOGPROXY_ROTATION_SIZE is not set, rotation-size will take the default value 104857600log_proxy_wrapper --rotation-size=1000000 --stdout=my_log_file_max_1MB.log --stderr=STDOUT -- your_app your_app_arg1 your_app_arg2Full help:
$ ./log_proxy_wrapper --help
Usage:
log_proxy_wrapper [OPTION?] -- COMMAND [COMMAND_ARG1] [COMMAND_ARG2] [...] - log proxy
Help Options:
-h, --help Show help options
Application Options:
-s, --rotation-size maximum size (in bytes) for a log file before rotation (0 => no maximum, default: content of environment variable LOGPROXY_ROTATION_SIZE or 104857600 (100MB))
-t, --rotation-time maximum lifetime (in seconds) for a log file before rotation (0 => no maximum, default: content of environment variable LOGPROXY_ROTATION_TIME or 86400 (24H))
-S, --rotation-suffix strftime based suffix to append to rotated log files (default: content of environment variable LOGPROXY_ROTATION_SUFFIX or .%%Y%%m%%d%%H%%M%%S)
-d, --log-directory directory to store log files (default: content of environment variable LOGPROXY_LOG_DIRECTORY or current directory), directory is created if missing
-n, --rotated-files maximum number of rotated files to keep including main one (0 => no cleaning, default: content of environment variable LOGPROXY_ROTATED_FILES or 5)
-T, --timestamps strftime prefix to prepend to every output line (default: content of environment variable LOGPROXY_TIMESTAMPS or none)
-c, --chmod if set, chmod the logfile to this value, '0700' for example (default: content of environment variable LOGPROXY_CHMOD or NULL)
-o, --chown if set, try (if you don't have sufficient privileges, it will fail silently) to change the owner of the logfile to the given user value
-g, --chgrp if set, try (if you don't have sufficient privileges, it will fail silently) to change the group of the logfile to the given group value
-m, --use-locks use locks to append to main log file (useful if several process writes to the same file)
-O, --stdout stdout file path (NULL string (default) can be used to redirect to /dev/null)
-E, --stderr stderr file path (STDOUT string (default) can be used to redirect to the same file than stdout)
-F, --fifo-tmp-dir directory where to store tmp FIFO for log_proxy (default: content of environment variable TMPDIR if set, /tmp if not)A Linux/Unix distribution with standard development tools (git, gcc, make, pkg-config) and glib2 library with devel support (provided for example in CentOS 6 in the glib2-devel standard package).
git clone https://github.com/metwork-framework/log_proxy # or download/unpack a zip with the github interface
cd log_proxy
makeThen as root user or prefixed with sudo:
make installThis will install log_proxy and log_proxy_wrapper in /usr/local/bin, by default.

make install also supports PREFIX=... for installing (for example) into a /usr directory other than the one in /usr/local, and DESTDIR=... for installing into $DESTDIR/$PREFIX/bin which is useful when making packages.


See CONTRIBUTING.md file.
See CODE_OF_CONDUCT.md file.
(If you are officially paid to work on MetWork Framework, please contact us to add your company logo here!)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent