wwalker / beaver Goto Github PK
View Code? Open in Web Editor NEWbeaver - logs its stdin to a file, and auto-rotates and compresses the log files
License: GNU General Public License v3.0
beaver - logs its stdin to a file, and auto-rotates and compresses the log files
License: GNU General Public License v3.0
Work around the bug in the linux kernel on Amazon Linux 2 (AL2) that randomly deletes the log file. We see this exact same behavior in elasticsearch
and logstash
(both are running in Java and using auto-rotation via Log4j
), only on AL2...
The short story - the AL2 linux kernel appears to have a race condition triggered bug which causes a rename()
call to first rename the file as requested (which is a dirent action), and then within a second or two, it also deletes the dirent for a new file of the same name that we just previously rename()
d...
In Amazon Linux 2 (AL2) there is a bug in the kernel (in the current version (4.14.314-238.539.amzn2.x86_64
), and in previous versions), where, sometimes (often as frequently as 99% of the time, sometimes as rarely as 1 in 20,000 rotates) where the kernel removes the dirent for a newly created file, if a file of the same name was recently rename()
d.
We can only reproduce this on an AWS EC2 instance running AL2. It can always be reproduced within an hour, often within 10 seconds.
beaver
makes these kernel calls:
22:37:16.316753 access("/var/log/funnel.log", F_OK) = -1 ENOENT (No such file or directory)
^ This is OK, we try to rename the file at start
# Now we create the first funnel.log
22:37:16.316792 open("/var/log/funnel.log", O_RDWR|O_CREAT|O_APPEND|O_LARGEFILE|O_CLOEXEC, 0644) = 7
# and it is there until we decide to rotate it:
22:37:16.316928 access("/var/log/funnel.log", F_OK) = 0
22:37:27.637259 access("/var/log/funnel.log", F_OK) = 0
22:37:27.637432 access("/var/log/funnel.log", F_OK) = 0
22:37:27.637682 access("/var/log/funnel.log", F_OK) = 0
22:37:27.637793 access("/var/log/funnel.log", F_OK) = 0
# We rename it and funnel.log is, expectedly, gone:
22:37:27.637854 rename("/var/log/funnel.log", "/var/log/funnel.log.2023-08-23T22.37.27.9637776910Z") = 0
22:37:27.637908 access("/var/log/funnel.log", F_OK) = -1 ENOENT (No such file or directory)
# We open a new funnel.log, and it is there a millisecond after we create it
22:37:27.638023 open("/var/log/funnel.log", O_RDWR|O_CREAT|O_APPEND|O_LARGEFILE|O_CLOEXEC, 0644) = 7
22:37:27.638186 access("/var/log/funnel.log", F_OK) = 0
# One second later, the NEW funnel.log is gone. We didn't delete it.
# Looking in ls -l /proc/$$/fd we see fd 7 is open and the file is growing, but ...
# its name is now (deleted), meaning the kernel has removed the dirent for the
# NEW funnel.log
22:37:28.640370 access("/var/log/funnel.log", F_OK) = -1 ENOENT (No such file or directory)
22:37:31.609737 access("/var/log/funnel.log", F_OK) = -1 ENOENT (No such file or directory)
22:37:31.609922 access("/var/log/funnel.log", F_OK) = -1 ENOENT (No such file or directory)
22:37:31.610257 access("/var/log/funnel.log", F_OK) = -1 ENOENT (No such file or directory)
22:37:31.610462 access("/var/log/funnel.log", F_OK) = -1 ENOENT (No such file or directory)
22:37:31.611112 rename("/var/log/funnel.log", "/var/log/funnel.log.2023-08-23T22.37.31.9610427158Z") = -1 ENOENT (No such file or directory)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.