Giter Club home page Giter Club logo

beaver's People

Contributors

deepsource-io[bot] avatar ober avatar wwalker avatar

Watchers

 avatar  avatar

beaver's Issues

workaround kernel bug

Work around the bug in the linux kernel on Amazon Linux 2 (AL2) that randomly deletes the log file. We see this exact same behavior in elasticsearch and logstash (both are running in Java and using auto-rotation via Log4j), only on AL2...

The short story - the AL2 linux kernel appears to have a race condition triggered bug which causes a rename() call to first rename the file as requested (which is a dirent action), and then within a second or two, it also deletes the dirent for a new file of the same name that we just previously rename()d...

In Amazon Linux 2 (AL2) there is a bug in the kernel (in the current version (4.14.314-238.539.amzn2.x86_64), and in previous versions), where, sometimes (often as frequently as 99% of the time, sometimes as rarely as 1 in 20,000 rotates) where the kernel removes the dirent for a newly created file, if a file of the same name was recently rename()d.

We can only reproduce this on an AWS EC2 instance running AL2. It can always be reproduced within an hour, often within 10 seconds.

beaver makes these kernel calls:

22:37:16.316753 access("/var/log/funnel.log", F_OK) = -1 ENOENT (No such file or directory)
^ This is OK, we try to rename the file at start

# Now we create the first funnel.log
22:37:16.316792 open("/var/log/funnel.log", O_RDWR|O_CREAT|O_APPEND|O_LARGEFILE|O_CLOEXEC, 0644) = 7

# and it is there until we decide to rotate it:
22:37:16.316928 access("/var/log/funnel.log", F_OK) = 0
22:37:27.637259 access("/var/log/funnel.log", F_OK) = 0
22:37:27.637432 access("/var/log/funnel.log", F_OK) = 0
22:37:27.637682 access("/var/log/funnel.log", F_OK) = 0
22:37:27.637793 access("/var/log/funnel.log", F_OK) = 0

# We rename it and funnel.log is, expectedly, gone:
22:37:27.637854 rename("/var/log/funnel.log", "/var/log/funnel.log.2023-08-23T22.37.27.9637776910Z") = 0
22:37:27.637908 access("/var/log/funnel.log", F_OK) = -1 ENOENT (No such file or directory)

# We open a new funnel.log, and it is there a millisecond after we create it
22:37:27.638023 open("/var/log/funnel.log", O_RDWR|O_CREAT|O_APPEND|O_LARGEFILE|O_CLOEXEC, 0644) = 7
22:37:27.638186 access("/var/log/funnel.log", F_OK) = 0

# One second later, the NEW funnel.log is gone.  We didn't delete it.
# Looking in ls -l /proc/$$/fd we see fd 7 is open and the file is growing, but ...
# its name is now (deleted), meaning the kernel has removed the dirent for the
# NEW funnel.log
22:37:28.640370 access("/var/log/funnel.log", F_OK) = -1 ENOENT (No such file or directory)
22:37:31.609737 access("/var/log/funnel.log", F_OK) = -1 ENOENT (No such file or directory)
22:37:31.609922 access("/var/log/funnel.log", F_OK) = -1 ENOENT (No such file or directory)
22:37:31.610257 access("/var/log/funnel.log", F_OK) = -1 ENOENT (No such file or directory)
22:37:31.610462 access("/var/log/funnel.log", F_OK) = -1 ENOENT (No such file or directory)
22:37:31.611112 rename("/var/log/funnel.log", "/var/log/funnel.log.2023-08-23T22.37.31.9610427158Z") = -1 ENOENT (No such file or directory)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.