lukeshirnia / out-of-memory Goto Github PK

Out Of Memory Analyzer can be used to obtain and summarize "out of memory" issues logged by a Linux kernel (when oom-killer is invoked)

License: Apache License 2.0

Python 94.10% Dockerfile 1.35% Shell 4.55%

linux-kernel linux-distribution linux oom ubuntu centos redhat debian memory-allocation memory

out-of-memory's Introduction

Out-Of-Memory Investigation .py

Python 2.7.x - 3.x compatible

The following python script can be used to calculate the estimated RSS (RAM) value of each service at the time a kernel invoked OOM killer.

At the time of an OOM incident, the system logs the estimated RSS value of each service in its system log. Based off of this information the script will calculate how much RAM the services were "theoretically" trying to use, the total RAM value of all services and how much RAM your system actually has to offer these services. Allowing for further investigation into the memory usage of the top "offending" service(s).

The script can check:

Default system log
Journald
Dmesg

This script is designed to run on the following systems (that have python 2.7+):

Ubuntu
Debian
RHEL
CentOS
RockyLinux
AlamaLinux
OSX

More systems can be added. Please request here if you want something specific.

Running

Installation/Running

There are 3 methods of running the script:

Via git:
- git clone [email protected]:LukeShirnia/out-of-memory.git;
- cd out-of-memory;
- python oom_investigate.py
Installation via pip:
- git clone [email protected]:LukeShirnia/out-of-memory.git;
- cd out-of-memory;
- pip install .
- oom_investigate
curl https://raw.githubusercontent.com/LukeShirnia/out-of-memory/v2/oom_investigate.py | python
- WARNING: This isn't safe. You should probably grab the latest commit and curl that instead if you REALLY want to curl.

Usage:

Output Information:

The output from this script can be broken down into 4 main sections:

System info
Log file info
Warning overview
Incident overview

Section 1 - System Info

This section gives you a quick breakdown of the sytem the script has been executed on

Section 2 - Log File Info

Information about which log file the script is using

Section 3 - Warning Overview

If at least one oom incident is detected, the script will first run, then create a summary overview.
It will attempt to inform you of the following information:

How many incidents have occurred in log
What services were killed and how many times
What incident was the worst and the RAM consumed at the time of this incident

Section 4 - Incident Overview

By default the script will show the first 5 OOM incidents. You can show more or less. You can also show the incidents in reverse with --reverse flag.

If there are a large amount of oom incidents, the top RAM consuming incident probably won't be in the script output unless --all is specified.
To make sure we always show the worst incident, a new section will be created at the beginning of the output:

out-of-memory's People

Stargazers

Watchers

Forkers

visjedi davidcoheny massaox daverigby

out-of-memory's Issues

FR: Add override for 300MB file limit

Currently the script does not run when the file is larger than 300Mb.
Add an option like --override to allow to bypass this limit if the device has a significant amount of RAM.

Note: Limit was put in place for small devices. Maybe add a check for free RAM and base the size of the file on that.

FR: Find OOM incident where a specific process was killed

Add a flag (such as --killed) that can be used to find and display oom incidents where a specific service was killed. Maybe accept a list for multiple killed services.

Add journalctl compatibility

Fedora 28 (and soon RHEL 8) will require journalctl compatibility as they do not log to /var/log/messages as before.
Add this functionality

--quick missing file

-q, --quick option doesn't report on the main log file

Option to quickly scan all logs --quick-scan

Need to add the ability to scan all logs, including compressed, with date and time (occurrences)

Ctrl+c, ctrl+d error (--quick)

[root@lga-db ~]# monkey.py -o -- -q
Downloading oom tool ...
----------------------------------------
      _____ _____ _____ 
     |     |     |     |
     |  |  |  |  | | | |
     |_____|_____|_|_|_|
     Out Of Memory Analyser

Disclaimer:
If system OOMs too viciously, there may be nothing logged!
Do NOT take this script as FACT, investigate further
----------------------------------------
Checking other logs, select an option:
Option: 1  /var/log/messages          - Occurrences: 1
           /var/log/messages-20170806.gz - Occurrences: 0
Option: 2  /var/log/messages-20170814.gz - Occurrences: 1
Option: 3  /var/log/messages-20170820.gz - Occurrences: 1
           /var/log/messages-20170827.gz - Occurrences: 0

Which file should we check next?
Select an option number between 1 and 3: ^C
Traceback (most recent call last):
  File "/home/rack/monkeys/monkey-2517a72ed6.py", line 1479, in <module>
    main()
  File "/home/rack/monkeys/monkey-2517a72ed6.py", line 1437, in main
    external_script_action(opt, args)
  File "/home/rack/monkeys/monkey-2517a72ed6.py", line 1402, in external_script_action
    p.communicate(script)
  File "/usr/lib64/python2.7/subprocess.py", line 797, in communicate
    self.wait()
  File "/usr/lib64/python2.7/subprocess.py", line 1376, in wait
    pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
  File "/usr/lib64/python2.7/subprocess.py", line 478, in _eintr_retry_call
    return func(*args)
KeyboardInterrupt

Different Return Codes

Investigate implementing different return codes depending on what is returned by the script

Example:

Run script with the following output:

NO files have OOM issues - return code 1
OOM Found in 1 file - return code 2
Loads of OOM found (> 5) - return code 3

This allows automation tools to implement different "time saved" values depending on the scripts return code

oom in dmesg but not system logs

Dmesg reporting OOM issues but script reporting nothing

This is because dmesg is reporting very old oom incidents.
System has rotated and purged all logs since that report so there is nothing left in the log file.

Grab the oldest date (1st line in oldest compressed file) and print message to explain that its an old message, there are no incidents since $date

--show Select date/number of OOM instances to show

If there are loads of oom instances, you will only be able to show 3
Implement an option to view more or less, and data range

Report # of processes

Add functionality to report on the total number of processes recorded when the system ooms

Command Line Help

Need to add command line help to the script

Prompt for other log files if main file has no oom but other files do have oom

'NoneType' object has no attribute 'endswith'

Here's the output I'm getting on Fedora 25:

Python 2.7.13

~/out-of-memory $ python oom-investigate.py 
----------------------------------------
      _____ _____ _____ 
     |     |     |     |
     |  |  |  |  | | | |
     |_____|_____|_|_|_|
     Out Of Memory Analyser

Disclaimer:
If system OOMs too viciously, there may be nothing logged!
Do NOT take this script as FACT, investigate further
----------------------------------------
Unsupported OS

Error:
'NoneType' object has no attribute 'endswith'

----------------------------------------

File "<stdin>", line 80
    with open("/proc/meminfo", "r") as meminfo:
            ^
SyntaxError: invalid syntax

Note: RHEL5 and python 2.4.x are EOL. I will not go out of my way to accommodate for EOL OS's and python versions.

Date objects

Fix date objects so the sort works correctly rather than the current sorting via alphabetical