Comments (3)
OOM killed apache
it kills apache but you suspect an issue with fail2ban?
f2b/a.sshd invoked oom-killer
This message neither means it was directly responsible for OOM, nor that fail2ban eating memory (causation is not necessarily a reason for memory leak). It can also just mean - this was a last drop.
- How much memory fail2ban uses normally (few minutes of work after restart)?
- Do you really see the leak (growth of memory by fail2ban)? Is it obvious?
- Do you have configured a large
findtime
(and/ormaxretry
) for some jail? - How your configuration look? (E. g. provide an output of config-dump
fail2ban-client -d
) - What do you see by
fail2ban-client status ?"$jail"?
(how many failed and banned tickets it show)?
If answer for 3) is yes, take a look for similar issue - #2843 (comment)
Shortly: decrease value of maxmatches
in jail.local (and/or dbmaxmatches
in fail2ban.local) for this or even all jails, or if you don't need them at all (e. g. your actions don't use <matches>
tags to show matched log-messages), simply set them to 0 for default (or some jails) section, as described in #2402 .
Although in v.0.11.2 the default maxmatches
has been reduced from 50 to 5, but even with 5 long log-lines per IP and many IPs in fail-list it may consume large amount of memory yet.
You can also try to reduce findtime
and maxretry
, if they are very very large (fail-manager list of IPs for such jails may be extremely large, depending on count of failures in log).
If it is not, we'd need definitely more info (how many jails, how looks the config, etc)...
from fail2ban.
- How much memory fail2ban uses normally (few minutes of work after restart)?
Its ~22M
- Do you really see the leak (growth of memory by fail2ban)? Is it obvious?
No it not obvious, probably memory peak
- Do you have configured a large findtime (and/or maxretry) for some jail?
no, just using jail.conf default with seded ignoreips
- How your configuration look? (E. g. provide an output of config-dump fail2ban-client -d)
['set', 'syslogsocket', 'auto']
['set', 'loglevel', 'INFO']
['set', 'logtarget', '/var/log/fail2ban.log']
['set', 'dbfile', '/var/lib/fail2ban/fail2ban.sqlite3']
['set', 'dbmaxmatches', 10]
['set', 'dbpurgeage', '1d']
['add', 'sshd', 'auto']
['set', 'sshd', 'usedns', 'warn']
['set', 'sshd', 'prefregex', '^<F-MLFID>(?:\\[\\])?\\s*(?:<[^.]+\\.[^.]+>\\s+)?(?:\\S+\\s+)?(?:kernel:\\s?\\[ *\\d+\\.\\d+\\]:?\\s+)?(?:@vserver_\\S+\\s+)?(?:(?:(?:\\[\\d+\\])?:\\s+[\\[\\(]?sshd(?:\\(\\S+\\))?[\\]\\)]?:?|[\\[\\(]?sshd(?:\\(\\S+\\))?[\\]\\)]?:?(?:\\[\\d+\\])?:?)\\s+)?(?:\\[ID \\d+ \\S+\\]\\s+)?</F-MLFID>(?:(?:error|fatal): (?:PAM: )?)?<F-CONTENT>.+</F-CONTENT>$']
['set', 'sshd', 'maxlines', 1]
['multi-set', 'sshd', 'addfailregex', ['^[aA]uthentication (?:failure|error|failed) for <F-USER>.*</F-USER> from <HOST>( via \\S+)?(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^User not known to the underlying authentication module for <F-USER>.*</F-USER> from <HOST>(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^Failed publickey for invalid user <F-USER>(?P<cond_user>\\S+)|(?:(?! from ).)*?</F-USER> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}(?: ssh\\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)', '^Failed (?:<F-NOFAIL>publickey</F-NOFAIL>|\\S+) for (?P<cond_inv>invalid user )?<F-USER>(?P<cond_user>\\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)</F-USER> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}(?: ssh\\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)', '^<F-USER>ROOT</F-USER> LOGIN REFUSED FROM <HOST>', '^[iI](?:llegal|nvalid) user <F-USER>.*?</F-USER> from <HOST>(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^User <F-USER>\\S+|.*?</F-USER> from <HOST> not allowed because not listed in AllowUsers(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^User <F-USER>\\S+|.*?</F-USER> from <HOST> not allowed because listed in DenyUsers(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^User <F-USER>\\S+|.*?</F-USER> from <HOST> not allowed because not in any group(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^refused connect from \\S+ \\(<HOST>\\)', '^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}:\\s*3: .*: Auth fail(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^User <F-USER>\\S+|.*?</F-USER> from <HOST> not allowed because a group is listed in DenyGroups(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', "^User <F-USER>\\S+|.*?</F-USER> from <HOST> not allowed because none of user's groups are listed in AllowGroups(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$", '^<F-NOFAIL>pam_[a-z]+\\(sshd:auth\\):\\s+authentication failure;</F-NOFAIL>(?:\\s+(?:(?:logname|e?uid|tty)=\\S*)){0,4}\\s+ruser=<F-ALT_USER>\\S*</F-ALT_USER>\\s+rhost=<HOST>(?:\\s+user=<F-USER>\\S*</F-USER>)?(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^maximum authentication attempts exceeded for <F-USER>.*</F-USER> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}(?: ssh\\d*)?(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^User <F-USER>\\S+|.*?</F-USER> not allowed because account is locked(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*', '^<F-MLFFORGET>Disconnecting</F-MLFFORGET>(?: from)?(?: (?:invalid|authenticating)) user <F-USER>\\S+</F-USER> <HOST>(?: (?:port \\d+|on \\S+)){0,2}:\\s*Change of username or service not allowed:\\s*.*\\[preauth\\]\\s*$', '^Disconnecting: Too many authentication failures(?: for <F-USER>\\S+|.*?</F-USER>)?(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^<F-NOFAIL>Received <F-MLFFORGET>disconnect</F-MLFFORGET></F-NOFAIL> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}:\\s*11:', '^<F-NOFAIL><F-MLFFORGET>(Connection closed|Disconnected)</F-MLFFORGET></F-NOFAIL> (?:by|from)(?: (?:invalid|authenticating) user <F-USER>\\S+|.*?</F-USER>)? <HOST>(?:(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*|\\s*)$', '^<F-MLFFORGET><F-MLFGAINED>Accepted \\w+</F-MLFGAINED></F-MLFFORGET> for <F-USER>\\S+</F-USER> from <HOST>(?:\\s|$)', '^<F-NOFAIL>Connection from</F-NOFAIL> <HOST>']]
['set', 'sshd', 'datepattern', '{^LN-BEG}']
['set', 'sshd', 'addjournalmatch', '_SYSTEMD_UNIT=sshd.service', '+', '_COMM=sshd']
['set', 'sshd', 'maxretry', 5]
['set', 'sshd', 'maxmatches', 5]
['set', 'sshd', 'findtime', '10m']
['set', 'sshd', 'bantime', '10m']
['set', 'sshd', 'ignorecommand', '']
['set', 'sshd', 'addignoreip', '127.0.0.1/8', 'XXX.YYY.ZZZ.10/32', 'XXX.YYY.ZZZ.141/32', 'XXX.YYY.ZZZ.213/32', 'XXX.YYY.ZZZ.71/32', 'XXX.YYY.ZZZ.253/32', 'XXX.YYY.ZZZ.232/32', 'XXX.YYY.ZZZ.231/32', 'XXX.YYY.ZZZ.147/32', 'XXX.YYY.ZZZ.219/32', 'XXX.YYY.ZZZ.108/32', 'XXX.YYY.ZZZ.227/32', 'XXX.YYY.ZZZ.146/32', 'XXX.YYY.ZZZ.232/32', 'XXX.YYY.ZZZ.235/32', 'XXX.YYY.ZZZ.237/32', 'XXX.YYY.ZZZ.254/32', 'XXX.YYY.ZZZ.0/24', 'XXX.YYY.ZZZ.0/24', 'XXX.YYY.ZZZ.0/26', 'XXX.YYY.ZZZ.0/24']
['set', 'sshd', 'logencoding', 'auto']
['set', 'sshd', 'addlogpath', '/var/log/auth.log', 'head']
['set', 'sshd', 'addaction', 'iptables-multiport']
['multi-set', 'sshd', 'action', 'iptables-multiport', [['actionstart', '<iptables> -N f2b-sshd\n<iptables> -A f2b-sshd -j RETURN\n<iptables> -I INPUT -p tcp -m multiport --dports ssh -j f2b-sshd'], ['actionstop', '<iptables> -D INPUT -p tcp -m multiport --dports ssh -j f2b-sshd\n<iptables> -F f2b-sshd\n<iptables> -X f2b-sshd'], ['actionflush', '<iptables> -F f2b-sshd'], ['actioncheck', "<iptables> -n -L INPUT | grep -q 'f2b-sshd[ \\t]'"], ['actionban', '<iptables> -I f2b-sshd 1 -s <ip> -j <blocktype>'], ['actionunban', '<iptables> -D f2b-sshd -s <ip> -j <blocktype>'], ['port', 'ssh'], ['protocol', 'tcp'], ['chain', '<known/chain>'], ['name', 'sshd'], ['actname', 'iptables-multiport'], ['blocktype', 'REJECT --reject-with icmp-port-unreachable'], ['returntype', 'RETURN'], ['lockingopt', '-w'], ['iptables', 'iptables <lockingopt>'], ['blocktype?family=inet6', 'REJECT --reject-with icmp6-port-unreachable'], ['iptables?family=inet6', 'ip6tables <lockingopt>']]]
['start', 'sshd']
- What do you see by fail2ban-client status ?"$jail"? (how many failed and banned tickets it show)?
Status for the jail: sshd
|- Filter
| |- Currently failed: 7
| |- Total failed: 5052
| `- File list: /var/log/auth.log
`- Actions
|- Currently banned: 6
|- Total banned: 868
`- Banned IP list: 162.248.100.144 218.92.0.116 43.156.35.214 103.101.160.198 150.109.198.246 203.130.255.2
If you dont see anything at first sight just close the ticket. Ill give another chance to server. If this occurs again i reopen ticket.
from fail2ban.
- No it not obvious, probably memory peak
[picture]
Well, it doesn't show to me any reason to blame fail2ban here (it is complete memory usage, not of the fail2ban process), however this picture doesn't illustrate the OOM-kill reason at all (at least it is not visible).
If this occurs again i reopen ticket.
But next time please provide the whole oom-killer's log (excerpt from kernel-log or from journal or dmesg), up to something like this:
processname invoked oom-killer: ...
...
[ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[ 678] 0 678 11111 7 25 417 -1000 systemd-udevd
[ 679] 0 679 66666 71 32 82 0 whatever
...
[ 1234] 1005 1234 7482510 2656268 8492 1473287 0 apache
...
Out of memory: Kill process 1234 (apache) score 765 or sacrifice child
Killed process 1234 (apache) total-vm:29930040kB, anon-rss:10625048kB, file-rss:0kB, shmem-rss:24kB
As one can see above, it would also output the result of top
for all running processes, so one is able to see what fail2ban currently uses or which process(es) eating the most of memory at the moment of disaster.
Anyway since it was apache, which has been killed, I guess it was the evildoer that used most of memory at that point.
from fail2ban.
Related Issues (20)
- [RFE]: Change cloudflare.conf to use WAF Custom Rules rather than Firewall Access Rules due to deprecation
- [BR]: basic setup fail HOT 2
- Not working filter apache logs HOT 1
- [FR]: qbittorrent-nox HOT 8
- [BR]: README.md typos
- Fail2ban - Raspberry Pi5 64bit Bookworm - not working as expected, not reading systemd logs? HOT 6
- [FR]: Ubuntu 22.04.4 LTS fail2ban Unable to match some authentication failure logs HOT 4
- [BR]: Test testStatusStats fails with 1.1.0 on Fedora Rawhide HOT 10
- Help Needed: Creating Fail2ban Filter for Exchange Autodiscover Failed Login Attempts HOT 1
- [BR]: installing fail2ban on ubuntu 24.04 with apt-get showing errors and is not starting HOT 2
- [BR]: ERROR No module named 'asynchat' on Ubuntu 24.04 HOT 7
- [FR]: Wordpress Fail2ban filter not processing authentication failures on Debian bookworm HOT 7
- [BR]:The jail set in the configuration file is not loaded HOT 6
- >Jail not being loaded issue HOT 2
- Failed to execute ban HOT 8
- [RFE]: Extend ignoreip to cater for dynamic IP scenarios by resolving FDNs assigned HOT 1
- [BR]: Can't start fail2ban service on Ubuntu 24.04 HOT 1
- [BR]: You can use root commands HOT 2
- [BR]: Python 3.12 shows DeprecationWarning about use of os.fork() HOT 1
- [FR]: SNMPv3 filters for Unknown User, Bad AUTH password and PRIV password, multiline, mutiple filters HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fail2ban.