Giter Club home page Giter Club logo

Comments (3)

sebres avatar sebres commented on July 20, 2024

OOM killed apache

it kills apache but you suspect an issue with fail2ban?

f2b/a.sshd invoked oom-killer

This message neither means it was directly responsible for OOM, nor that fail2ban eating memory (causation is not necessarily a reason for memory leak). It can also just mean - this was a last drop.

  1. How much memory fail2ban uses normally (few minutes of work after restart)?
  2. Do you really see the leak (growth of memory by fail2ban)? Is it obvious?
  3. Do you have configured a large findtime (and/or maxretry) for some jail?
  4. How your configuration look? (E. g. provide an output of config-dump fail2ban-client -d)
  5. What do you see by fail2ban-client status ?"$jail"? (how many failed and banned tickets it show)?

If answer for 3) is yes, take a look for similar issue - #2843 (comment)
Shortly: decrease value of maxmatches in jail.local (and/or dbmaxmatches in fail2ban.local) for this or even all jails, or if you don't need them at all (e. g. your actions don't use <matches> tags to show matched log-messages), simply set them to 0 for default (or some jails) section, as described in #2402 .
Although in v.0.11.2 the default maxmatches has been reduced from 50 to 5, but even with 5 long log-lines per IP and many IPs in fail-list it may consume large amount of memory yet.

You can also try to reduce findtime and maxretry, if they are very very large (fail-manager list of IPs for such jails may be extremely large, depending on count of failures in log).

If it is not, we'd need definitely more info (how many jails, how looks the config, etc)...

from fail2ban.

pschonmann avatar pschonmann commented on July 20, 2024
  1. How much memory fail2ban uses normally (few minutes of work after restart)?
Its ~22M
  1. Do you really see the leak (growth of memory by fail2ban)? Is it obvious?
No it not obvious, probably memory peak

obrazek

  1. Do you have configured a large findtime (and/or maxretry) for some jail?
no, just using jail.conf default with seded ignoreips
  1. How your configuration look? (E. g. provide an output of config-dump fail2ban-client -d)
['set', 'syslogsocket', 'auto']
['set', 'loglevel', 'INFO']
['set', 'logtarget', '/var/log/fail2ban.log']
['set', 'dbfile', '/var/lib/fail2ban/fail2ban.sqlite3']
['set', 'dbmaxmatches', 10]
['set', 'dbpurgeage', '1d']
['add', 'sshd', 'auto']
['set', 'sshd', 'usedns', 'warn']
['set', 'sshd', 'prefregex', '^<F-MLFID>(?:\\[\\])?\\s*(?:<[^.]+\\.[^.]+>\\s+)?(?:\\S+\\s+)?(?:kernel:\\s?\\[ *\\d+\\.\\d+\\]:?\\s+)?(?:@vserver_\\S+\\s+)?(?:(?:(?:\\[\\d+\\])?:\\s+[\\[\\(]?sshd(?:\\(\\S+\\))?[\\]\\)]?:?|[\\[\\(]?sshd(?:\\(\\S+\\))?[\\]\\)]?:?(?:\\[\\d+\\])?:?)\\s+)?(?:\\[ID \\d+ \\S+\\]\\s+)?</F-MLFID>(?:(?:error|fatal): (?:PAM: )?)?<F-CONTENT>.+</F-CONTENT>$']
['set', 'sshd', 'maxlines', 1]
['multi-set', 'sshd', 'addfailregex', ['^[aA]uthentication (?:failure|error|failed) for <F-USER>.*</F-USER> from <HOST>( via \\S+)?(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^User not known to the underlying authentication module for <F-USER>.*</F-USER> from <HOST>(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^Failed publickey for invalid user <F-USER>(?P<cond_user>\\S+)|(?:(?! from ).)*?</F-USER> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}(?: ssh\\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)', '^Failed (?:<F-NOFAIL>publickey</F-NOFAIL>|\\S+) for (?P<cond_inv>invalid user )?<F-USER>(?P<cond_user>\\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)</F-USER> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}(?: ssh\\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)', '^<F-USER>ROOT</F-USER> LOGIN REFUSED FROM <HOST>', '^[iI](?:llegal|nvalid) user <F-USER>.*?</F-USER> from <HOST>(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^User <F-USER>\\S+|.*?</F-USER> from <HOST> not allowed because not listed in AllowUsers(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^User <F-USER>\\S+|.*?</F-USER> from <HOST> not allowed because listed in DenyUsers(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^User <F-USER>\\S+|.*?</F-USER> from <HOST> not allowed because not in any group(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^refused connect from \\S+ \\(<HOST>\\)', '^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}:\\s*3: .*: Auth fail(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^User <F-USER>\\S+|.*?</F-USER> from <HOST> not allowed because a group is listed in DenyGroups(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', "^User <F-USER>\\S+|.*?</F-USER> from <HOST> not allowed because none of user's groups are listed in AllowGroups(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$", '^<F-NOFAIL>pam_[a-z]+\\(sshd:auth\\):\\s+authentication failure;</F-NOFAIL>(?:\\s+(?:(?:logname|e?uid|tty)=\\S*)){0,4}\\s+ruser=<F-ALT_USER>\\S*</F-ALT_USER>\\s+rhost=<HOST>(?:\\s+user=<F-USER>\\S*</F-USER>)?(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^maximum authentication attempts exceeded for <F-USER>.*</F-USER> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}(?: ssh\\d*)?(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^User <F-USER>\\S+|.*?</F-USER> not allowed because account is locked(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*', '^<F-MLFFORGET>Disconnecting</F-MLFFORGET>(?: from)?(?: (?:invalid|authenticating)) user <F-USER>\\S+</F-USER> <HOST>(?: (?:port \\d+|on \\S+)){0,2}:\\s*Change of username or service not allowed:\\s*.*\\[preauth\\]\\s*$', '^Disconnecting: Too many authentication failures(?: for <F-USER>\\S+|.*?</F-USER>)?(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$', '^<F-NOFAIL>Received <F-MLFFORGET>disconnect</F-MLFFORGET></F-NOFAIL> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}:\\s*11:', '^<F-NOFAIL><F-MLFFORGET>(Connection closed|Disconnected)</F-MLFFORGET></F-NOFAIL> (?:by|from)(?: (?:invalid|authenticating) user <F-USER>\\S+|.*?</F-USER>)? <HOST>(?:(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*|\\s*)$', '^<F-MLFFORGET><F-MLFGAINED>Accepted \\w+</F-MLFGAINED></F-MLFFORGET> for <F-USER>\\S+</F-USER> from <HOST>(?:\\s|$)', '^<F-NOFAIL>Connection from</F-NOFAIL> <HOST>']]
['set', 'sshd', 'datepattern', '{^LN-BEG}']
['set', 'sshd', 'addjournalmatch', '_SYSTEMD_UNIT=sshd.service', '+', '_COMM=sshd']
['set', 'sshd', 'maxretry', 5]
['set', 'sshd', 'maxmatches', 5]
['set', 'sshd', 'findtime', '10m']
['set', 'sshd', 'bantime', '10m']
['set', 'sshd', 'ignorecommand', '']
['set', 'sshd', 'addignoreip', '127.0.0.1/8', 'XXX.YYY.ZZZ.10/32', 'XXX.YYY.ZZZ.141/32', 'XXX.YYY.ZZZ.213/32', 'XXX.YYY.ZZZ.71/32', 'XXX.YYY.ZZZ.253/32', 'XXX.YYY.ZZZ.232/32', 'XXX.YYY.ZZZ.231/32', 'XXX.YYY.ZZZ.147/32', 'XXX.YYY.ZZZ.219/32', 'XXX.YYY.ZZZ.108/32', 'XXX.YYY.ZZZ.227/32', 'XXX.YYY.ZZZ.146/32', 'XXX.YYY.ZZZ.232/32', 'XXX.YYY.ZZZ.235/32', 'XXX.YYY.ZZZ.237/32', 'XXX.YYY.ZZZ.254/32', 'XXX.YYY.ZZZ.0/24', 'XXX.YYY.ZZZ.0/24', 'XXX.YYY.ZZZ.0/26', 'XXX.YYY.ZZZ.0/24']
['set', 'sshd', 'logencoding', 'auto']
['set', 'sshd', 'addlogpath', '/var/log/auth.log', 'head']
['set', 'sshd', 'addaction', 'iptables-multiport']
['multi-set', 'sshd', 'action', 'iptables-multiport', [['actionstart', '<iptables> -N f2b-sshd\n<iptables> -A f2b-sshd -j RETURN\n<iptables> -I INPUT -p tcp -m multiport --dports ssh -j f2b-sshd'], ['actionstop', '<iptables> -D INPUT -p tcp -m multiport --dports ssh -j f2b-sshd\n<iptables> -F f2b-sshd\n<iptables> -X f2b-sshd'], ['actionflush', '<iptables> -F f2b-sshd'], ['actioncheck', "<iptables> -n -L INPUT | grep -q 'f2b-sshd[ \\t]'"], ['actionban', '<iptables> -I f2b-sshd 1 -s <ip> -j <blocktype>'], ['actionunban', '<iptables> -D f2b-sshd -s <ip> -j <blocktype>'], ['port', 'ssh'], ['protocol', 'tcp'], ['chain', '<known/chain>'], ['name', 'sshd'], ['actname', 'iptables-multiport'], ['blocktype', 'REJECT --reject-with icmp-port-unreachable'], ['returntype', 'RETURN'], ['lockingopt', '-w'], ['iptables', 'iptables <lockingopt>'], ['blocktype?family=inet6', 'REJECT --reject-with icmp6-port-unreachable'], ['iptables?family=inet6', 'ip6tables <lockingopt>']]]
['start', 'sshd']
  1. What do you see by fail2ban-client status ?"$jail"? (how many failed and banned tickets it show)?
Status for the jail: sshd
|- Filter
|  |- Currently failed:	7
|  |- Total failed:	5052
|  `- File list:	/var/log/auth.log
`- Actions
   |- Currently banned:	6
   |- Total banned:	868
   `- Banned IP list:	162.248.100.144 218.92.0.116 43.156.35.214 103.101.160.198 150.109.198.246 203.130.255.2

If you dont see anything at first sight just close the ticket. Ill give another chance to server. If this occurs again i reopen ticket.

from fail2ban.

sebres avatar sebres commented on July 20, 2024
  1. No it not obvious, probably memory peak
    [picture]

Well, it doesn't show to me any reason to blame fail2ban here (it is complete memory usage, not of the fail2ban process), however this picture doesn't illustrate the OOM-kill reason at all (at least it is not visible).

If this occurs again i reopen ticket.

But next time please provide the whole oom-killer's log (excerpt from kernel-log or from journal or dmesg), up to something like this:

processname invoked oom-killer: ...
...
[ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[  678]     0   678    11111        7      25      417         -1000 systemd-udevd
[  679]     0   679    66666       71      32       82             0 whatever
...
[ 1234]  1005  1234  7482510  2656268    8492  1473287             0 apache
...
Out of memory: Kill process 1234 (apache) score 765 or sacrifice child
Killed process 1234 (apache) total-vm:29930040kB, anon-rss:10625048kB, file-rss:0kB, shmem-rss:24kB

As one can see above, it would also output the result of top for all running processes, so one is able to see what fail2ban currently uses or which process(es) eating the most of memory at the moment of disaster.

Anyway since it was apache, which has been killed, I guess it was the evildoer that used most of memory at that point.

from fail2ban.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.