Giter Club home page Giter Club logo

fail2ban-geo-exporter's Introduction

fail2ban-geo-exporter

It's all in the description already. This little program will expose Prometheus metrics for every IP banned by fail2ban. You can have them geotagged or disable tagging if you just want the IP and the jail it's coming from.

NOTE: This software assumes your fail2ban jail definitions are all in a single file. Multiple definites under jail.d/ are currently not supported.

Disclaimer

This exporter goes deliberately against best practices and ist not suitable for deployments at scale. It's intended to be used in a homelab alike setting and won't even provide any sane metric to alert on. This may change in the future, but it more than likely will not.

By enabling grouping in the conf.yml, the growth of label cardinality can be reduced, but this is still far from ideal.

Metrics

This exporter provides a single time series named fail2ban_banned_ip for each IP in a fail2ban jail. Default labels are: jail and ip with their respective values.

More labels can be provided by enabling geoIP annotation. At this point, the maxmind provider adds city, latitude, and longitude in addition the default labels.

If you enable grouping in the conf.yml, you will instead receive two sets of metrics and no more data about single IPs. Instead you get a gauge fail2ban_location which counts the number of banned IPs per location. The labels are the same as above, just without jail and ip.

The second metric is fail2ban_jailed_ips which is a gauge, that displays all currently banned IPs per jail. jail is the only label in this metric.

It's highly recommended to enable grouping, in order to reduce the cardinality of your labels.

A small guide to creating your own geoIP provider can be found in the Extensibility section of this README.

Configuration

server:
    listen_address:
    port:
geo:
    enabled: True
    provider: 'MaxmindDB'
    enable_grouping: False
    maxmind:
        db_path: '/f2b-exporter/db/GeoLite2-City.mmdb'
f2b:
    conf_path: '/etc/fail2ban'
    db: '/var/lib/fail2ban/fail2ban.sqlite3'

Just plug in the port and IPv4 address you want your exporter to be listening on. If you want to enable geotagging, there is only one method at this time and for that you will need to sign up for a free account at https://www.maxmind.com, download their city database and plug the path to the db in geo.maxmind.db_path. Their paid tier claims to have increased accuracy and is interchangable with their free database, so that should work as a data source for this exporter as well. At the time of writing I can neither deny, nor confirm these claims.

f2b.conf_path assumes default directory structure of fail2ban. So your jails can be defined in /etc/fail2ban/jail.local or in /etc/fail2ban/jail.d/*.local. Default values defined in jail.local (i.e.: bantime) will be picked up and consequently applied to all jails defined under jail.d.

Missing entries in the MaxmindDB will be discarded by default. If you want to keep track of missing entries, you can provide default values to be used instead. In your conf.yml add a section under geo.maxmind like so:

geo:
    maxmind:
        on_error:
            city: 'Atlantis'
            latitude: '0'
            longitude: '0'

Installation

As a systemd service

Pick your favourite working directory and git clone https://github.com/vdcloudcraft/fail2ban-geo-exporter.git .

Now run these commands to set up a python virtual environment:

python -m venv .
. bin/activate
pip install -r requirements.txt

Create a file called fail2ban-geo-exporter.service at /etc/systemd/system/

Open that file and paste following content in there:

[Unit]
Description=fail2ban geo exporter
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=<path>
Environment=PYTHONPATH=<path>/bin
ExecStart=<path>/bin/python3 <path>/fail2ban-exporter.py
Restart=always

[Install]
WantedBy=multi-user.target

Make sure to replace all four(4) instances of <path> in that config with your actual working directory.

Make sure you have a conf.yml in your working directory as described in Configuration

When that is all done, run following commands and your exporter is running and will survive reboots:

sudo systemctl daemon-reload
sudo systemctl enable fail2ban-geo-exporter.service
sudo service fail2ban-geo-exporter start

You should see a list of metrics when you run curl http://localhost:[port]/metrics

As a Docker container

Make sure you have prepared a config as described in Configuration.

Docker images are provided via Docker Hub

To run the exporter in a Docker container, execute the following command

docker run -d \
        -v /etc/fail2ban:/etc/fail2ban:ro \
        -v /var/lib/fail2ban/fail2ban.sqlite3:/var/lib/fail2ban/fail2ban.sqlite3:ro \
        -v /<path-to-your-db>/GeoLite2-City.mmdb:/f2b-exporter/db/GeoLite2-City.mmdb:ro \
        -v /<path-to-your-conf.yml>/conf.yml:/f2b-exporter/conf.yml \
        --name fail2ban-geo-exporter \
        --restart unless-stopped \
        vdcloudcraft/fail2ban-geo-exporter:latest

Make sure that your paths to jail.local and fail2ban.sqlite3 are correct.

Extensibility

Currently there is only one way to geotag IPs and that is with the Maxmind db. But there is a way to provide custom geoIP providers.

If you wish to implement your own method for annotating your IPs, you need to create a Python class and save the module in ./geoip_provider/

You need to ensure following requirements are fullfilled:

  • Your module name is a lower case version of your class name. E.g.: Class FancyProvider, module ./geoip_providers/fancyprovider.py
  • Your class has a constructor that accepts a single parameter. This will be the parsed conf.yml that will be passed to the class.
  • Your class implements a method annotate(ip), that takes in a single IP as a string and returns a dictionary with all additional labels for Prometheus to use. Do not include the IP itself as a label.
  • Your class implements a method get_labels() that returns a list of strings with the label names it's going to provide.

When all that is given, you can just put your class name with[!] capitalization into the configuration under geo.provider and enjoy the fruits of your labour.

Be aware that the enable_grouping setting will use only the labels provided by your class to aggregate locations.

If you do create another provider class and think other people might find it useful too, I'll gladly review pull requests.

Grafana dashboard

The files dashboard-*.json include a complete definition for either grouping configuration to display your banned IPs on a worldmap and count all banned IPs per jail.

fail2ban-geo-exporter's People

Contributors

vdcloudcraft avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

fail2ban-geo-exporter's Issues

Rename conf.yml in repo

Suggestion: Rename conf.yaml to conf.examle.yml in this repo.

At the moment, if updating the repository via "git pull" it gives an error that the local changes to the conf.yml would be overwritten.
Has two benefits:

  • update via git pull
  • always newest conf.example.yml and no need to search for new values

Also requires an update of the readme

Bantime not read correctly

Hello,
I just started to use this project. First of all, thanks for all the work put into this!

I realized that my bantime configs are not parsed correctly from the jail files, either resulting in a crash of the program or resulting in no bans exported to prometheus.

Case 1: strings

fail2ban allows to set the bantime as string, e.g. 1d for 1 day. If I read the code correctly, currently this exporter requires the bantime to be set in seconds, as it is converted to int:

jail.bantime = int(bantime)

fail2ban uses the method str2seconds in mytime.py for conversion. I think it would be best to use the very same method here to avoid different behaviours. So fail2ban needs to be included in the requirements.txt, right?
A simple workaround is to specify all bantimes in seconds in the jail files.

Case 2: permanent bans

I use fail2ban to ban some IPs forever, therefore I set the bantime to negative values. At least this causes no crash, since the value can be converted to integer. But those bans are not included in the metrics, since the selection from the database makes sense only for positive bantimes. If the bantime is negative, one could omit the WHERE clause completely in the database query.

I'll will try to find some time to fiddle around with this and eventually create a PR.
Feedback on these ideas is very much welcome!

Crashes on "AddressNotFoundError

The exporter crashes when it gets an ip address which is not in the database:

Jun 02 19:38:21 vps07 python3[34060]: 127.0.0.1 - - [02/Jun/2020 19:38:06] "GET /metrics HTTP/1.1" 500 59
Jun 02 19:38:21 vps07 python3[34060]: Traceback (most recent call last):
Jun 02 19:38:21 vps07 python3[34060]: File "/usr/lib/python3.5/wsgiref/handlers.py", line 137, in run
Jun 02 19:38:21 vps07 python3[34060]: self.result = application(self.environ, self.start_response)
Jun 02 19:38:21 vps07 python3[34060]: File "/opt/fail2ban-geo-exporter/lib/python3.5/site-packages/prometheus_client/exposition.py", line 45, in prometheus_app
Jun 02 19:38:21 vps07 python3[34060]: output = encoder(r)
Jun 02 19:38:21 vps07 python3[34060]: File "/opt/fail2ban-geo-exporter/lib/python3.5/site-packages/prometheus_client/openmetrics/exposition.py", line 14, in generate_latest
Jun 02 19:38:21 vps07 python3[34060]: for metric in registry.collect():
Jun 02 19:38:21 vps07 python3[34060]: File "/opt/fail2ban-geo-exporter/lib/python3.5/site-packages/prometheus_client/registry.py", line 75, in collect
Jun 02 19:38:21 vps07 python3[34060]: for metric in collector.collect():
Jun 02 19:38:21 vps07 python3[34060]: File "/opt/fail2ban-geo-exporter/fail2ban-exporter.py", line 65, in collect
Jun 02 19:38:21 vps07 python3[34060]: self.assign_location()
Jun 02 19:38:21 vps07 python3[34060]: File "/opt/fail2ban-geo-exporter/fail2ban-exporter.py", line 61, in assign_location
Jun 02 19:38:21 vps07 python3[34060]: entry.update(self.geo_provider.annotate(entry['ip']))
Jun 02 19:38:21 vps07 python3[34060]: File "/opt/fail2ban-geo-exporter/geoip_provider/maxminddb.py", line 10, in annotate
Jun 02 19:38:21 vps07 python3[34060]: lookup = reader.city(ip)
Jun 02 19:38:21 vps07 python3[34060]: File "/opt/fail2ban-geo-exporter/lib/python3.5/site-packages/geoip2/database.py", line 114, in city
Jun 02 19:38:21 vps07 python3[34060]: return self._model_for(geoip2.models.City, 'City', ip_address)
Jun 02 19:38:21 vps07 python3[34060]: File "/opt/fail2ban-geo-exporter/lib/python3.5/site-packages/geoip2/database.py", line 194, in _model_for
Jun 02 19:38:21 vps07 python3[34060]: (record, prefix_len) = self._get(types, ip_address)
Jun 02 19:38:21 vps07 python3[34060]: File "/opt/fail2ban-geo-exporter/lib/python3.5/site-packages/geoip2/database.py", line 190, in _get
Jun 02 19:38:21 vps07 python3[34060]: "The address %s is not in the database." % ip_address)
Jun 02 19:38:21 vps07 python3[34060]: geoip2.errors.AddressNotFoundError: The address 10.255.30.162 is not in the database.

What would limit lines from exporter

Im only getting about 300 lines from about 8000 banned ips, including header info, none of the standard footer info. it just is the last rule it reads.

is this a prometheus setup problem?

my iptables_exports gives me 21445 lines... I use aggressice banning since im a mining pool. We've been under massive bot attack for last week. I'd like to get the full list if possible and maybe set "scrape_interval" to 10 minutes since its so hard on cpu.

Configurable IP Address

By default the exporter is listening on the ip 0.0.0.0. That's every IP from the host.
Make it configurable so it can be e.g. 127.0.0.1.

Or better, make it IPv6 compatible so it can be [..1] ;)

Logic errors

There are 2 errors essentially in the code:

  1. the time conversion only considers seconds, but as soon as other values than seconds are given, the usage is incorrect.

I recommend to use the original function from fail2ban itself (MyTime.str2seconds): https://github.com/fail2ban/fail2ban/blob/095aeda8407b433098df35424cde2764a09566a6/fail2ban/server/mytime.py

  1. within the loop the same-initialized class "jail" is called, which however does not correspond to the current iteration value of the run variable. It returns always the same values of the last entry from the jail class.

Just rename the name of the iteration values from to something else, then it should work.

Fail2Ban exporter does not export data

When I install the Fail2Ban exporter using docker I can't seem to get data out of it.
Here is my configuration:
docker-compose.yml (only relevant parts):

prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    command:
      - --config.file=/etc/prometheus/prometheus.yml
    volumes:
      - prometheus-storage:/prometheus
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    depends_on:
       - fail2ban-exporter
fail2ban-exporter:
    image: vdcloudcraft/fail2ban-geo-exporter:latest
    container_name: fail2ban-exporter
    restart: unless-stopped
    expose:
        - "9555"
    volumes:
        - /etc/fail2ban/jail.local:/etc/fail2ban/jail.local:ro
        - /var/lib/fail2ban/fail2ban.sqlite3:/var/lib/fail2ban/fail2ban.sqlite3:ro
        - ./fail2ban-geo-exporter/GeoLite2-City.mmdb:/f2b-exporter/db/GeoLite2-City.mmdb
        - ./fail2ban-geo-exporter/conf.yml:/f2b-exporter/conf.yml

prometheus.yml:

scrape_configs:
  - job_name: fail2ban-exporter
    scrape_interval: 60s
    static_configs:
    - targets:
      - fail2ban-exporter:9555

fail2ban-geo-exporter conf.yml:

server:
    listen_address: 127.0.0.1
    port: 9555
geo:
    enabled: True
    provider: 'MaxmindDB'
    enable_grouping: True
    maxmind:
        db_path: '/f2b-exporter/db/GeoLite2-City.mmdb'
f2b:
    conf_path: '/etc/fail2ban'
    db: '/var/lib/fail2ban/fail2ban.sqlite3'
    conf: ''

Output when running sudo fail2ban-client status sshd:

Status for the jail: sshd
|- Filter
|  |- Currently failed: 5
|  |- Total failed:     454
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 23
   |- Total banned:     89
   `- Banned IP list:   <censored>

Output when using the provided grafana dashboard:
0 IPs banned, no marks on the map widget

KeyError: 'sshd'

Hello,

I get the following error when I try to start the exporter:

Traceback (most recent call last):
  File "/f2b-exporter/fail2ban-exporter.py", line 127, in <module>
    REGISTRY.register(F2bCollector(conf))
  File "/usr/lib/python3.7/site-packages/prometheus_client/registry.py", line 24, in register
    names = self._get_names(collector)
  File "/usr/lib/python3.7/site-packages/prometheus_client/registry.py", line 64, in _get_names
    for metric in desc_func():
  File "/f2b-exporter/fail2ban-exporter.py", line 76, in collect
    self.get_jailed_ips()
  File "/f2b-exporter/fail2ban-exporter.py", line 59, in get_jailed_ips
    bantime = config[j[0]]['bantime'].split(';')[0].strip()
  File "/usr/lib/python3.7/configparser.py", line 958, in __getitem__
    raise KeyError(key)
KeyError: 'sshd'

fail2ban_banned_ip metric is always empty

Hi,

We configured this fail2ban exporter to export metrics. We followed instructions to install it as a system service.Everything seems to work properly with the following configuration

server:
  listen_address: 0.0.0.0
  port: 9412
geo:
    enabled: True
    provider: 'MaxmindDB'
    enable_grouping: False
    maxmind:
        db_path: "/opt/vpn-vol/fail2ban_exporter/geoip/data/GeoLite2-City.mmdb"
f2b:
    conf_path: '/etc/fail2ban'
    db: '/opt/vpn-vol/fail2ban/fail2ban.sqlite3'

Note the custom location of the database, both the exporter and fail2ban are pointing to this file.

Config file is located under /opt/vpn-vol/fail2ban_exporter/env/conf.yml

Service configuration is as follows

[Unit]
Description=fail2ban geo exporter
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/vpn-vol/fail2ban_exporter/env
Environment=PYTHONPATH=/opt/vpn-vol/fail2ban_exporter/env/bin
ExecStart=/opt/vpn-vol/fail2ban_exporter/env/bin/python3 /opt/vpn-vol/fail2ban_exporter/env/fail2ban-exporter.py
Restart=always

[Install]
WantedBy=multi-user.target

The problem we are experiencing is that the metric is never printed

...
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024.0
# HELP fail2ban_banned_ip IP banned by fail2ban
# TYPE fail2ban_banned_ip gauge

Even when whe have some IP banned

Status for the jail: sshd
|- Filter
|  |- Currently failed:	0
|  |- Total failed:	20
|  `- File list:	/var/log/auth.log
`- Actions
   |- Currently banned:	1
   |- Total banned:	1
   `- Banned IP list:	172.22.5.71

What are we doing wrong?

KeyError: 'city'

Hello,
What is this error?
The exporter was working fine and suddenly he made this mistake, not possible to find out why

` Traceback (most recent call last):

File "./fail2ban-exporter.py", line 127, in <module>

REGISTRY.register(F2bCollector(conf))

File "/usr/local/lib/python3.8/site-packages/prometheus_client/registry.py", line 24, in register

names = self._get_names(collector)

File "/usr/local/lib/python3.8/site-packages/prometheus_client/registry.py", line 64, in _get_names

for metric in desc_func():

File "./fail2ban-exporter.py", line 80, in collect

yield self.expose_grouped()

File "./fail2ban-exporter.py", line 107, in expose_grouped

location_key = tuple([ entry[x] for x in self.extra_labels ])

File "./fail2ban-exporter.py", line 107, in <listcomp>

location_key = tuple([ entry[x] for x in self.extra_labels ])

KeyError: 'city'

`

Thanks for your help

Dimo

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.