Author: Martin 'BruXy' Bruchanov
This is a website monitoring tool using Apache Kafka messaging framework.
producer.py
collects metrics and sends them via Kafka toconsumer.py
which stores the received message to the PostgreSQL database.
To run producer/consumer use (exit with Ctrl-c
):
./producer.py monitor.conf
./consumer.py monitor.conf
A configuration example can be found in: example_monitor.conf.
Overview:
Producer is using src/https_status.py
to check website metrics in defined
period. It collects:
- timestamp,
- GET status/errors,
- website response time and
- optionally tests if the page HTML contains text defined by regex.
If there is a TCP connection error when requesting some URL, the status metric and response time is set to 0. Timeouts for TCP and HTTP can be set in the configuration file.
Each website monitor runs in a separate thread and sends collected metric to Apache Kafka.
Consumer receives messages from Apache Kafka and stores them in a PostgreSQL
database. Each monitoring stream is saved to a separate database table. All
methods for database queries are defined in src/database.py
.
Both scripts are using src/read_config.py
to parse text configuration file.
- Clone repository:
git clone https://github.com/BruXy/kafka-web-monitor.git
. - If necessary for executing a new Python virtual environment, there is
make venv
to setup a local environment inPYTHON_VENV_DIR
as defined in theMakefile
. - Install required packages:
make pip-install
Check Makefile
for additional support tools. To use a PostreSQL database you
need to have postgres
(psql
) client and libpq-devel
installed, use make dev-install
to add these packages on RPM based GNU/Linux distributions.
Code syntax check and unit tests can be executed by invoking make
without any
parameters, recipes for these tests are defined in target "syntax" and "test".