WMS monitors the state of websites and logs it into a database. It is a homework for Aiven Backend developer position and an excercise of asyncio. The system consists of two parts:
- Producer - monitors websites listed in
websites.json
asyncronosly and sends the status packages to Kafka topicwebsite_check
- Consumer - asyncronosly consumes packages and sends them to
results_log
table ofcheck_results
database WMS relies on Aiven Kafka and Aiven PostrgeSQL. The following instruction will describe how to set everything up.
To get things running you'll need:
- A running Aiven Kafka with
website_check
topic. Three partitions are recomended to feel the power of async. That could easily be done from the web interface. - A running PostgreSQL database with the name
results_log
. We'll use a default avnadmin username. First to set up database run
CREATE DATABASE check_results
WITH
OWNER = avnadmin
ENCODING = 'UTF8'
LC_COLLATE = 'en_US.UTF-8'
LC_CTYPE = 'en_US.UTF-8'
TABLESPACE = pg_default
CONNECTION LIMIT = -1;
And to create a table
CREATE TABLE IF NOT EXISTS public.results_log
(
"timestamp" numeric NOT NULL,
url text COLLATE pg_catalog."default" NOT NULL,
result text COLLATE pg_catalog."default" NOT NULL,
response_time real,
regex_result boolean
)
TABLESPACE pg_default;
ALTER TABLE IF EXISTS public.results_log
OWNER to avnadmin;
Producer and Consumer run in separate environments and are built independently. This could be done on different environments. They use the same Kafka access keys, that are stored in resources
folder as well as lists of websites to check and PostgreSQL username and passwords.
- To build Producer and Consumer you'll need
Python 3.10
(there are some fancy type hinting features one cannot refuse to employ). Once you have cloned current repository to your local machine please installPipenv
pip install --user pipenv
- In
producer
andconsumer
folders run
pipenv install
- Download yur Kafka access keys (also - web interface) and put them into the directory
\resources\certificates
. Or you could use a directory of your choice and specify it in\resources\kafka_certificates.json
.
service.key
- Access Keyservice.cert
- Access Certificateca.pem
- CA Certificate
And enter your Kafaka service uri in \resources\kafka_certificates.json
- Enter your PostgreSQL credentials to
\resources\db_access_data.json
user
- PostgreSQL User name, by default avnadmin (if the SQL frome above is not modified - that's username you need)password
- PostgreSQL passwordhost
- your Aiven PostgreSQL address (will look like ******.aivencloud.com",port
- your Aiven PostgreSQL port. The default is27160
but it is advised to check.
- Look through website lists
Websites to be checked are stored in
\resources\websites.json
There is an example of a very diverse group of blocked sites that are monitored by Citizen Lab - https://github.com/citizenlab/test-lists
You could set up your own group using syntax:
[<url>,<regex>]
where url
is an adress and regex
is Python re regular expression rule to check on the site body. Currently it works only on the first 500 kb of the body. Or [<url>]
if you don't need regex check
- Run Consumer by running
pipenv run consumer.py
- Run Producer by running
pipenv run producer.py