This is the tix-time-processor
microservice. It's main task is to recurrently process the data packets of every
installation for each user, and post the data collected to the tix-api
. It alsa uses the IP to AS service, to localize
the source of the packets.
It uses a Celery app to schedule the processing every 10 minutes. That is why it should be connected to the RabbitMQ service in the server.
Currently in dev. Installation instructions are a wishful thinkng right now
The tix-time-processor
is intended to be used a Docker container. So to use it, you just need to download the latest image
and run it with docker.
It is necessary to create two volumes. One which will contain the reports from the tix-time-condenser
. That is how this two
micro-services communicate with each other.
The other volume is the one which will contain the scheduler file.
The tix-time-processor
uses a handful of environment variables to run. Most of them have defaults which are the bare
minimum to be executed. But some of the most important not.
TIX_REPORTS_BASE_PATH
: The path, either in the container or in the machine where the reports are found. (Default: '/tmp/reports')TIX_RABBITMQ_USER
: RabbitMQ user ()needed by Celery) (Default: 'guest')TIX_RABBITMQ_PASS
: RabbitMQ password (needed by Celery) (Default: 'guest')TIX_RABBITMQ_HOST
: RabbitMQ host (needed by Celery) (Default: 'localhost')TIX_RABBITMQ_PORT
: RabbitMQ port (needed by Celery) (Default: 5672)TIX_API_SSL
: If this environment variable has any value whatsoever (i.e.:True
,False
,-1
,sarasa
) then HTTPS protocol communication with the API will be enabled. If left undefined or empty, it will use HTTP alone. (Default: Empty)TIX_API_HOST
: The API host where is located (Default: 'localhost')TIX_API_PORT
: The API port in the host where is located(Default: 80 for HTTP, 443 for HTTPS)TIX_API_USER
: The API username for thetix-time-processor
that is used to authenticate. If left empty, the POST request wont be effectuated and the result will be left in the failed results' directory for the installation. (Default: Epmty)TIX_API_PASSWORD
: The API password for thetix-time-processor
that is used to authenticate. If left empty, the POST request wont be effectuated and the result will be left in the failed results' directory for the installation. (Default: Empty)
This a Celery scheduled app and has three modes of running.
- As a standalone app
- As a scheduler
- As a worker
When the service is run in standalone mode, it will schedule and execute the tasks on its own. But since this is not the method preferred for production environment given the many complications it would have as a distributed service, the beat and worker mode exist. The scheduler mode will serialize the tasks and queue them in the RabbitMQ Service. While the workers will take each of the serialized tasks and execute them. It is important to note that while many workers instances may exists at the same time, only one scheduler instance must exists, because having more than one may derive in concurrency problems. This is the same reason why in productive environments it is advised not to use the standalone mode.
To run it as a Celery app, outside the Docker container the following commands are available:
As a stand-alone app
$> celery -A processor.tasks worker -B [-s celerybeat_scheduler_file_path] [-l celery_log_level]
As a scheduler app
$> celery -A processor.tasks beat [-s celerybeat_scheduler_file_path] [-l celery_log_level]
As a worker app
$> celery -A processor.tasks worker [-l celery_log_level]
If you want to use it as a Docker Container, you should create a volume or use the volume that the tix-time-condenser
is using to drop the report files, and the volume that is used to store the scheduler file
$> docker volume create --name ReportsVolume
$> docker volume create --name SchedulerVolume
After that, you should simply need to run
$> docker run -e PROCESSOR_TYPE=<PROCESSOR_TYPE> -v ReportsVolume:/tmp/reports -v SchedulerVolume:/tmp/celerybeat-schedule.d -dit tixmeasurements/tix-time-processor:latest-<type>
Where the environment variable PROCESSOR_TYPE
might take the following values:
STANDALONE
(This is the default value)BEAT
WORKER
This works for both, the standalone and the scheduler / beat type.
For the worker type, you should omit the SchedulerVolume, like this
$> docker run -e PROCESSOR_TYPE=WORKER -v ReportsVolume:/tmp/reports -dit tixmeasurements/tix-time-processor:latest-<type>
There are other configuration variables available for the Docker Container besides PROCESSOR_TYPE
CELERY_BEAT_SCHEDULE_DIR
: The directory where the Celery Beat schedule file will be stored. (Default: /tmp/celerybeat-schedule.d)CELERY_LOG_LEVEL
: The logging level for the Celery app. (Default: INFO)
This module also contains a Jupyter Notebook for analyzing calibration tests results. This module can be used after running the calibration test described in the tix-time-system-test module, in order to analyze the results of those tests.
These instructions were tested on Ubuntu 16.04 (64 bits).
Check that you have python3 along with pip3 and virtualenv:
python3 --version
pip3 --version
virtualenv --version
If these packages are not installed, install them running the following:
sudo apt update
sudo apt install python3-dev python3-pip
sudo pip3 install -U virtualenv # system-wide install
From the tix-time-processor
directory, execute:
virtualenv --no-site-packages -p python3 tix-processor-venv
source tix-processor-venv/bin/activate
The virtualenv name could be any name. Here we chose tix-processor-venv
.
Install dependencies in the virtualenv:
pip install -r requirements-notebook.txt
Before running this analysis, make sure you have the following files as a result of running the test:
├── description.yml
├── network_usage.log
├── tix-time-client-logs
│ ├── 1543816931101-21731100485417-tix-log.json
│ ├── 1543816992082-21792081601989-tix-log.json
│ ├── 1543817053093-21853092557779-tix-log.json
│ ├── 1543817114112-21914111663645-tix-log.json
│ ├── 1543817175075-21975074536638-tix-log.json
│ ├── 1543817236110-22036109811562-tix-log.json
| ...
│ ├── 1543838833090-43633089121888-tix-log.json
│ └── 1543838894094-43694093401424-tix-log.json
└── torrent_client.log
description.yml
: Test configuration file used during testnetwork_usage.log
: Log with the estimated download speed in kbps from the local interfacetorrent_client.log
: Log with stdout of torrent client spawned during calibration testtix-time-client-logs/
: Directory for tix-time-client log files. The naming format of the log files should be<unix_timestamp>-<java_timestapmp>-tix-log.json
First we need to group the log files generated by the TiX client into multiple batches. This is to imitate the way the tix-time-processor takes the files and computes them by separating them into different directories.
In order to run the formatter, simply run from tix-time-processor/reports_batch_formatter
:
python formatter.py \
--source_directory=path/to/tix-time-client-logs \
--output_directory=path/to/output \
This will generate a file called batch-test-report.tar.gz
in the output_directory
. Extract the tar in a directory of its own (we will need to reference that directory later).
cd <output_directory>
mkdir batch-test-report
tar -xzf batch-test-report.tar.gz -C batch-test-report/
In order to run the notebook for visualizing the results, from the tix-time-processor
directory run:
jupyter notebook
This should open up a Jupyter Notebook in your browser. From there, open the general_analysis_notebook.ipynb
notebook.
In the first cell of the notebook, you can specify the paths to the reports batches directory (generated in the previous step), the YAML configuration file used for the test, and the network usage log file. For example:
BATCHES_DIR = '/path/to/batch-test-report'
TEST_CONFIG_FILE = '/path/to/description.yml'
NET_USAGE_FILE='/path/to/network_usage.log'
Afterwards, you can simply execute all the cells in the notebook, and should be able to visualize the results.