This package is designed to record metrics of the application into a backend. With the help of grafana you can easily create dashboards with them
- Sending data to multiple backends
- Custom backends
- Hooks
- InfluxDB 0.8
- Elasticsearch 5
- Kafka
If you want to use it with ElasticSearchBackend
$ pip install timeexecution[elasticsearch]
with InfluxBackend
$ pip install timeexecution[influxdb]
with KafkaBackend
$ pip install timeexecution[kafka]
or if you prefer to have all backend to easily switch between them
$ pip install timeexecution[all]
To use this package you decorate the functions you want to time its execution. Every wrapped function will create a metric consisting of 3 default values:
- name - The name of the series the metric will be stored in
- value - The time it took in ms for the wrapped function to complete
- hostname - The hostname of the machine the code is running on
See the following example
from time_execution import settings, time_execution
from time_execution.backends.influxdb import InfluxBackend
from time_execution.backends.elasticsearch import ElasticsearchBackend
# Setup the desired backend
influx = InfluxBackend(host='influx', database='metrics', use_udp=False)
elasticsearch = ElasticsearchBackend('elasticsearch', index='metrics')
# Configure the time_execution decorator
settings.configure(backends=[influx, elasticsearch])
# Wrap the methods where u want the metrics
@time_execution
def hello():
return 'World'
# Now when we call hello() and we will get metrics in our backends
hello()
This will result in an entry in the influxdb
[
{
"name": "__main__.hello",
"columns": [
"time",
"sequence_number",
"value",
"hostname",
],
"points": [
[
1449739813939,
1111950001,
312,
"machine.name",
]
]
}
]
And the following in Elasticsearch
[
{
"_index": "metrics-2016.01.28",
"_type": "metric",
"_id": "AVKIp9DpnPWamvqEzFB3",
"_score": null,
"_source": {
"timestamp": "2016-01-28T14:34:05.416968",
"hostname": "dfaa4928109f",
"name": "__main__.hello",
"value": 312
},
"sort": [
1453991645416
]
}
]
It's also possible to run backend in different thread with logic behind it, to send metrics in bulk mode.
For example:
from time_execution import settings, time_execution from time_execution.backends.threaded import ThreadedBackend # Setup threaded backend which will be run on separate thread threaded_backend = ThreadedBackend( backend=ElasticsearchBackend, backend_kwargs={ "host" : "elasticsearch", "index": "metrics", } ) # there is also possibility to configure backend by import path, like: threaded_backend = ThreadedBackend( backend="time_execution.backends.kafka.KafkaBackend", #: any other configuration belongs to backend backend_kwargs={ "hosts" : "kafka", "topic": "metrics" } ) # Configure the time_execution decorator settings.configure(backends=[threaded_backend]) # Wrap the methods where u want the metrics @time_execution def hello(): return 'World' # Now when we call hello() we put metrics in queue to send it either in some configurable time later # or when queue will reach configurable limit. hello()
It's also possible to decorate coroutines or awaitables in Python >=3.5.
For example:
import asyncio
from time_execution import time_execution_async
# ... Setup the desired backend(s) as described above ...
# Wrap the methods where you want the metrics
@time_execution_async
async def hello():
await asyncio.sleep(1)
return 'World'
# Now when we schedule hello() we will get metrics in our backends
loop = asyncio.get_event_loop()
loop.run_until_complete(hello())
time_execution supports hooks where you can change the metric before its being sent to the backend.
With a hook you can add additional and change existing fields. This can be useful for cases where you would like to add a column to the metric based on the response of the wrapped function.
A hook will always get 3 arguments:
- response - The returned value of the wrapped function
- exception - The raised exception of the wrapped function
- metric - A dict containing the data to be send to the backend
- func_args - Original args received by the wrapped function.
- func_kwargs - Original kwargs received by the wrapped function.
From within a hook you can change the name if you want the metrics to be split into multiple series.
See the following example how to setup hooks.
# Now lets create a hook
def my_hook(response, exception, metric, func_args, func_kwargs):
status_code = getattr(response, 'status_code', None)
if status_code:
return dict(
name='{}.{}'.format(metric['name'], status_code),
extra_field='foo bar'
)
# Configure the time_execution decorator, but now with hooks
settings.configure(backends=[backend], hooks=[my_hook])
You can also send any metric you have manually to the backend. These will not add the default values and will not hit the hooks.
See the following example.
loadavg = os.getloadavg()
write_metric('cpu.load.1m', value=loadavg[0])
write_metric('cpu.load.5m', value=loadavg[1])
write_metric('cpu.load.15m', value=loadavg[2])
Writing a custom backend is very simple, all you need to do is create a class with a write method. It is not required to extend BaseMetricsBackend but in order to easily upgrade I recommend u do.
from time_execution.backends.base import BaseMetricsBackend
class MetricsPrinter(BaseMetricsBackend):
def write(self, name, **data):
print(name, data)
You have something to contribute ? Great ! A few things that may come in handy.
Testing in this project is done via docker. There is a docker-compose to easily get all the required containers up and running.
There is a Makefile with a few targets that we use often:
- make test
- make isort
- make lint
- make build
- make setup.py
All of these make targets can be prefixed by docker/. This will execute the target inside the docker container instead of on your local machine. For example make docker/build.