labbsr0x / bb-promster Goto Github PK
View Code? Open in Web Editor NEWA promster image definition to properly work with the Big Brother project
License: MIT License
A promster image definition to properly work with the Big Brother project
License: MIT License
The current configuration of bb-promster uses the same scheme of Prometheus at Alertmanager config:
alerting:
alertmanagers:
- scheme: {{.scheme}}
static_configs:
- targets: ['$ALERT_MANAGER_URLS']
It may be necessary to change this configuration to use different schemes.
Issue #15 creates a mechanism to update alert rules and also shows an example of a comparative alert using the error rate metric. Create more examples of comparative alerts using latency and traffic metrics would be interesting.
After a container/pod restart, when there's /versions registered in ETCD, the bb-promster container breaks.
The problem happens due to the main.go file, that takes care of updating alert rules files. It needs further investigation to find out why the problem is happening.
Bb-promster is being used in production for a while. Before bb-promster, scrape targets were set statically on building run-time. bb-promster introduces an ETCD database to manage all IPs that Prometheus needs to monitor, speeding up the targets update process.
However, it has some issues to update some attributes that belong to a target. Currently, bb-promster only watch changes on IPs, but on real monitoring, it's also necessary to keep tracking of scrape path, relabel_configs, and SSL certificate. It could be interesting to allow bb-promster to watch changes on other attributes, so the need to modify the deploy will be minimum.
Considering that, we propose bb-promster not watch just IPs, but a complete app object as bb-bot does.
App structure will be responsible for keeping data related to a context being monitored, attributes like name, scrapePath, and so on. App also has an internal struct called Env to manage contexts with blue/green deployment.
Settings is an auxiliary structure to keep data related to all bb-promster, as the remote write URL and etcd service where apps are saved.
It is an initial proposal that can also include the bb-promsters status or pod names in order to centralize information for operation teams.
Thanks to @KarineValenca for helping me with the text.
The new functionality should be documented on BB Promster's readme.
We should provide a cortex example
It's necessary to add more labels to comparatives alerts to give more information about the alert to the user. Useful labels:
prsn -> informs the App in which the alarm is firing. Also, this label is mandatory in order to bb-bot send the alert to Telegram.
addr -> inform the URL in which the alarm is firing.
In order to interact to telegram bot, the alertmanager rules must have an annotation app. The annotation will be used to search all subscriptions. The follow example shows how to define it:
- alert: dependency_unresponsive
expr: dependency_up < 1
for: 2m
annotations:
app: {{$labels.prsn}}
description: A dependência {{$labels.name}} do serviço {{$labels.prsn}} não está respondendo.
summary: Alerta para caso uma dependência passe mais de 2 minutos sem resposta
Add cortex tag on docker-compose in cortex folder.
Prometheus remote write config can add two types of Authorization headers: basic auth e bearer token.
As described on the documentation:
# The URL of the endpoint to send samples to.
url: <string>
# Timeout for requests to the remote write endpoint.
[ remote_timeout: <duration> | default = 30s ]
# List of remote write relabel configurations.
write_relabel_configs:
[ - <relabel_config> ... ]
# Sets the `Authorization` header on every remote write request with the
# configured username and password.
# password and password_file are mutually exclusive.
basic_auth:
[ username: <string> ]
[ password: <string> ]
[ password_file: <string> ]
# Sets the `Authorization` header on every remote write request with
# the configured bearer token. It is mutually exclusive with `bearer_token_file`.
[ bearer_token: <string> ]
# Sets the `Authorization` header on every remote write request with the bearer token
# read from the configured file. It is mutually exclusive with `bearer_token`.
[ bearer_token_file: /path/to/bearer/token/file ]
The run.sh script, on bb-promster, just add the the remote_url as the code shows:
# the user can also configure a remote url to send prometheus metrics to
if [[ "$REMOTE_WRITE_URL" != "" ]]; then
cat >> "/prometheus.yml.tmpl" <<- EOM
remote_write:
- url: $REMOTE_WRITE_URL
EOM
fi
In order to support others prometheus' remote config, I propose adding more two env variables: REMOTE_BEARER_TOKEN and REMOTE_BASIC_AUTH. So the run.sh script can add the config if one of them is set.
Em todos níveis deveriamos ter a liberdade de configurar regras de alertas e endereço do alertmanager.
BB Promster should be able to deal with canary deployment comparison rules.
To be able to do that, we'll leverage ETCD to register which is the pilot version and which is the prod version. BB Promster's alerting "sidecar" will then reach for such configs and load new template based alerting rules.
The sidecar should be installed in BB Promster's docker image by leveraging Docker's multi-stage build capabilities.
Add the alert rules from big-brother monitor and create an option to user activate these alerts or not.
Remove restrictions that only allow alertmanager to manage alerts on level 1 of the federation.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.