Comments (1)
At least you first need prometheus to record maintenance windows. Either some system that reports this as metric, or if its a fixed time, you could build a recording rule with the day_of_week and hour functions.
Then I'd probably cut my losses and just define a inhibit rule to avoid sending alerts during maintenance period (maybe add a buffer around the period). The slo calculations and boards would still take errors during maintenance period into account though. I would find that advantageous though as I'd anyways encourage trying to limit impact of maintenance periods.
Still if you really need calculations to exclude maintenance periods, then the approach would likely depend on your query. Assuming maintenance_period
recording rule that reports 1
during mainteance, 0
otherwise, then maybe queries like this would do the trick
error_query: |
sum_over_time((
sum(rate(<error_counter>[30s]))
* scalar(1-maintenance_period)
)[{{.window}}:])
total_query: |
(sum_over_time((
sum(rate(<total_counter>[30s]))
* scalar(1-maintenance_period)
)[{{.window}}:]) > 0) or vector(1)
Basically if you take rate over full window period, you wouldn't know which errors happened during maintenance period. sum_over_time
should still ensure the error ratio is a quite good approximation for the entire window period. > 0 or vector(1)
will be quite important to include as the error ratios would otherwise have 0 denominator inside any maintenance period
from sloth.
Related Issues (20)
- Option to generate sloth yaml - using 5m record rule chaining
- Sloth Alerting Rules Not Firing - Graphs Empty on Query Test HOT 1
- Help on Latency SLO definition HOT 1
- Testing an operator which manages Sloth SLOs HOT 1
- Question: Is there a way to refer totalQuery via template variable in errorQuery
- promql expr validation issues HOT 2
- Overriding the `sloth_id` doesn't work
- Feature Request: Provision SLOs from Helm install HOT 2
- Issues making Sloth work with Google Managed Prometheus HOT 1
- what does the current remaining buget -4.69e -12% mean in sloth HOT 2
- Alerting expression changes in Prometheus Alerts browser HOT 1
- 🔴 Project Status HOT 9
- Confusing definitions of errorQuery and totalQuery
- Have you considered creating 'totalQuery' as a recording rule as well?
- grafana dashboard broken for SLOs with dots in the name
- error: "generate" command failed: invalid spec, could not load with any of the supported spec types HOT 1
- NaN in SLO dashboard HOT 5
- Sloth pod is not showing SLO metrics HOT 1
- error: "generate" command failed: invalid spec, could not load with any of the supported spec types"
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sloth.