Comments (5)
Hi @slok,
Sorry for causing trouble again. Unfortunately we have some services that e.g. are only in use while formula1 events happen, which is some hours every Sunday and such, that would suffer a bit from this. I actually see it a bit differently, that it could be an ok approximation for alerting, while the slo-period is the actual contract the team commits to. Alerting is just there to help the team reach the contract.
Completely agree that you are better off doing >0 or vector(1)
things to avoid NaNs in the first place. The only real shortcoming I see the library could solve is that if you mess up the query, you have to wait 30d until the NaNs are scrubbed from prometheus. So maybe the optimized rule should more do
sum_over_time(x > 0) /
count_over_time(x > 0)
to simply disregard NaNs.
from sloth.
Awesome @tokheim @xairos, thanks for your inputs and raising your hand. Is very valuable to see that is not an isolated case.
I'll think about this, but most likely that this will be on the next release :)
Again many thanks!
from sloth.
Oh, and using the non-optimized calculation would also partially help for people experiencing NaNs like #231 in the period budget. If there is a NaN in any 5 minute slice, then the whole optimized sli recorder expression will evaluate to NaN. I guess there are other ways this could be improved though...
from sloth.
Hi again @tokheim
That optimization was made because normally most people only want the 30d as an informative metric (not used for alerting or similar things) and the approximation works in most cases (well, Prometheus itself is also an approximation of reality), not having it would kill a lot of Prometheus installations (due to queries) or end having no metric because of timeouts on the rules.
You are the first person (that I'm aware of) that this doesn't work for him/her.
Let me think about adding this option, in case it's added, it would be done at the application level (with a flag) and not at the SLO level, I would prefer to maintaint the specs as simple as possible.
Regarding the NaNs, it's a different problem that should be solved with the right queries (using a combination of >0
, vector(0)
...).
from sloth.
You are the first person (that I'm aware of) that this doesn't work for him/her
I wanted to add a +1 to say that we'd benefit as well from this!
I actually see it a bit differently, that it could be an ok approximation for alerting, while the slo-period is the actual contract the team commits to. Alerting is just there to help the team reach the contract.
I have this perspective as well, and before looking at the PromQL that Sloth generates, I actually thought that Sloth would use this method for calculating slo:period_error_budget_remaining:ratio
. I had been (erroneously) expecting that the SLOs would be calculated using good events / total events
(rather than good time / total time
using the optimized calculation).
We have many services with varying traffic patterns, ex. internal apps used by customer service that have almost no traffic outside of customer support hours. If those SLOs could be calculated using good events / total events
, they would become more useful for reporting 👍🏻
from sloth.
Related Issues (20)
- Please add example manifest for OpenSLO with alerting section
- Long run of slo and slo versioning HOT 2
- How can I reset the error budget remaining to 100 for 7 days from 30days HOT 3
- Option to generate sloth yaml - using 5m record rule chaining
- Sloth Alerting Rules Not Firing - Graphs Empty on Query Test HOT 1
- Help on Latency SLO definition HOT 1
- Testing an operator which manages Sloth SLOs HOT 1
- Question: Is there a way to refer totalQuery via template variable in errorQuery
- promql expr validation issues HOT 2
- Overriding the `sloth_id` doesn't work
- Feature Request: Provision SLOs from Helm install HOT 2
- Issues making Sloth work with Google Managed Prometheus HOT 1
- what does the current remaining buget -4.69e -12% mean in sloth HOT 2
- Alerting expression changes in Prometheus Alerts browser HOT 1
- 🔴 Project Status HOT 9
- Confusing definitions of errorQuery and totalQuery
- Have you considered creating 'totalQuery' as a recording rule as well?
- grafana dashboard broken for SLOs with dots in the name
- error: "generate" command failed: invalid spec, could not load with any of the supported spec types HOT 1
- How can one add a weekly maintenance window into the calculations for SLO's with sloth? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sloth.