Comments (8)
You are right this is not a good outcome, however, is how prometheus works,NaN != Null != 0
. What you can do to fix this situation, is only create metrics when you know that the metrics will be valid, this is when the total events are >0
.
So, having this SLI:
sli:
events:
error_query: sum(rate(http_request_duration_seconds_count{job="myservice",code=~"(5..|429)"}[{{.window}}]))
total_query: sum(rate(http_request_duration_seconds_count{job="myservice"}[{{.window}}]))
Only get SLIs when we can have them:
sli:
events:
error_query: sum(rate(http_request_duration_seconds_count{job="myservice",code=~"(5..|429)"}[{{.window}}]))
total_query: sum(rate(http_request_duration_seconds_count{job="myservice"}[{{.window}}])) > 0
This way, the total SLI query will return Null (no metrics) when there are no total metric datapoints and there will not be NaNs that will corrupt the rules that are created from these SLI metrics.
To test this you could deploy a new SLO with this change along with your current SLO and check if its its working for you while you see both in action.
This is the simplest thing I think it would work, let me know if this works for your use case. If not we can think of a better query using raw
SLI type (I did this in the past and worked, however is a bit more complex).
If it works with this simple change, I'm wondering if Sloth could do this by default or let the user do this and add to the F.A.Q, not strong opinion at this moment.
from sloth.
Thanks for the informations. I will give it a try.
I also found this interesting video from SLOconf which might solve some of the issues; not 100% sure if its related but I think it is. I wouldn't have a clue how to express it in PromQL. First feeling is it might need some recording rule
https://www.youtube.com/watch?v=R4nCsgt1qEU
from sloth.
Thanks @weyert!
The video is very interesting when the services have few metrics!
from sloth.
@simonjefford Any news on this one? AFAIK @weyert has tested this and seems that worked for Weyert's use case.
from sloth.
Apologies - I haven't had a chance to give this a go yet. Hoping to do so this afternoon. Thank you for your response!
from sloth.
Hey @slok,
Just throwing my two cents here - we had similar problem on low traffic endpoints and '>0' for total_query seems to resolve the issue.
from sloth.
Hi @mrtworo!
Thanks for the information! is very valuable!
Due to the inactivity of @simonjefford and your answer that seems to solve the problem I'll assume the problem is solved.
@simonjefford if you continue with the problem, please reopen the issue :)
Thanks to all of you that contributed to this issue!
from sloth.
from sloth.
Related Issues (20)
- Please add example manifest for OpenSLO with alerting section
- Long run of slo and slo versioning HOT 2
- How can I reset the error budget remaining to 100 for 7 days from 30days HOT 3
- Option to generate sloth yaml - using 5m record rule chaining
- Sloth Alerting Rules Not Firing - Graphs Empty on Query Test HOT 1
- Help on Latency SLO definition HOT 1
- Testing an operator which manages Sloth SLOs HOT 1
- Question: Is there a way to refer totalQuery via template variable in errorQuery
- promql expr validation issues HOT 2
- Overriding the `sloth_id` doesn't work
- Feature Request: Provision SLOs from Helm install HOT 2
- Issues making Sloth work with Google Managed Prometheus HOT 1
- what does the current remaining buget -4.69e -12% mean in sloth HOT 2
- Alerting expression changes in Prometheus Alerts browser HOT 1
- 🔴 Project Status HOT 9
- Confusing definitions of errorQuery and totalQuery
- Have you considered creating 'totalQuery' as a recording rule as well?
- grafana dashboard broken for SLOs with dots in the name
- error: "generate" command failed: invalid spec, could not load with any of the supported spec types HOT 1
- How can one add a weekly maintenance window into the calculations for SLO's with sloth? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sloth.