Comments (10)
sstarcher
commented on August 22, 2024
in addition availability and error budget on the main page all show no data.
from pyrra.
sstarcher
commented on August 22, 2024
Looking at the query being ran.
QueryRange is responding with {"matrix":{}}
The query in the payload looks correct and is what shows up when you click the prometheus link which has a correct graph.
end
:
"1691592397"
query
:
"((1 - 0.99) - (sum(grpc_server_handled:increase2w{app_kubernetes_io_name=\"xx\",code=~\"yyy\",grpc_method=~\"x\",slo=\"xx\"} or vector(0)) / sum(grpc_server_handled:increase2w{app_kubernetes_io_name=\"xx\",grpc_method=~\"xx\",slo=\"xx\"}))) / (1 - 0.99)"
start
:
"1691588797"
step
:
"3"
from pyrra.
metalmatze
commented on August 22, 2024
Does the data show up after a bit?
This is most likely due to the recording rules only being created and computed once the SLO has been created.
If the graphs eventually fill with data this issue would be actually related to prometheus/prometheus#10202 where one idea would be to add an API to Prometheus to trigger backfilling of data for newly created recording rules.
from pyrra.
sstarcher
commented on August 22, 2024
I have about 2 hours of data now. Do I need to wait longer or would suspect that 2 hours would show?
from pyrra.
metalmatze
commented on August 22, 2024
The first data should show up within 3min and from there every 2.5min a new data point will be added to the recording rule's metric name. I suspect something about the time series and the config you have must be mismatched. It's hard to say with the information we have in this issue now. I would try looking into the grpc_server_handled:increase2w time series in Prometheus and check the queries in Prometheus directly that Pyrra sends to Prometheus (the one you have pasted above).
from pyrra.
sstarcher
commented on August 22, 2024
grpc_server_handled:increase2w shows up perfectly fine in Prometheus.
An entry looks like this and I have about a dozen of them
grpc_server_handled:increase2w{app_kubernetes_io_name="xxx", code="xxx", grpc_method="xxx", slo="xxx", team="xxx"}
record:[grpc_server_handled:increase2w](https://dataservices.dev.telemetry.bwce.io/graph?g0.expr=grpc_server_handled%3Aincrease2w&g0.tab=1&g0.stacked=0&g0.show_exemplars=0.g0.range_input=1h.)
expr:[sum by (code, grpc_method, route) (increase(grpc_server_handled_total{app_kubernetes_io_name="xxx",grpc_method=~"xxx"}[2w]))](https://dataservices.dev.telemetry.bwce.io/graph?g0.expr=sum%20by%20(code%2C%20grpc_method%2C%20route)%20(increase(grpc_server_handled_total%7Bapp_kubernetes_io_name%3D%22cachefiller%22%2Cgrpc_method%3D~%22Cache%22%7D%5B2w%5D))&g0.tab=1&g0.stacked=0&g0.show_exemplars=0.g0.range_input=1h.)
labels:
app_kubernetes_io_name: xxx
slo: xxx
team: xxx
Could the addition of code be causing your logic some issue?
As I mentioned the queries that Pyrra links work when I run them in Prometheus. This has now been running for over 24 hours and I still see the same result.
Let me know if I can provide anything else or do anything on my side, but I don't see anything in the Pyrra logs.
My SLO looks like this
---
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
creationTimestamp: "2023-08-09T12:13:08Z"
generation: 2
labels:
prometheus: k8s
pyrra.dev/team: xxx
role: alert-rules
name: xxx
namespace: qqqqq
spec:
description: Pyrra's API requests and response errors over time grouped by route.
indicator:
ratio:
errors:
metric: grpc_server_handled_total{app_kubernetes_io_name="xxx",
grpc_method=~"xxx", code=~"OK|NOT_FOUND|INVALID_ARGUMENT|PERMISSION_DENIED|FAILED_PRECONDITION|RESOURCE_EXHAUSTED"}
grouping:
- route
total:
metric: grpc_server_handled_total{app_kubernetes_io_name="xxx",
grpc_method=~"xxx"}
target: "99"
window: 2w
---
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
creationTimestamp: "2023-08-09T12:28:00Z"
generation: 1
labels:
prometheus: k8s
pyrra.dev/method: yyyyy
pyrra.dev/team: test
role: alert-rules
name: fffff
namespace: qqqqq
spec:
description: Pyrra's API requests and response errors over time grouped by route.
indicator:
ratio:
errors:
metric: grpc_server_handled_total{grpc_method=~"yyyyy",
code=~"OK|NOT_FOUND|INVALID_ARGUMENT|PERMISSION_DENIED|FAILED_PRECONDITION|RESOURCE_EXHAUSTED"}
grouping:
- route
total:
metric: grpc_server_handled_total{grpc_method=~"yyyyy"}
target: "99"
window: 2w
---
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
creationTimestamp: "2023-08-09T12:07:16Z"
generation: 1
labels:
prometheus: k8s
pyrra.dev/team: operations
role: alert-rules
name: pyrra-api-errors
namespace: qqqqq
spec:
description: Pyrra's API requests and response errors over time grouped by route.
indicator:
ratio:
errors:
metric: grpc_server_handled_total{grpc_method=~"xxx", code=~"OK|NOT_FOUND|INVALID_ARGUMENT|PERMISSION_DENIED|FAILED_PRECONDITION|RESOURCE_EXHAUSTED"}
grouping:
- route
total:
metric: grpc_server_handled_total{grpc_method=~"xxx"}
target: "99"
window: 2w
---
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
creationTimestamp: "2023-08-09T12:16:28Z"
generation: 2
labels:
prometheus: k8s
pyrra.dev/method: zzzz
pyrra.dev/team: zzzz
role: alert-rules
name: zzzzz
namespace: qqqqq
spec:
description: Pyrra's API requests and response errors over time grouped by route.
indicator:
ratio:
errors:
metric: grpc_server_handled_total{app_kubernetes_io_name="zzzz",
grpc_method=~"zzzz", code=~"OK|NOT_FOUND|INVALID_ARGUMENT|PERMISSION_DENIED|FAILED_PRECONDITION|RESOURCE_EXHAUSTED"}
grouping:
- route
total:
metric: grpc_server_handled_total{app_kubernetes_io_name="zzzz",
grpc_method=~"zzzz"}
target: "99"
window: 2w
---
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
creationTimestamp: "2023-08-09T12:16:28Z"
generation: 2
labels:
prometheus: k8s
pyrra.dev/method: zzzzz
pyrra.dev/team: zzzz
role: alert-rules
name: qqqq
namespace: qqqqq
spec:
description: Pyrra's API requests and response errors over time grouped by route.
indicator:
ratio:
errors:
metric: grpc_server_handled_total{app_kubernetes_io_name="zzzz",
grpc_method=~"zzzzz", code=~"OK|NOT_FOUND|INVALID_ARGUMENT|PERMISSION_DENIED|FAILED_PRECONDITION|RESOURCE_EXHAUSTED"}
grouping:
- route
total:
metric: grpc_server_handled_total{app_kubernetes_io_name="zzzz",
grpc_method=~"zzzzz"}
target: "99"
window: 2w
---
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
creationTimestamp: "2023-08-09T11:57:05Z"
generation: 2
labels:
prometheus: k8s
pyrra.dev/team: operations
role: alert-rules
name: pyrra-api-errors
namespace: monitoring
spec:
description: Pyrra's API requests and response errors over time grouped by route.
indicator:
ratio:
errors:
metric: grpc_server_handled_total{grpc_method=~"xxx", code=~"OK|NOT_FOUND|INVALID_ARGUMENT|PERMISSION_DENIED|FAILED_PRECONDITION|RESOURCE_EXHAUSTED"}
grouping:
- route
total:
metric: grpc_server_handled_total{grpc_method=~"xxx"}
target: "99"
window: 2w
from pyrra.
sstarcher
commented on August 22, 2024
Checked back and still no info in logs and no graphs are visible.
from pyrra.
giz33
commented on August 22, 2024
I have the same issue here.
On the home page of pyrra it shows No data on Availbility and Budget:
But when I click on it, it shows the values normally(Don't know if 100% is ok for them, but at least show some values):
Can someone help please?
Tks in advance.
from pyrra.
giz33
commented on August 22, 2024
I was able to make it works!
Again, was a mismatch between label that the prometheus operator uses and the label that pyrra puts on the ServiceLevelObectives and PrometheusRules.
After change the Prometheus to look for the labels used on the Pyrra's objects it works.
Tks again.
from pyrra.
prein
commented on August 22, 2024
Same here. No data in Pyrra UI but all visible in Grafana. I noticed the same issue on this public Pyrra UI
Can it be related to prometheus / thanos version? I think it happens more after an upgrade I performed recently
@giz33 can you elaborate on the labels change? In my case it started working after adding some labels but I don't understand how it worked.
PS. Possibly same issue #632 - solved by getting SLO resources to live in the same NS with Pyrra
@metalmatze do you think it would be possible to make issues like this visible in SLO CR status (available with kubectl describe ServiceLevelObjective)? Or is it strictly UI?
from pyrra.
Related Issues (20)
- No data Budget and Availability HOT 1
- Doubt - Error Budget
- Issue when showing some SLOs (white screen) HOT 3
- Pyrra causes instability of Prometheus HOT 1
- Fix typo in Kubernetes example readme HOT 1
- Allow providing an error rate query rather than a error count query HOT 1
- ConfigMapMode not working with option --config-map-mode=true
- Webhook Panic on Unexpected Parser Expression HOT 2
- Allow Prometheus Rules to Publish Rules to Azure Managed Prometheus HOT 3
- Grafana Alerting & Mimir support HOT 9
- Latency format incorrect when using milliseconds HOT 2
- Latency SLOs - Duplicated "p100" percentile in duration graph (UI) HOT 2
- Duration graph legend shows inaccurate value when duration is greater than 1 second
- No data and generic rules not created
- How do I create a SLO for this kafka metrics
- Absent Alert not disabled when using `spec.alerting.disabled=true`
- Missing precision for Multi Burn Rate Alerts graphs (UI)
- bool_gauge SLO's budget is burning continouosly
- The GenericRules is using sum() in counter metrics HOT 3
- Add support for Prometheus offset modifier HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyrra.