Grafana dashboard not generates metrics to Burning rate about sloth HOT 5 OPEN

Hi @Eddie4Frost!

In order to help you, I need more context :D

• Sloth logs.
• SLO.
• Are the metrics on the prometheus.
• The generated rules are loaded on Prometheus.
• ...

from sloth.

Hi @slok, my bad! Thank you for your response, below:
The metrics available on Prometheus:

The sloth log:
INFO[0000] SLI plugins loaded plugins=0 svc=storage.FileSLIPlugin version=v0.9.0 window=30d INFO[0000] SLO period windows loaded svc=alert.WindowsRepo version=v0.9.0 window=30d windows=2 INFO[0000] Loading Kubernetes configuration... version=v0.9.0 INFO[0000] Kubernetes controller running version=v0.9.0 window=30d INFO[0000] starting controller controller-id=sloth lib=kooper service=kooper.controller version=v0.9.0 window=30d INFO[0000] Hot-reload http server listening addr=":8082" version=v0.9.0 window=30d INFO[0000] Metrics http server listening addr=":8081" version=v0.9.0 window=30d INFO[0000] OS signals listener started version=v0.9.0 window=30d INFO[0000] Hot-reload manager running version=v0.9.0 window=30d INFO[0001] Hot-reload triggered from http webhook version=v0.9.0 window=30d INFO[0001] SLI plugins loaded plugins=15 svc=storage.FileSLIPlugin version=v0.9.0 window=30d

The rule:

Grafana Query Inspector query:
Object request:Object url:"api/datasources/proxy/1/api/v1/query_range" method:"POST" data:Object query:"1-( sum_over_time( ( slo:sli_error:ratio_rate1h{sloth_service="myservice",sloth_slo="requests-availability"} * on() group_left() ( month() == bool vector(12) ) )[32d:1h] ) / on(sloth_id) ( slo:error_budget:ratio{sloth_service="myservice",sloth_slo="requests-availability"} *on() group_left() (24 * days_in_month()) ) )" start:1638327600 end:1641006000 step:3600 hideFromInspector:false response:Object status:"**success**" data:Object resultType:"matrix" result:Array[0]

Thank you

from sloth.

Do other SLOs work for you, or happen on all of them?. Maye it's just that you aren't burning anything (rare, but who knows...) so there's nothing to chart. To test, you can use the "fake" plugin to simulate different burn rates (this should show you something always):

---
apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
  name: slo-test
  labels:
    prometheus: prometheus
    role: alert-rules
spec:
  service: "test"
  slos:
    - name: "five-nines"
      objective: 99.999
      sli:
        plugin:
          id: "sloth-common/fake"
          options:
            burn_rate: "0.65"
            jitter_percent: "10"
      alerting:
        name: "TestSLOFiveNinesFailing"
        pageAlert:
          disable: true
        ticketAlert:
          disable: true

from sloth.

@slok do you think adding a vector(0) in this case would make sense?

1-(
  sum_over_time(
    (
       slo:sli_error:ratio_rate1h{sloth_service="${service}",sloth_slo="${slo}"}
       * on() group_left() (
         month() == bool vector(${__to:date:M})
       )
    )[32d:1h]
  )
  / on(sloth_id)
  (
    slo:error_budget:ratio{sloth_service="${service}",sloth_slo="${slo}"} *on() group_left() (24 * days_in_month())
  ) or vector(0)
) 

from sloth.

Hi @slok I do have the same problem, here's a screenshot for the fake plugin as you mentioned on your last comment:

I'm using the helm chart version
I don't see anything wrong on the sloth pod logs:
INFO[0000] SLI plugins loaded plugins=0 svc=storage.FileSLIPlugin version=v0.9.0 window=30d INFO[0000] SLO period windows loaded svc=alert.WindowsRepo version=v0.9.0 window=30d windows=2 INFO[0000] Loading Kubernetes configuration... version=v0.9.0 INFO[0000] Kubernetes controller running version=v0.9.0 window=30d INFO[0000] Hot-reload http server listening addr=":8082" version=v0.9.0 window=30d INFO[0000] Metrics http server listening addr=":8081" version=v0.9.0 window=30d INFO[0000] Hot-reload manager running version=v0.9.0 window=30d INFO[0000] OS signals listener started version=v0.9.0 window=30d INFO[0000] starting controller controller-id=sloth lib=kooper service=kooper.controller version=v0.9.0 window=30d INFO[0001] Hot-reload triggered from http webhook version=v0.9.0 window=30d INFO[0001] SLI plugins loaded plugins=15 svc=storage.FileSLIPlugin version=v0.9.0 window=30d
Thank you!

from sloth.