Hi, thanks you for make sloth, seems a beautiful tool to setting up SLO's! I hope the

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Prometheusrule didn't work. about sloth HOT 3 CLOSED

slok commented on July 21, 2024

Prometheusrule didn't work.

from sloth.

Comments (3)

slok commented on July 21, 2024 3

Hi @b0nete!

I think that I know whats happening, your resulting rules don't have the matching rules required by the prom operator like the issue you pointed in #125

This is because you mixed Kubernetes labels and Prometheus labels. Check it:

From this:

apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
  name: sloth-slo-grafana
  namespace: kube-prometheus-stack
spec:
  service: "kube-prometheus-stack-grafana"
  labels:
    app: "kube-prometheus-stack"
    release: "kube-prometheus-stack"
    owner: "myteam"
    repo: "myorg/myservice"
    tier: "2"
  slos:
    - name: "requests-availability"
      objective: 99.9
      description: "Common SLO based on availability for HTTP request responses."
      sli:
        events:
          errorQuery: sum(rate(grafana_http_request_duration_seconds_count{job="kube-prometheus-stack-grafana",status_code=~"(5..|4..)"}[{{.window}}]))
          totalQuery: sum(rate(grafana_http_request_duration_seconds_count{job="kube-prometheus-stack-grafana"}[{{.window}}]))
      alerting:
        name: MyServiceHighErrorRate
        labels:
          category: "availability"
        annotations:
          summary: "High error rate on 'myservice' requests responses"
        pageAlert:
          labels:
            severity: pageteam
            routing_key: myteam
        ticketAlert:
          labels:
            severity: "slack"
            slack_channel: "#alerts-myteam"

apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
  name: sloth-slo-grafana
  namespace: kube-prometheus-stack
  labels:
    app: "kube-prometheus-stack"
    release: "kube-prometheus-stack"
spec:
  service: "kube-prometheus-stack-grafana"
  labels:
    owner: "myteam"
    repo: "myorg/myservice"
    tier: "2"
  slos:
    - name: "requests-availability"
      objective: 99.9
      description: "Common SLO based on availability for HTTP request responses."
      sli:
        events:
          errorQuery: sum(rate(grafana_http_request_duration_seconds_count{job="kube-prometheus-stack-grafana",status_code=~"(5..|4..)"}[{{.window}}]))
          totalQuery: sum(rate(grafana_http_request_duration_seconds_count{job="kube-prometheus-stack-grafana"}[{{.window}}]))
      alerting:
        name: MyServiceHighErrorRate
        labels:
          category: "availability"
        annotations:
          summary: "High error rate on 'myservice' requests responses"
        pageAlert:
          labels:
            severity: pageteam
            routing_key: myteam
        ticketAlert:
          labels:
            severity: "slack"
            slack_channel: "#alerts-myteam"