Add label propagation to Pyrra about pyrra HOT 11 CLOSED

In the last few days, I've given this a lot more thought.
The idea of having special pyrra.dev/team: sre prefixed labels seems like a great idea actually.
That way commands like kubectl get slo -l pyrra.dev/team=sre will work and we can propagate these labels to the PrometheusRule on Kubernetes too, such that kubectl get prometheusrule -l pyrra.dev/team=sre should work as well.
It shouldn't be a big problem to extract them into the metrics either and have them as team="sre" in the rules and alerts as well as on the UI for the list.

from pyrra.

@obitech We could certainly use this feature and would greatly appreciate the effort!

from pyrra.

I'll take a look! My yearly PTO is coming up so it might take a bit...

from pyrra.

Working on this now :)

from pyrra.

Would having the labels in the spec make it safe to propagate all labels to the alerts? This would let people use their existing alert manager configuration.

from pyrra.

Yes, that's the idea with having the labels in the spec. They would essentially become SLO labels rather than Kubernetes labels, if you will.

from pyrra.

I'm not 100% sure if we really want to add the label to the metadata.labels. It could be better to have a simliar thing to what Deployments and StatefulSets do, where we would have a spec.metadata.labels. So we can have separate labels for Kubernetes and the SLO itself.

If we want to make a separation but don't want to create a new field under the spec, we could also prefix the labels which are meant to be propagated:

apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: your-slo-name-here
  labels:
    prometheus: k8s
    role: alert-rules
    pyrra.dev.alerts/team: pyrra

from pyrra.

That sounds reasonable. What would be the best way forward to implement this? I'd be happy to draft a PR for the label propagation, if you're not already on it 😌

from pyrra.

Currently, I'm not working on this yet. I'm mostly active on these: https://github.com/pyrra-dev/pyrra/projects/4
If you want to take it, go ahead! 🎉

from pyrra.

Any updates on this feature?

from pyrra.

Not sure about how other people feel about it, but now that an Alerting struct was introduced at #375 I would expect this struct to be the source of truth when it comes to alert customization.

I was thinking that this struct could customize not only alertname(#386), but also labels and annotations. Would it make sense to replace the label propagation strategy? I believe having two different ways of propagation for labels will lead to confusion.

Another strategy is to add annotation propagation with the same strategy we have for labels now: through k8s annotation propagation. So at least we have label and annotation propagation in a consistent way 🙂

from pyrra.