Deploying rollup rules about sloth HOT 4 OPEN

@slok so do we think to work across prometheus instances / thanos across clusters - do we think it could work to have a sloth instance in each cluster housing the raw metrics to only handle recording rules, then a higher-level upstream sloth instance to aggregate alerting?

I guess I'm not positive how that would work across kubernetes clusters - a recording rule created in cluster A won't be query-able from cluster B or C, right? Unless sloth had the ability to query across all clusters to create rules / alerts by somehow pointing to the aggregation point (like Thanos)?

I'm only now getting into the weeds with this, so I'm sure I'm over-simplifying something or misunderstanding something.

from sloth.

Hi @SuperQ!

Thanks for opening the issue.

I don't fully understand what you are suggesting, could you elaborate a bit more please :) At what level do you mean splitting the rules? and using the CLI? k8s operator?

Maybe with an example, I could understand the problem easier.

Many thanks!

from sloth.

So this is the pattern developed to handle this in GitLab's jsonnet-SLI/SLO system:

https://gitlab.com/gitlab-com/runbooks/-/raw/master/metrics-catalog/img/metrics-overview.png

Basically, we have part of the SLI metrics like error rate, traffic rate, etc generated in Prometheus. But because the same services exists in more than one Kubernetes cluster, we used Thanos Rule to evaluate the ratios like error rate / traffic rate globally.

Split the SLI component from the SLI aggregations.

from sloth.

Right! I think I get it.

In the past, the only split feature that was required by a user was "alerts/recordings". The approach taken here was to allow disabling things so people could execute the way they wanted to split the things on their own use case (e.g execute once for alerts, once for recordings):

Adding different flags to disable SLIs, SLOs, metadata rules... so a user could enable/disable the different types of executions to make their own custom split... Would this multiple disable-flag approach work for you?

from sloth.