variable grouping about sloth HOT 7 OPEN

Sorry for the Spam @cznewt, I already reported to Github.

from sloth.

Thank you @slok did you think about having a grouping by variable? nodes/handlers/services
I'm testing it now, i can let you know how it works with current setup in a few days.

from sloth.

Yes, that's something that happens at the Prometheus level. You need to have the correct queries. In the past, I had some custom dashboards for the grouping of some SLOs that I knew they exists.

The problem is the discovery of the unknown grouping labels, and having them on the generic dashboards.

Let me know how it goes, maybe we should add a section on the docs for this purpose or use case.

Thanks, @cznewt!

from sloth.

Guys, I have the same need, for example, I have 100 microservices, on 5 environments, which generate +- 10k of rules.
I need some features for generating fewer rules.

For example doesn't specify my service, using one by service_name on "istio".
or using labels from istio, for example, node.

from sloth.

Hey everyone, I would also be interested in a solution for this. We have lots of Kafka Clusters, each in their own namespace. Would be great to be able to filter by namespace.
I'm using the PrometheusServiceLevel CRD to define queries. I added "by (namespace)" to each of my queries but the SLOTH metrics like slo:objective:ratio do not contain the namespace -> can't use it in Grafana.
Is there a solution for this?

Edit: adding "by (namespace)" works now, don't know what went wrong the first time ...

from sloth.

Hi @bthdimension!

There is no solution yet, however, I'm already thinking about how to tackle this problem, without being complex for the user.

from sloth.

To add to the discussion, having a way to keep a Prometheus label as part of the aggregation would also help with keeping some extra labels in alerts.

For example I have some "environment" or "tenant_id" labels that I would like to keep in the alerts for routing purpose without having to explicitly set them in each alert definition.

from sloth.