Thanos Ruler and duplicate series about sloth HOT 2 CLOSED

Hi @nextrevision!

Thanks!

If I understood the problem correctly, Once Prometheus stops handling the rules and Thanos is the only one processing them, eventually, the Sloth metrics created by Prometheus will disappear, however that would be too long.

After Thanos is the only one processing the Sloth-generated rules, I suggest you change the SLO name, this will change the ID and it will be a new one that will not collide with the previous one.

e.g: http-availability-sloth-sample to http-availability-sloth-sample-v2, like a new iteration of the SLO.

Regarding Thanos and prometheus-operator, I don't use Thanos myself, however, I guess you already know/seen that the K8s labels you use on Sloth's CR, will be set on the generated prometheus-operator CR rules, so, having different rule selectors for the Prometheus RuleSelector and Thanos RuleSelector rules should be enough.

Hope this helps!

from sloth.

Thanks for the response and that helps a ton! I somehow missed the ruleSelector in the prometheus spec. That coupled with the v2 change worked wonderfully.

Thanks again!

from sloth.