promtool complains about duplicate rules about sloth HOT 7 CLOSED

Hi! I have "properly" fixed this by manually adding the apparently missing labels for category, sloth_id, sloth_service and sloth_slo.
A "properly" generated recording rule looks like this:

record: slo:sli_error:ratio_rate3d
expr: |
some expression
labels:
category: latency
sloth_id: api-lb-requests-latency-rest
sloth_service: api-lb
sloth_slo: requests-latency-rest
sloth_window: 3d

For some reason, for the 30d/4w time window (the largest time window corresponding to the default-slo-period) these labels are not being generated. And this happens consistently like so:

record: slo:sli_error:ratio_rate4w
expr: |
some expression
labels:
sloth_window: 4w

Without looking at the code, my guess would be that label generation is handled differently for a recording rule corresponding to the default slo-period, be that 30 days or 4 weeks.

IMHO this is worth fixing. If one uses more than one file containing recording rules or a large number of recording rules the output can easily get cluttered due to the verbose output and spotting a true issue can become cumbersome.

from sloth.

Hello there!

Seeing that now the promtool warnings are no longer warnings, I did some tests in Prom, and seems that doesn't complain anymore at runtime about labels being replaced on the same recording rule (the ones calculated by the rule, are replaced by the ones set on the rule configuration). I've merged a PR that ads the labels on the one type of SLI rule that was missing (the optimized one).

Best,

from sloth.

Hi @kbudde!

In this case, is telling you that depending on the query executed by recording rule, you could have a conflict (maybe not). The thing is that Prometheus, at runtime (while executing the recording rule), will tell you if it is really a problem or not, at a static validation level, it can't be sure (that's why the warning).

The resulting recording rule will not collide (Sloth already handles this) because it will have unique labels (sloth_id and sloth_slo) those labels are inherited from the query (as you can see it discards sloth_window but not the other labels).

If you are having duplicated rules at runtime, is very likely that is not because of the generated rules and is more related with 2 Sloth instances running at the same time handling the same SLOs or similar stuff.

Hope this helps! If you have collision at runtime, please write in this same issue :)

best,

from sloth.

Hi @slok,
it's clear to me that the linter is not pointing to a real issue in this case as it is already implicitly by sloth.

The question is would you accept a PR which adds the the labels (sloth_id, sloth_slo) explicitly to generated recording rules?
The generated rule would be the same; just the definition would be more verbose to satisfy the linter.

from sloth.

I can verify the same behaviour, the 30d recording doesn't inherit the labels.

from sloth.

Worth noting that duplicate rools reported by promtool now failf by default in Prometheus 2.35.0:

prometheus/prometheus#10435

from sloth.

Thanks for correcting this!

from sloth.