Rejecting valid exemplars with err-mimir-exemplar-labels-missing about mimir HOT 8 OPEN

The Prometheus TSDB code drops exemplars with no labels which is why Mimir rejects them on ingest. It wasn't a problem in practice until now because Prometheus never sends exemplars with no labels.

from mimir.

Discussion here when some of the logic to reject empty exemplars was added: #873

from mimir.

@gouthamve since this behaviour is by design in Prometheus, can the issue be closed? Or do you think it should be reconsidered in Prom?

from mimir.

I think it needs to be reconsidered in Prometheus. We should atleast not drop them with 400s because OTel seems produce a ton of these empty exemplars.

When that happens, the customers alerts start firing because their writes are failing, and the logs are full of the err-mimir-exemplar-labels-missing error message. This is a sub-par experience for customers.

from mimir.

I'm curious, what's the use case for an exemplar without labels?

from mimir.

From an OTel and specification perspective, its "example values". For this histogram, these are the example values we recorded. @RichiH / @fstab can you chime in further?

from mimir.

👍 yes, I think it's just example values.

As it's defined in the OpenMetrics spec, and in the OpenTelemetry spec, we should support this. It's maybe ok to ignore the Exemplar itself, but dropping the entire time series is certainly not spec compliant.

from mimir.

I think that dropping the entire time series is a critical bug that deserves a separate issue.

I also think that we should not reject the empty exemplars just because Prometheus doesn't send those.

from mimir.