Count active metrics that match given query within each tenant about mimir HOT 8 CLOSED

I was also considering exporting the matcher itself as a label

I have mixed feelings. Looks an anti-pattern to me.

from mimir.

Can you use an exporter that just does a count count({foo="bar"}) and exposes it as a metric?

from mimir.

How can we do that for all tenants?

from mimir.

Plus, we would be potentially opening the TSDBs for tenants who don't have any active metrics, right?

from mimir.

Using an exporter would require a deployment per cluster right? And then the process itself would be querying for cluster data to then store it back as a metric. So we thought we might just generate the metric from service itself and save us the overhead of the exporter.

from mimir.

I'm not sure about the exporter. I see two issues:

1. We need to do it for every tenant. The exporter needs the updated list of tenants first and then, for each tenant, run the count query.
2. The query looks very expensive. We need to count active series (over the last 20m) and continuously running such query, for every tenant.

Because of this, I'm in favour of adding the support directly into Mimir.

Now going to review the proposal.

from mimir.

The proposal LGTM, but I've some small feedback. Thanks!

We need a gauge metric that provides the number of active series within each tenant that match a pre-configured query.

I would say not a "pre-configured query", but "label matchers". I know it's just wording, but with "query" we mean a different thing.

active_matching_series

I would suggest to call it active_series_custom_trackers because they're custom trackers for active series.

Same configuration can be provided by using multiple ingester.active-matching-series flags

We learned the hard way it's not easy to handle multiple flags (eg. you can't do it in jsonnet with our config setup, but you need to inject custom args in the container, which is annoying). I would avoid it. You could use a semicolon-separated string.

Name: "cortex_ingester_active_matching_series",

Based on my feedback above, I would call the metric cortex_ingester_active_series_custom_tracker and I would call the label name instead of matcher (since I guess we'll export the tracker name, not the label matchers).

from mimir.

Thanks, I already started implementing in #42, still pending tests and apply this feedback.

I was also considering exporting the matcher itself as a label, apart from the name, since it would provide context to the querier without adding extra cardinality to the metric, WDYT?
I.e.: cortex_ingester_active_series_custom_tracker{name="foobar", matcher="{foo=\"bar\"}"}

from mimir.