Helm: missing support for spread-minizing token generation about mimir HOT 4 CLOSED

Adding

ingester:
  extraArgs:
    ingester.ring.tokens-file-path: ""
    ingester.ring.token-generation-strategy: spread-minimizing
    ingester.ring.spread-minimizing-zones: 1a,1b,1c
    ingester.ring.spread-minimizing-join-ring-in-order: true

Has done the trick for us!

from mimir.

reply from @davidkohr:

My guess is that the issue is when you enable zone awareness for ingesters, it does this by injecting this runtime config to each ingester:

# ingesters in zone-a:
    - args:
        - '-ingester.ring.instance-availability-zone=zone-a'

# ingesters in zone-b:
    - args:
        - '-ingester.ring.instance-availability-zone=zone-b'

Because this is being dynamically handled by the chart, you aren't the one configuring this in structuredConfig. Because of this, all of your Mimir components, at least upon init, think that the value of ingester.ring.instance-availability-zone is empty, hence the error message zone is not valid (notice the 2 spaces after zone).

Again, this is my guess, and I would love someone to verify this.

To get around this, we had to use Kustomize to inject runtime configs to each zone to enable spread-min token generation. This feels very hacky, so I'm looking forward to this being fixed in the chart, unless there's already a better way.

from mimir.

this looks reasonable @davidkohr. This looks more like a bug in the distributor code than a bug in the helm chart. The distributor shouldn't try to validate the ingester config.

A workaround would be to set the four config options to enable spread-minimizing tokens as CLI flags to the ingester statefulset: ingester.extraArgs in the values.yaml. This way the distributor wouldn't see this config and wouldn't try to validate the ingester config.

from mimir.

duplicate of #7822

from mimir.