Receiving failed to enqueue request 500s about mimir HOT 2 OPEN

Hi, to add a bit of context, when upgrading from Mimir 2.10 to Mimir 2.12 we started to see increased latency and small error rate on the read path. At the same time we noticed the number of tcp connections to query-scheduler went from stable to going up and down.

This seems to have been caused by this change https://github.com/grafana/mimir/pull/7269/files#diff-7fd5824797e825650064e35cfdea31cf25162114e24bc754f648de77cff4ff06L53
removing extra args from the query-scheduler.

- "-server.grpc.keepalive.max-connection-age=2562047h" # 100000 days, effectively infinity
- "-server.grpc.keepalive.max-connection-age-grace=2562047h" # 100000 days, effectively infinity

Which were previously added as part of https://github.com/grafana/mimir/pull/3262/files.

Looking at sample traces where the requests ended with http status code 500 seems like retries were exhausted before new connection was established.

Another example would be when queries just take roughly 1s to enqueue, succeeding but increasing latency even on light queries.

As mention on slack in this thread https://grafana.slack.com/archives/C039863E8P7/p1715625953274669?thread_ts=1714333917.446309&cid=C039863E8P7, adding this args back to query-scheduler seems to fix / minimize the issue.

- "-server.grpc.keepalive.max-connection-age=2562047h" # 100000 days, effectively infinity
- "-server.grpc.keepalive.max-connection-age-grace=2562047h" # 100000 days, effectively infinity

fyi @dimitarvdimitrov

We made the same change today for now it seems to work, but will update tomorrow / the day after if the issue truly went away for us too.

from mimir.

I can confirm now after having the change deployed in production for a few days it fully fixed the issue for us.

from mimir.