Querier "connection termination" to query-frontend with vanilla deployment of mimir-distributed helm chart about mimir HOT 4 CLOSED

ℹ️ I removed some of the complexity by allowing mimir to create the Minio instance as well. The issue still persists with the below configuration:

helm chart values:

ingester:
  replicas: 1
  zoneAwareReplication:
    enabled: false
store_gateway:
  zoneAwareReplication:
    enabled: false
querier:
  replicas: 1
  extraArgs:
    "log.level": "debug"
minio:
  enabled: true
alertmanager:
  enabled: false
ruler:
  enabled: false
vaultAgent:
  enabled: false
overrides_exporter:
  enabled: false
query_scheduler:
  enabled: false

and the helm chart:

resource "helm_release" "mimir" {
  count = var.uses_mimir ? 1 : 0

  depends_on = [kubernetes_namespace.o11y]
  name       = "mimir"
  namespace  = kubernetes_namespace.o11y.metadata[0].name
  repository = "https://grafana.github.io/helm-charts"
  chart      = "mimir-distributed"
  version    = "5.2.3"

  values = ["${local.grafana_mimir_template}"]

  set {
    name  = "minio.rootUser"
    value = kubernetes_secret.minio_creds.data["root-user"]
  }

  set {
    name  = "minio.rootPassword"
    value = var.minio_root_password
  }

No errors in other components that I can see, but constant erroring on the querier.

from mimir.

ts=2024-04-23T18:03:17.693470735Z 
caller=frontend_processor.go:91
level=error
msg="error processing requests"
address=100.64.3.196:9095
err="rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection termination"

The upstream connect error or disconnect/reset before headers doesn't look as something that comes from Mimir or any of the Mimir's own dependencies. In fact, after looking around, such an error looks very similar to the one mentioned in Envoy's source.

@SeanKilleen does your infrastructure happen to use something like Istio/Envoy or other service mesh tech? If yes, could istio/istio#50073 be the cause?

from mimir.

@narqo this is a really interesting theory. We do indeed use Istio (1.21.x too I think, need to cinfirm) and I don't think we happen to have other gRPC workloads in our cluster so that could be why we only saw it with this.

For expediency's sake we chose to move on from Mimir entirely; I just added another Prometheus instance that only accepts the metrics we care about via remote write and retains them for longer.

Is there any way to force this chart/setup to use http communication rather than gRPC? If so I can redeploy to confirm.

from mimir.

Great to know that stock Prometheus serves your use case for the moment 🎉

As for the particular issue, replacing Mimir's internal gRPC inter-service communication layer with something else is not what's supported. In theory, for an experiment, one could try deploying Mimir in the read-write mode, where querier and the query-frontend are deployed as one process. But this mode is still an experiment and isn't yet supported by the Helm chart.

I will close the issue for now, assuming it came, indeed, in the particular infrastructure. Let's revisit it later, should anyone bump into the same in the future.

from mimir.