Thanos query fanout has high latency when a strict-store is unavailable about thanos HOT 3 OPEN

Do you have this option enabled https://github.com/thanos-io/thanos/blob/main/cmd/thanos/query.go#L212? It should solve your issue.

from thanos.

Do you have this option enabled https://github.com/thanos-io/thanos/blob/main/cmd/thanos/query.go#L212? It should solve your issue.

Yes, i tried with store.response-timeout=15s and then 5s, but it still has similar elevated latency. I do have query.timeout=30s.

i notice in the code that store.response-timeout is a Timer for a kinda manual timeout while waiting for cl.Recv call 1. So if the cl.Recv call itself is taking longer than that amount (e.g. 40s) then the call would be that long as well. And the grpc context for cl.Recv doesnt have timeout specified AFAICT.

Ref:

from thanos.

I've dive through the code and I think I found what the problem is.

When a store is listed as store-strict, it will always be part of the active store set, as called out in this proposed (and completed) design 202001-thanos-query-health-handling.

However, since it's always be part of the active store set, they will always get incoming queries as long as the labels published by that store endpoint matches.

The problem is when that store is completely down like in my example above, the Dial's ClientConn would keep retrying to establish connection (in the background), on the application layer, grpc calls to that service is not aware of this fact and keep on innocently sending grpc requests with a regular timeout (dictated by Thanos' CLI arg query.timeout -- which is default to 2m, and in my case I've set it to 30s). Since there's no actual pod behind the store, it would get timed out instead of exceeding the store.response-timeout (see ref 1).

This is seems fine for individual queries since queries are timing out "as expected". However, when there are sizable number of queries that would go through this path (and timing out), it will fill up the "queue" from query-frontend, create a cascading failure scenario where other queries -- even though they don't go through this path (of hitting the strict store that's completely down) -- would also start timing out.

Proposed fix:

Since the healthiness of a store is actually being checked every 5s (see code ref 2), that means at query time, the query flow knows which store is unhealthy (in this case timeout). The original proposal (see ref 3) is to not ignore the strict store (and return partial response) instead of completely forgetting about the store (and return an illusional success response). We can still accommodate that idea but fail fast on unhealthy strict-store by checking whether the strict store is unhealthy and not even attempt to send gRPC request to it, and return an error.

If it SGTY I can prep a PR to introduce this fail-fast behavior.

Ref:

(1): the store.response-timeout only gets triggered if there was some thing sent back from the server and that's when the response-timeout is used to do early cut off if consecutive data received from the server is longer than that number:

(2): Check endpoint set every 5s

(3): Original proposal of store-strict: https://thanos.io/v0.29/proposals-done/202001-thanos-query-health-handling.md/

from thanos.