Query: Network bandwidth usage upward of 500MB/s between Querier and configured stores about thanos HOT 17 OPEN

How many blocks do you have? Can you bump to 0.35.0?

from thanos.

How many blocks do you have? Can you bump to 0.35.0?

What do you mean by how many blocks?

Bumped thanos versions to 0.35.1, redeployed with all the stores set. The red line that is spiking in the graphs is caused by the EKS node which has storegateway on it:

from thanos.

I guess it's fetching block metadata on startup ( the gateway) does it stabilize eventually and is the querier less noisy?

from thanos.

It stays that high indefinitely. The fact that it "stabilizes" is not really a good thing when it stays at 500+MB/s bandwidth 😅

from thanos.

Storage gw is also 0.35.0 right?

from thanos.

Storage gw is also 0.35.0 right?

Yeah all thanos components are now 0.35.1

from thanos.

How many blocks do you have in object storage roughly? Is your compactor working well?

from thanos.

Compactor seems to be doing it's job quite well and there are less than 800 objects reported totaling little under 30GB of data.

from thanos.

@sourcehawk traffic between querier and store gateway is triggered by incoming queries. We can't say 500 MB/s is unnatural unless we know a bunch of things, like:

• Is this querier getting lots of queries as soon as it comes up?
• Are these queries over long time frames?
• Are these queries touching many series?
• What's the size of your blocks?

from thanos.

After much debugging we've come to realize the traffic is probably being generated due to an infinite recursion loop happening between the thanos ruler and the querier when the ruler is added as a store on the querier. My best bet is because the ruler queries the querier but the querier also queries the ruler, causing an infinite call loop to both the sidecar and the store gateway.

This is the network traffic after removing thanos ruler, as can be seen, the traffic of all types drops almost instantaneously to zero.

from thanos.

Interesting. You can deploy a separate querier that will query almost everything, excluding the Ruler, and point the Ruler to this one.

from thanos.

Just out of curiosity - if you'd enable remote_write on Ruler that would stop Store API on it - possibly that could help?

from thanos.