Cortex 1.16.0 Upgrade Error:LabelValues() from merge generic querier for label about cortex HOT 8 OPEN

Looks like the error is coming from https://github.com/cortexproject/cortex/blob/master/vendor/github.com/prometheus/prometheus/storage/merge.go#L165

from cortex.

Context canceled are either timeout or context canceled. If it is requests being canceled then it should be fine. If it is timeout then probably you need to check your service.

caller=push.go:53 level=warn org_id=fake msg="push refused" err="rpc error: code = Code(400) desc = maxFailure (quorum) on a given error family, rpc error: code = Code(400) desc = addr=redact:9095 state=ACTIVE zone=, rpc error: code = Code(400) desc = user=fake: err: duplicate sample for timestamp

maxFailure (quorum) is also expected.

from cortex.

@yeya24 we haven't been able to figure out where the requests are coming from because when we look at the querier and ruler we don't see timeouts to correlate the error above to. Are there other places we should look to track it down? W

We decided to test rolling back and the errors stopped once we rolled back to v1.15.3

from cortex.

Are you seeing some impact?

@yeya24 could this be because we fixed some context cancelletion propagation and now we are actually canceling the queries that times out? (if so this is actually a good thing?)

I think to prove that this may be the case, could you keep the ingester on 1.16 and all other query components on 1.15.3 (queriers, querry-frontend and -if used - query scheduler?)

from cortex.

Hey @alanprot I was able to bump the ingesters up to v1.16.0 but we still see these in the logs:
ts=2023-12-15T14:26:27.753624641Z caller=grpc_logging.go:43 level=warn method=/cortex.Ingester/LabelValues duration=1.341282224s err="LabelValues() from merge generic querier for label instance: block: 01HredactR11R: fetching postings for matchers: context canceled" msg=gRPC ts=2023-12-15T14:26:39.726258409Z caller=grpc_logging.go:43 level=warn method=/cortex.Ingester/LabelValues duration=47.266176ms err="LabelValues() from merge generic querier for label type: block: 01HHredactV72C: fetching postings for matchers: context canceled" msg=gRPC ts=2023-12-15T14:26:55.57439212Z caller=grpc_logging.go:64 level=warn method=/cortex.Ingester/QueryStream duration=154.802µs err="block: 01HHredactP6B: get postings offset entry: context canceled" msg=gRPC ts=2023-12-15T14:27:12.000973732Z caller=grpc_logging.go:43 level=warn method=/cortex.Ingester/LabelValues duration=44.081753ms err="LabelValues() from merge generic querier for label type: block: 01HHredactV72C: fetching postings for matchers: context canceled" msg=gRPC ts=2023-12-15T14:27:12.068824183Z caller=grpc_logging.go:43 level=warn method=/cortex.Ingester/LabelValues duration=4.919062ms err="LabelValues() from merge generic querier for label type: block: 01HHPredactC09V72C: fetching postings for matchers: context canceled" msg=gRPC ts=2023-12-15T14:27:12.160218629Z caller=grpc_logging.go:43 level=warn method=/cortex.Ingester/LabelValues duration=16.558308ms err="LabelValues() from merge generic querier for label type: block: 01Hredact9V72C: fetching values of label type: context canceled" msg=gRPC ts=2023-12-15T14:27:12.33095397Z caller=grpc_logging.go:43 level=warn method=/cortex.Ingester/LabelValues duration=96.702µs err="LabelValues() from merge generic querier for label instance_name: fetching postings for matchers: context canceled" msg=gRPC

It seems like it continuously repeats. We haven't seen impact or have heard from users regarding any impact. Its just new log errors we haven't seen before and don't want to move to production in case there is some impact that we don't understand.

from cortex.