[query] Difference between direct Prom and query graph (~staleness) about thanos HOT 4 CLOSED

Mh, so they look largely the same. Do you mean the difference where Thanos shows the data flatlining across the gaps rather than actually showing no data?
That would smell like staleness handling, but that's all in imported packages – so not sure where any divergence would come from.

from thanos.

Coming back to this, it seems that somehow the staleness samples are not being treated properly. We use the default PromQL engine, so that cannot be it.
Staleness samples are just regular samples with a float64 values. However, there are ~2^52 ways to represent NaN. We picked one discrete one of them to indicate staleness and normalize actually scraped NaN values to another one.

By our research, this is valid usage according to IEEE754:

"Two kinds of NaN: a quiet NaN (qNaN) and a signaling NaN (sNaN). A NaN may carry a payload that is intended for diagnostic information indicating the source of the NaN. The sign of a NaN has no meaning, but it may be predictable in some circumstances."

I wonder now whether somewhere along the way another NaN representation is substituted for our staleness samples for some reason. It would be easy to blame gRPC or something, but we are actually moving around encoded chunks there.

from thanos.

@Bplotka can you check whether this is still an issue.

Did a quick check against GCS data. The metric (green line) stops but increases until the end, i.e. does not show the 5 minute flatline at the end. That generally looks correct.

from thanos.

We haven't seen this in a while so I'd consider it resolved. We can reopen should someone encounter it again.

from thanos.