[BUG] Extra samples added when Cloudwatch metric stops about yet-another-cloudwatch-exporter HOT 5 OPEN

👋 @qcattez, the behavior you are seeing is due to the usage of nilToZero: true in your configuration. Removing it or explicitly setting it to false (the default is false) should fix your issue.

What's happening is that CloudWatch continues to export resources in ListMetrics for a variable length of time even when they have no data points. When the exporter calls GetMetricData and no data point is found in the time window nilToZero dictates if a zero should be produced or a nil. Producing zero gives you continuity in your graph panels while nil leaves gaps much like cloudwatch.

from yet-another-cloudwatch-exporter.

Hi @kgeckhart 👋

Thanks for helping out ;)

I tried setting nilToZero: false but it doesn't change the behavior I mentionned : there is still an extra sample with the same value (not zero).
Furthermore, as we can see in the screenshots and graphs, even if I had nilToZero: true, I didn't get a continuity in my graph :/

Do you have another idea ?

from yet-another-cloudwatch-exporter.

Just commenting to get some help on this 🙏

from yet-another-cloudwatch-exporter.

You say you want continuity in your graphs and for that to happen you need consistent sampling or prometheus will mark the value as stale. Nils can be connected through settings if you are using grafana. Continuity cannot be achieved if the sample is dropped which is more important?

from yet-another-cloudwatch-exporter.

I think there's a misunderstanding : my problem is about a wrong extra sample, not about continuity.
I annotated the screenshots to be more explicit.

Here are the samples from Cloudwatch :

And here are the samples retrieved by YACE and presented in Grafana :

We see that the data is wrong with a sample that shouldn't exist.
nilToZero has no impact on this behavior.

Can someone reproduce it ?
Is it expected looking at my configuration ?

from yet-another-cloudwatch-exporter.