Grafana dashboard reflects inconsistent value where node memory utilization is different with exact usage in kube-state-metrics about karpenter-provider-aws HOT 3 CLOSED

Can you describe the exact metrics from kube-state that you are using for monitoring and expecting to match Karpenter's metrics?

from karpenter-provider-aws.

From the left side which is kube-state-metrics dashboard, it shows node (ip-10-4-105-63.ec2.internal, 16GB assigned) is using around 18.39% RAM Usage, around 3GB as Memory Usage.

From the right side, Node Summary node ip-10-4-105-63.ec2.internal is using 85.8% memory utilization which is not the same.

Grafana expression:
((karpenter_nodes_total_daemon_requests{resource_type="memory",arch="$arch",capacity_type="$capacity_type",instance_type="$instance_type",nodepool="$nodepool",zone="$zone"} or karpenter_nodes_allocatable0) + \n(karpenter_nodes_total_pod_requests{resource_type="memory",arch="$arch",capacity_type="$capacity_type",instance_type="$instance_type",nodepool="$nodepool",zone=~"$zone"} or karpenter_nodes_allocatable0)) / \nkarpenter_nodes_allocatable{resource_type="memory",arch="$arch",capacity_type="$capacity_type",instance_type="$instance_type",nodepool="$nodepool",zone="$zone", cluster="$cluster"}"

I tried the expression karpenter_nodes_total_daemon_requests + karpenter_nodes_total_pod_requests with proper filter, and it shows huge gap than kube-state-metrics usage. Not sure other have similar issue or not.

from karpenter-provider-aws.

I think I have wrong impression on the metrics...

This metrics is the total daemonset + pod requested memory inside karpenter node, and the metrics is not indiciating current memory usage.

Reference:

from karpenter-provider-aws.