Observed unusual scaling (up/down) activity in Karpenter even when pods are not changes about karpenter-provider-aws HOT 4 CLOSED

Since Karpenter is metrics based, the scaling behavior is all about the metric. The easiest way to debug is to observe the metric is to go look at prometheus and search karpenter_ and your metric should show up.

There's also a 5 minute scale down delay window, which should prevent the "flapping" behavior you're talking about.

With respect to "spreading", Karpenter isn't a scheduler, so it has no control over whether or not the pods get scheduled to the same node. All it's doing is looking at how many resources are requested by pods that were assigned to the node (by the scheduler) and then scaling up once it passes a threshhold.

Can you share your configs?

from karpenter-provider-aws.

Reading a bit more into your custom scheduler. Does it set pod.spec.nodeName at some point? Assuming you're using reserved capacity, we rely on that for making calculations: https://github.com/awslabs/karpenter/blob/main/pkg/metrics/producers/reservedcapacity/producer.go#L51

from karpenter-provider-aws.

Thanks for the help. This is due to a minor bug in the config files. After fixing it, Karpenter is able to scale fine. We can close the issue.

from karpenter-provider-aws.

This is fixed after fixing a bug in the application config files

from karpenter-provider-aws.