bug: deprecated AMIs should be able to be listed when specifying ami id on the amiSelectorTerms about karpenter-provider-aws HOT 6 OPEN

Wondering if there is a reason for using a deprecated AMI?

from karpenter-provider-aws.

Wondering if there is a reason for using a deprecated AMI?

When an AMI is deprecated, it affects the entire node pool. So any scale up/scheduling events for pods are stuck when Karpenter can't find the AMI.

We have to pin AMI's using ids due to issues for example and also to be able to use drift so Karpenter doesn't always get unvetted amis to production workloads. By default existing ASGs and LaunchTemplates are able to detect deprecated AMIs and continue functioning without affecting workloads but Karpenter fails in this scenario.

Edit:

To provide more context, let's assume there are 5 different node pools per EKS cluster across 10 clusters, each in multiple AWS accounts. Keeping track of AMI deprecations and updating every AMI for each node pool and architecture to ensure workloads are uninterrupted can become a significant operational burden. Existing node pools should continue to function and avoid causing pod scheduling failures, even if a node pool declares an AMIs that has been deprecated.

from karpenter-provider-aws.

By default existing ASGs and LaunchTemplates are able to detect deprecated AMIs and continue functioning without affecting workloads but Karpenter fails in this scenario

What about prioritizing the non-deprecated AMIs in front of the deprecated ones? By nature of what the word "deprecated" means, this seems like the right behavior to me -- discover them but just don't take them if you non-deprecated ones available

from karpenter-provider-aws.

We have to pin AMI's using ids due to issues for example and also to be able to use drift so Karpenter doesn't always get unvetted amis to production workloads

I assume that a deprecation for your organization happens without a lot of coordination if the rug can get pulled out from under you like it did here. How do you expect to orchestrate the upgrade after the AMI has been deprecated? e.g. Something like continuing to launch instances on the deprecated AMI until you change the ID term to get to a non-deprecated one.

from karpenter-provider-aws.

I assume that a deprecation for your organization happens without a lot of coordination if the rug can get pulled out from under you like it did here

Not really there are set time periods for this to happen for different AMIs across different architectures. For karpenter to be able to detect deprecated AMIs means that the workloads are unaffected and continue functioning for whenever there is a burst in traffic and the HPA kicks in for example.

How do you expect to orchestrate the upgrade after the AMI has been deprecated?

Ideally, yes like you suggested. Vet the AMIs in non production environment and promote them to production by updating the ec2nodeclasses. With ASGs and cluster auto scaler this is how it would work, where the ASG would scale up the nodes and not affect the existing workloads and I understand that Karpenter != ASG but having somewhat of the similar functionality would help where it doesn't become a race against time to upkeep AMIs along with the EKS cluster upgrades :)

I am willing to work on this, if you and the community think this is something that can be included in Karpenter.

from karpenter-provider-aws.

What about prioritizing the non-deprecated AMIs in front of the deprecated ones? By nature of what the word "deprecated" means, this seems like the right behavior to me -- discover them but just don't take them if you non-deprecated ones available

Thats one option but Karpenter currently cannot even discover AMIs that have been deprecated. Another probable solution would be to have a field includeDeprecated on the amiSelectorTerms when an id is specified - which by default can be set to false.

Edit: somewhat similar to how the terraform ami datasource does it https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami#include_deprecated

from karpenter-provider-aws.