Comments (6)
Wondering if there is a reason for using a deprecated AMI?
from karpenter-provider-aws.
Wondering if there is a reason for using a deprecated AMI?
When an AMI is deprecated, it affects the entire node pool. So any scale up/scheduling events for pods are stuck when Karpenter can't find the AMI.
We have to pin AMI's using ids due to issues for example and also to be able to use drift so Karpenter doesn't always get unvetted amis to production workloads. By default existing ASGs and LaunchTemplates are able to detect deprecated AMIs and continue functioning without affecting workloads but Karpenter fails in this scenario.
Edit:
To provide more context, let's assume there are 5 different node pools per EKS cluster across 10 clusters, each in multiple AWS accounts. Keeping track of AMI deprecations and updating every AMI for each node pool and architecture to ensure workloads are uninterrupted can become a significant operational burden. Existing node pools should continue to function and avoid causing pod scheduling failures, even if a node pool declares an AMIs that has been deprecated.
from karpenter-provider-aws.
By default existing ASGs and LaunchTemplates are able to detect deprecated AMIs and continue functioning without affecting workloads but Karpenter fails in this scenario
What about prioritizing the non-deprecated AMIs in front of the deprecated ones? By nature of what the word "deprecated" means, this seems like the right behavior to me -- discover them but just don't take them if you non-deprecated ones available
from karpenter-provider-aws.
We have to pin AMI's using ids due to issues for example and also to be able to use drift so Karpenter doesn't always get unvetted amis to production workloads
I assume that a deprecation for your organization happens without a lot of coordination if the rug can get pulled out from under you like it did here. How do you expect to orchestrate the upgrade after the AMI has been deprecated? e.g. Something like continuing to launch instances on the deprecated AMI until you change the ID term to get to a non-deprecated one.
from karpenter-provider-aws.
I assume that a deprecation for your organization happens without a lot of coordination if the rug can get pulled out from under you like it did here
Not really there are set time periods for this to happen for different AMIs across different architectures. For karpenter to be able to detect deprecated AMIs means that the workloads are unaffected and continue functioning for whenever there is a burst in traffic and the HPA kicks in for example.
How do you expect to orchestrate the upgrade after the AMI has been deprecated?
Ideally, yes like you suggested. Vet the AMIs in non production environment and promote them to production by updating the ec2nodeclasses. With ASGs and cluster auto scaler this is how it would work, where the ASG would scale up the nodes and not affect the existing workloads and I understand that Karpenter != ASG but having somewhat of the similar functionality would help where it doesn't become a race against time to upkeep AMIs along with the EKS cluster upgrades :)
I am willing to work on this, if you and the community think this is something that can be included in Karpenter.
from karpenter-provider-aws.
What about prioritizing the non-deprecated AMIs in front of the deprecated ones? By nature of what the word "deprecated" means, this seems like the right behavior to me -- discover them but just don't take them if you non-deprecated ones available
Thats one option but Karpenter currently cannot even discover AMIs that have been deprecated. Another probable solution would be to have a field includeDeprecated
on the amiSelectorTerms
when an id
is specified - which by default can be set to false.
Edit: somewhat similar to how the terraform ami datasource does it https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami#include_deprecated
from karpenter-provider-aws.
Related Issues (20)
- Unable to schedule Pods on g6e instances
- Wrong url for getting-started-with-karpenter/cloudformation.yaml for v.1.0.0 HOT 1
- Karpenter overwriting supplied Ec2NodeClass settings?! HOT 7
- Karpenter to handle hard failure of underlying EC2 host
- Karpenter ami replacement HOT 1
- Add metrics for the status of NodeClaims? HOT 4
- Will not disrupt bad node: Cannot disrupt NodeClaim: state node isn't initialized HOT 4
- End of Life Policy for Karpenter
- Only generation 5 for R instance families ? HOT 1
- Clarify v1 migration HOT 1
- The Discription of DisruptionBudge with non-percentage value is incorrect
- How can I set the SpotPrice?
- Conversion webhook hardcoded at kube-system namespace HOT 8
- The bitnami/kubectl image at public.ecr.aws is only having amd64 architecture HOT 1
- Service account cannot list resource "mutatingwebhookconfigurations" after installing in `karpenter` namespace
- error - missing required startup taint, karpenter.sh/unregistered HOT 1
- Karpenter metrics not working in Datadog after v1.0.0 migration. HOT 1
- digest value does not exist within aws public ecr registry
- Error from server: failed to prune fields: failed add back owned items: failed to convert pruned object at version karpenter.sh/v1: HOT 1
- Support ENI Idle connection tracking timeout settings
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from karpenter-provider-aws.