Comments (4)
You can setup a Pod Disruption Budget (PDB) on your pods which will ensure that a configured number are always running. If a spot interruption notice is received, aws-node-termination-handler will attempt to cordon and drain the node but the PDB could stall the evictions if evicting would cause the pod count to drop below your configured threshold. But keep in mind that if the 2-minute window ends, your pods may be forced to be shutdown ungracefully since the instance itself will be automatically shutdown.
You can read more about Pod Disruption Budgets here: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
from amazon-ec2-metadata-mock.
Thank you for the explanation, I just set the PDB and indeed its not allowing the eviction on the pods.
However the cluster-autoscaler doesn't receive an event to scale the nodes and move the pods to the new one. I know this is question not related to here but since we are talking what would be the best approach to handle that?
from amazon-ec2-metadata-mock.
@roeP the PDB is only going to prevent the eviction, you would need another mechanism to trigger a scale up since cluster-autoscaler will only scale up based on the "pending pods" metric. You can over-provision your cluster to achieve no downtime when a node disappears. That is probably your best bet right now. Over provisioning involves scheduling a configurable amount of headroom via low priority pause-pods. This will ensure that you have space available for pods on another node.
When you receive a spot interruption notice, the aws-node-termination-handler will try to evict and then your pods will get rescheduled on-top of the pause-pods that were over provisioned for you.
NOTE: the following functionality does not exist today and probably doesn't make sense to add:
The aws-node-termination-handler could have functionality added to look at the pods it is evicting and scale up the number of pods on the node that is being drained after it cordons the node. I'm not sure this would be very helpful though. If a node has not already been booted, it's unlikely to have a node be launched and the pods rescheduled within the 2 minute window, so the better solution is still over-provisioning the cluster.
Hope that helps
More info on cluster-overprovisioner:
- https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-configure-overprovisioning-with-cluster-autoscaler
- https://medium.com/scout24-engineering/cluster-overprovisiong-in-kubernetes-79433cb3ed0e
from amazon-ec2-metadata-mock.
Thank you. That was helpful. Will try it
from amazon-ec2-metadata-mock.
Related Issues (20)
- standard_init_linux.go:228: exec user process caused: exec format error HOT 3
- Deploy Helm Chart HOT 3
- Add `tags/instance` path HOT 2
- Emulate specifying `PUT` response hop limit HOT 1
- /latest/user-data ? HOT 2
- Conform the real IMDS by not listing dirs
- Dump all metadata to a file HOT 3
- Userdata formatting HOT 1
- Userdata in getHandlerPairs()? HOT 1
- marketplaceProductCodes field is typed as a string instead of a list of strings HOT 1
- Please document how to serve real credentials in the mocked data HOT 11
- Passing null value in AEMM config parameter HOT 6
- Optionally support the removal of instances tags within returned metadata HOT 2
- [bug] cannot add a new tag using config overrides HOT 3
- Provide this mock as a Java library HOT 2
- [feature request] Watch config file for changes and reload HOT 2
- Programmatic use in Go tests HOT 2
- Incorrect Content-Type response header HOT 4
- IAM credentials should have realistic values HOT 2
- Missing EC2ws header HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazon-ec2-metadata-mock.