Comments (5)
While this issue can be easily reproduced with node reboot, I believe it can also happen if device plugin suddenly becomes unhealthy after pod is scheduled and before kubelet allocates the resources.
The expected behavior is that the failure should be retried just like CNI failures.
from kubernetes.
/sig node
from kubernetes.
I can confirm this issue. Luckily for us the pods are managed by Deployments and they are restarted. But we're still stuck with 'ghost' pods in an UnexpectedAdmissionError
state.
from kubernetes.
What happened?
When a node is rebooted, pod using resources allocated by device plugin will encounter UnexpectedAdmissionError error as below:
Warning UnexpectedAdmissionError 84s kubelet Allocate failed due to no healthy devices present; cannot allocate unhealthy devices xxx, which is unexpected
Hi! This behavior wants to make evident and recoverable a previously hidden breakage when pod actually started, but noone allocated the device they requested: #109595
I don't know the mechanics of that specific GPU device plugin, but it seems likely to me that this problem can totally happen also for gpu devices. IOW, from what I gathered it was not device-specific but rather a flaw in how the devices are handled in kubelet, hence the flag.
What makes it really bad is if it's a raw pod, it stucks in such state and never recover.
This behavior can be surprising indeed, but when pod reach terminal state the system doesn't try to recover them, this is why the platform introduce and recommends higher level controllers like deployment
What did you expect to happen?
The pod should be retried until device plugin is ready
Yes, this is something we've being discussed since a while but there's unfortunately nothing concrete yet :\
from kubernetes.
/triage accepted
kubelet retries are being discussed since a while and there's general agreement it's a desirable behavior
from kubernetes.
Related Issues (20)
- PodDeletionCost occasionally doesn't work HOT 9
- [Failing Test] CSI Volumes .. should preempt lower priority pods using ReadWriteOncePod volumes fails when Beta Feature gates are enabled HOT 3
- Why doesn't the device plugin pass the containerID parameter HOT 13
- flaky test: VolumeAttributesClass e2e HOT 2
- [Failing Test] Test DefaultProcMount - `will mask proc mounts by default` is failing HOT 3
- CVE-2024-5321: Incorrect permissions on Windows containers logs HOT 1
- Intermittent error on new Nodes: "Unable to locate credentials" HOT 4
- Error during pulling of v.1.30.3 images HOT 20
- ec2-eks jobs are failing to create the cluster successfully HOT 6
- API Server fails validation for CRDs embedding resources with optional fields without omitempty when there is webhook HOT 2
- [failing-test] ProcMount Specific SecurityContext set fails HOT 2
- kubelet may provide a warning event or log when pod mount a wrong subpath from configmap HOT 4
- Taint Tolerance Tied to Pod Disruption Budget Availability HOT 3
- [Flaking Test] [sig-node] [NodeFeature:NodeProblemDetector] should run without error HOT 3
- [FG:InPlacePodVerticalScaling] Minimum CPU request is displayed when only memory request is configured HOT 2
- integration tests are painfully slow HOT 23
- WatchList failed when Accept content-type is Table HOT 5
- Need assistance troubleshooting CoreDNS stuck in "ContainerCreating" state HOT 7
- CVE-2024-5037 and CVE-2023-44487 not listed in the official feed? HOT 6
- Add repobeats to the readme HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubernetes.