Comments (11)
What version of the Spin shim are you using? Could you gather containerd logs related to the pods that are stuck in container creating?
from enterprise-wasm.
@devigned @Mossaka I hope this journalctl -u kubelet
of the WASM-shimed AKS node helps: aks-kubelet.txt
from enterprise-wasm.
related issue : containerd/runwasi#418
from enterprise-wasm.
additionally, when I try to scale
$ kubectl scale deployment distributor --replicas 3
pods get stuck in ContainerCreating
NAME READY STATUS RESTARTS AGE
distributor-6765d649b5-77444 0/1 ContainerCreating 0 3m15s
distributor-6765d649b5-g6k8w 1/1 Running 0 115m
distributor-6765d649b5-m4mlr 0/1 ContainerCreating 0 3m15s
distributor-dapr-x4spd 1/1 Running 0 115m
events e.g. for first pod:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 81s default-scheduler Successfully assigned default/distributor-6765d649b5-77444 to aks-npspin-84373721-vmss000000
from enterprise-wasm.
(side note: you can refer to this issue for accessing containerd logs in your cluster deislabs/containerd-wasm-shims#197)
from enterprise-wasm.
What version of the Spin shim are you using? Could you gather containerd logs related to the pods that are stuck in container creating?
I am currently using kwasm-node-installer
main branch and that copies Spin : /deislabs/containerd-wasm-shims/releases/download/v0.10.0/containerd-wasm-shims-v2-spin-linux-$(uname -m).tar.gz
I think I found the kubelet / containerd logs on the AKS node. I will redeploy with single pods to get rid of noice in the logs to make analysis easier.
from enterprise-wasm.
It would be great to see containerd logs
from enterprise-wasm.
It would be great to see containerd logs
All of the details about the shim starting and possibly failing will be contained at the containerd level. The Kubelet is a little too high level to provide the detail required to diagnose the issue.
from enterprise-wasm.
It would be great to see containerd logs
All of the details about the shim starting and possibly failing will be contained at the containerd level. The Kubelet is a little too high level to provide the detail required to diagnose the issue.
Do you have any pointers for me where I could find containerd logs on an AKS node?
from enterprise-wasm.
Can you try journalctl -u containerd
?
from enterprise-wasm.
thanks @Mossaka - that worked ... but + @devigned - since I switched to kwasm-node-installer
this behavior seems to be gone
$ k scale deployment --replicas 8 distributor
deployment.apps/distributor scaled
$ k get pod
NAME READY STATUS RESTARTS AGE
distributor-54576c4fd5-9srhk 1/1 Running 0 14m
distributor-54576c4fd5-crpfn 1/1 Running 0 14m
distributor-54576c4fd5-fx86x 1/1 Running 0 19m
distributor-54576c4fd5-hjcgk 1/1 Running 0 14m
distributor-54576c4fd5-hn52r 1/1 Running 0 19m
distributor-54576c4fd5-jz42v 1/1 Running 0 14m
distributor-54576c4fd5-ldqs9 1/1 Terminating 0 14m
distributor-54576c4fd5-vxddt 1/1 Running 0 19m
distributor-54576c4fd5-zbwh5 1/1 Running 0 14m
distributor-dapr-5cd8c7cb9b-9w8mc 1/1 Running 0 19m
distributor-dapr-5cd8c7cb9b-rwkk6 1/1 Running 0 14m
distributor-dapr-5cd8c7cb9b-xgdgf 1/1 Running 1 (19m ago) 19m
kwasm-debug-bdq4l 1/1 Running 0 5m54s
kwasm-initializer-9sr5q 1/1 Running 0 22m
... after a few seconds ...
$ k get pod
NAME READY STATUS RESTARTS AGE
distributor-54576c4fd5-9srhk 1/1 Running 0 15m
distributor-54576c4fd5-crpfn 1/1 Running 0 15m
distributor-54576c4fd5-fx86x 1/1 Running 0 20m
distributor-54576c4fd5-hjcgk 1/1 Running 0 15m
distributor-54576c4fd5-hn52r 1/1 Running 0 20m
distributor-54576c4fd5-jz42v 1/1 Running 0 15m
distributor-54576c4fd5-vxddt 1/1 Running 0 20m
distributor-54576c4fd5-zbwh5 1/1 Running 0 15m
distributor-dapr-5cd8c7cb9b-9w8mc 1/1 Running 0 20m
distributor-dapr-5cd8c7cb9b-rwkk6 1/1 Running 0 15m
distributor-dapr-5cd8c7cb9b-xgdgf 1/1 Running 1 (20m ago) 20m
kwasm-debug-bdq4l 1/1 Running 0 7m8s
kwasm-initializer-9sr5q 1/1 Running 0 23m
from enterprise-wasm.
Related Issues (6)
- Dapr>Spin>Dapr integration take 15-30sec for some requests HOT 2
- Dapr error error invoking app when invoking Spin app HOT 1
- InternalError: runtime-error (error 4) when accessing http://distributor-dapr/v1.0/metadata HOT 1
- needed to do a manual npm install in helpers/orderdata-ts
- Dapr pods do not initialize after re-deployment HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from enterprise-wasm.