Comments (4)
adrienjt
commented on July 1, 2024
Cilium's two DaemonSets have the broadest tolerations:
spec:
template:
spec:
tolerations:
- operator: Existswhich means "tolerate all taints", but they should not extend to virtual nodes.
The only way to restrict the toleration, i.e., to say "tolerate all taints except for one" is to add a node anti-affinity:
spec:
template:
spec:
tolerations:
- operator: Exists
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: virtual-kubelet.io/provider
operator: DoesNotExistPlease let me know if that works. This should be added to our readme.
from admiralty.
purplepalmdash
commented on July 1, 2024
Thanks. Added the nodeAffinity I could make multicluster-scheduler working together with cilium.
But another issue is coredns pods remains creating and stays at Pending status:
root@cilium-1:~# kubectl get pods --all-namespaces | grep -i coredns | grep -i running
kube-system coredns-77b5f79dbf-74zd9 1/1 Running 0 4m56s
kube-system coredns-77b5f79dbf-r89bl 1/1 Running 0 4m56s
root@cilium-1:~# kubectl get pods --all-namespaces | grep -i coredns | grep -i pending | wc -l
780
root@cilium-1:~# kubectl get rs -n kube-system
NAME DESIRED CURRENT READY AGE
cilium-etcd-operator-584788b99c 1 1 1 62m
cilium-operator-7cd598bdf6 1 1 1 62m
coredns-77b5f79dbf 782 782 2 11m
dns-autoscaler-85f898cd5c 1 1 1 75m
etcd-operator-59cf4cfb7c 1 1 1 18m
kubernetes-dashboard-857df7d6f7 1 1 1 75m
describe a Pending pod:
root@cilium-1:~# kubectl describe po coredns-77b5f79dbf-ztz9v -n kube-system
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m27s (x3 over 8m1s) default-scheduler 0/4 nodes are available: 2 node(s) didn't match node selector, 2 node(s) didn't match pod affinity/anti-affinity.
Adding nodeAffinity seems doesn't works for coredns:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: virtual-kubelet.io/provider
operator: DoesNotExist
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: node-role.kubernetes.io/master
operator: In
values:
- ""
weight: 100
from admiralty.
purplepalmdash
commented on July 1, 2024
By editing "max":4 to dns-autoscaler's configmap I could shrink the coredns pods to 4(2 pending, 2 running),
# kubectl edit configmap dns-autoscaler -n kube-system
apiVersion: v1
data:
linear: '{"coresPerReplica":256,"min":2,"max":4,"nodesPerReplica":16,"preventSinglePointFailure":true}'
kind: ConfigMap
metadata:
creationTimestamp: "2020-07-23T08:10:19Z"
name: dns-autoscaler
namespace: kube-system
resourceVersion: "52908"
selfLink: /api/v1/namespaces/kube-system/configmaps/dns-autoscaler
uid: 4914ed87-d754-425b-bd49-cbb44f2ea6e4
coredns pods:
# kubectl get pods --all-namespaces -o wide | grep -i coredns
kube-system coredns-77b5f79dbf-74zd9 1/1 Running 0 39m 10.233.64.204 mouse-1 <none> <none>
kube-system coredns-77b5f79dbf-d6vm8 0/1 Pending 0 39m <none> <none> <none> <none>
kube-system coredns-77b5f79dbf-m5bd4 0/1 Pending 0 39m <none> <none> <none> <none>
kube-system coredns-77b5f79dbf-r89bl 1/1 Running 0 39m 10.233.65.73 mouse-2 <none> <none>
dns-autoscaler auto-caculate the max coredns pod number to be 782. I think we should delete the dns-autoscaler when using multicluster-scheduler
from admiralty.
adrienjt
commented on July 1, 2024
Please refer to #43, fixed in v0.10. Indeed, you can either disable DNS autoscaling, or install multicluster-scheduler v0.10.0-rc.1 (which reminds me we should release v0.10.0 proper, since the release candidate has baked long enough).
from admiralty.
Related Issues (20)
- Question: What is the stable k8s version where I can install and test admiralty between 2 clusters? HOT 3
- docs/quickstart: kind image load fails on Apple M1 HOT 4
- Job Pods deleted on target clusters cause extra untracked Pods to be created HOT 2
- kubectl attach via proxy pods HOT 3
- reconciliation for PodChaperons HOT 3
- Default priority is different in 2 clusters, preventing the federated pods from starting
- Different ingress host or different config-map HOT 2
- node keep deleting HOT 1
- Admiralty virtual node uses deprecated label for load balancer exclusion HOT 2
- EKS tries to create system processes (kube-system namespace) incorrectly on Admiralty virtual nodes HOT 4
- Failed to Call Webhook HOT 1
- docs/quickstart: cert-manager webhook CrashLoopBackOff on Apple M1 HOT 1
- CSIStorageCapacity no long supported from Kubernetes 1.27 HOT 6
- Enable customization of reinvocationPolicy for webhook
- Panic on nil pointer dereference HOT 2
- x509: certificate signed by unknown authority
- Wrong PODs assignment HOT 1
- Cloud Burst Configuration -> Pod does not trigger auto-scale HOT 2
- True decentralized architecture?
- Unable to Access Pod Logs in Source Cluster HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from admiralty.