Comments (8)
theatrus
commented on August 20, 2024
Could you grab the output of
sudo cni-ipvlan-vpc-k8s-tool eniif and sudo cni-ipvlan-vpc-k8s-tool free-ips ? I suspect we are not detecting the in-use IP correctly on this host configuration (this is a thorny issue with docker as it uses un-named network namespaces - we tested one configuration of K8S 1.8.3 + Fedora packaged Docker, but differences are likely to sprout up)
from cni-ipvlan-vpc-k8s.
liwenwu-amazon
commented on August 20, 2024
root@ip-10-0-55-131:/home/admin# ./cni-ipvlan-vpc-k8s-tool eniif
iface mac id subnet subnet_cidr secgrps vpc ips
eth0 0e:94:8f:aa:19:c4 eni-4247fbca subnet-3bf05866 10.0.32.0/19 [sg-34b79141] vpc-0066bd79 [10.0.55.131]
eth1 0e:e6:73:13:66:ac eni-e1dc6569 subnet-c2e8419f 10.0.5.0/24 [sg-34b79141] vpc-0066bd79 [10.0.5.154 10.0.5.23]
and
root@ip-10-0-55-131:/home/admin# ./cni-ipvlan-vpc-k8s-tool free-ips
Couldn't enumerate named namespaces
adapter ip
eth1 10.0.5.23
from cni-ipvlan-vpc-k8s.
liwenwu-amazon
commented on August 20, 2024
Here is the image I am using
kops get instancegroups nodes -oyaml
Using cluster from kubectl context: lyft-dec12.k8s-test.com
apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: 2017-12-12T20:40:40Z
labels:
kops.k8s.io/cluster: lyft-dec12.k8s-test.com
name: nodes
spec:
image: kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-12-02
machineType: t2.medium
maxSize: 3
minSize: 3
role: Node
subnets:
- us-east-1a
from cni-ipvlan-vpc-k8s.
theatrus
commented on August 20, 2024
We’re looking at this. Suspect it’s a race condition with the free-ip finder that triggers under Docker: if there is no running container on docker at the time the next queued IPAM job runs, or at least according to the Docker API, we wouldn’t detect that network namespace existing and re-issue the IP.
from cni-ipvlan-vpc-k8s.
theatrus
commented on August 20, 2024
I normalized the logic used in #11 to the dockershim code's Docker namespace as found in Kubernetes its self - this may help the situation.
On my test system, I can scale up and down 50 busybox pods on a single kubelet without issues. Specifics:
API version: 1.32
Go version: go1.8.3
Git commit: 19e2cf6
Built: Thu Dec 7 22:25:06 2017
OS/Arch: linux/amd64
Server:
Version: 17.09.1-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: 19e2cf6
Built: Thu Dec 7 22:26:29 2017
OS/Arch: linux/amd64
Experimental: false
(as packaged in Fedora 26)
Kernel: 4.14.4-200.fc26.x86_64
Kubernetes v1.8.3
Example scaling stress test:
default busybox-6986c7c9c7-2rdxl 1/1 Running 0 4m 172.31.157.8 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-2zjz6 1/1 Running 0 4m 172.31.149.202 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-4ghms 1/1 Running 0 4m 172.31.145.1 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-5gxql 1/1 Running 0 4m 172.31.202.50 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-5s6cx 1/1 Running 0 4m 172.31.144.162 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-6d8dd 1/1 Running 0 4m 172.31.205.104 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-6jl9b 1/1 Running 0 46s 172.31.157.69 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-7cr9p 1/1 Running 0 4m 172.31.150.40 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-7nrlj 1/1 Running 0 4m 172.31.196.175 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-7t8h4 1/1 Running 0 46s 172.31.163.33 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-9sc5j 1/1 Running 0 4m 172.31.202.26 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-b82nj 1/1 Running 0 46s 172.31.166.37 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-bp6zl 1/1 Running 0 4m 172.31.147.89 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-btxqs 1/1 Running 0 4m 172.31.193.235 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-cktdg 1/1 Running 0 46s 172.31.170.74 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-fxb2n 1/1 Running 0 46s 172.31.148.179 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-gmzz8 1/1 Running 0 4m 172.31.200.84 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-hclrk 1/1 Running 0 46s 172.31.174.36 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-hcnz6 1/1 Running 0 46s 172.31.157.208 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-hjq4k 1/1 Running 0 46s 172.31.169.125 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-hpwp4 1/1 Running 0 4m 172.31.200.213 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-kdz8b 1/1 Running 0 4m 172.31.147.248 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-l6vns 1/1 Running 0 4m 172.31.146.115 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-mjqd4 1/1 Running 0 4m 172.31.147.249 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-ms48p 1/1 Running 0 46s 172.31.154.38 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-mt4th 1/1 Running 0 4m 172.31.146.254 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-nzzxw 1/1 Running 0 4m 172.31.193.242 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-rzwjc 1/1 Running 0 4m 172.31.194.78 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-shsgk 1/1 Running 0 46s 172.31.158.253 ip-172-31-37-168.ec2.internal
default busybox-6986c7c9c7-tphmd 1/1 Running 0 4m 172.31.207.90 ip-172-31-37-168.ec2.internal
kube-system cluster-autoscaler-b48f465cf-n9fm2 1/1 Running 0 39m 172.31.41.154 ip-172-31-41-154.ec2.internal
kube-system cluster-autoscaler-b48f465cf-q8m27 1/1 Running 0 39m 172.31.34.190 ip-172-31-34-190.ec2.internal
kube-system heapster-69c44d5864-sb4h9 1/1 Running 0 13m 172.31.198.230 ip-172-31-37-168.ec2.internal
kube-system kube-dns-7797cb8758-56tng 3/3 Running 0 13m 172.31.194.45 ip-172-31-37-168.ec2.internal
kube-system kube-dns-7797cb8758-ccz8z 3/3 Running 0 13m 172.31.192.181 ip-172-31-37-168.ec2.internal
kube-system kube-dns-autoscaler-7db47cb9b7-6zp29 1/1 Running 0 13m 172.31.203.152 ip-172-31-37-168.ec2.internal
kube-system kube2iam-ts8tf 1/1 Running 0 22m 172.31.37.168 ip-172-31-37-168.ec2.internal
kube-system kubernetes-dashboard-747c4f7cf-9vp5c 1/1 Running 0 13m 172.31.197.209 ip-172-31-37-168.ec2.internal
kube-system npd-v0.4.1-c65sr 1/1 Running 0 16m 172.31.37.168 ip-172-31-37-168.ec2.internal
kube-system rescheduler-6df54645b7-zbvvt 1/1 Running 0 13m 172.31.37.168 ip-172-31-37-168.ec2.internal
I have not had a chance to re-create this using Ubuntu and/or Kubernetes 1.7.x. Will give that a go next.
from cni-ipvlan-vpc-k8s.
liwenwu-amazon
commented on August 20, 2024
Does it mean that this CNI plugin will NOT work with other Container Runtime except Docker? thanks
from cni-ipvlan-vpc-k8s.
theatrus
commented on August 20, 2024
We use two paths
- Try to find unnamed namespaces by looking through running docker containers
- Looking through named network namespaces in the conventional
/var/run/netnspath which is used by the external CRI runtime. We've only tested this with CRI-O and not containerd.
If no Docker daemon is detected we simply skip populating the docker data. I'm still doing more testing on my change :)
from cni-ipvlan-vpc-k8s.
theatrus
commented on August 20, 2024
Had a chance to run #11 through a mixed CRIO/Docker cluster using conformance tests on 1.8.5 and didn't notice any leakage, in addition to some stress testing by hand. I still haven't had a chance to test on 1.7 or Ubuntu kernels/docker distributions. However, #11 is a lot more resilient to how namespaces are handled and matches the logic inside Kubernetes so I expect this problem to be mitigated. We'll cut a release tonight or tomorrow.
from cni-ipvlan-vpc-k8s.
Related Issues (20)
- Clarify configuration and setup instructions
- The CNI plugin only works on OS where interfaces are called ethX
- Additional interfaces could be in the same subnet HOT 8
- Plugin does not work with containerd runtime HOT 6
- missing routes on first pod for an interface HOT 3
- Nodeports not working properly HOT 8
- Incorrect handling of errors in IPAM HOT 1
- Inconsistency between ENI allocated IPs and OS configuration HOT 16
- CNI plugin: `portmap` when chained with cni-ipvlan-vpc-k8s does not work
- Listing VPC CIDRs to create routes in pods does not filter on association state
- New release HOT 1
- Disabling ipMasq
- Cannot connect to pod outside VPC HOT 4
- Pods loosing connection to each other on particular node from time to time HOT 13
- Build binaries for 0.5.0 release HOT 1
- aws-vpc-cni-k8s vs lyft cni HOT 3
- IP Batching changes the default behavior HOT 1
- Brother, I compile for a day. Compilation is unsuccessful. Please find a compiled package.
- Binary Release for 0.6.1 HOT 3
- support for containerd
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cni-ipvlan-vpc-k8s.