Comments (14)
Let us assume that clear is not going to change any of their defaults. I think we can add the change in setup_system with a note to the calico thread and get past this issue for now. We can roll it back later once upstreams settle down on the right fix.
from cloud-native-setup.
Looks like this is still does not fix it.
from cloud-native-setup.
This may be a crio or crio configuration issue. k3s (i.e. containerd with flannel) works fine on Clearlinux.
from cloud-native-setup.
clear@clr-01 ~/clr-k8s-examples $ swupd info
Installed version: 29880
clear@clr-01 ~/clr-k8s-examples $ crio --version
crio version 1.14.1
commit: ""
clear@clr-01 ~/clr-k8s-examples $ /lib/systemd/systemd --version
systemd 241 (241)
+PAM +AUDIT -SELINUX +IMA -APPARMOR -SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=legacy
clear@clr-01 ~/clr-k8s-examples $ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
clear@clr-01 ~/clr-k8s-examples $ kubectl scale --replicas=3 deployment nginx
deployment.extensions/nginx scaled
clear@clr-01 ~/clr-k8s-examples $ for i in $(kubectl get pod -l app=nginx -o jsonpath='{.items[*].status.podIP}'); do curl -s -o /dev/null -w "%{http_code}\n" $i; done
200
200
200
clear@clr-01 ~/clr-k8s-examples $ sysctl -ar '\.rp_filter'
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.enp0s8.rp_filter = 0
net.ipv4.conf.enp0s9.rp_filter = 0
net.ipv4.conf.flannel/1.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 0
from cloud-native-setup.
clear@clr-01 ~/clr-k8s-examples $ swupd info
Installed version: 29890
Version URL: https://cdn.download.clearlinux.org/update
Content URL: https://cdn.download.clearlinux.org/update
clear@clr-01 ~/clr-k8s-examples $ crio --version
crio version 1.14.2
commit: ""
clear@clr-01 ~/clr-k8s-examples $ /lib/systemd/systemd --version
systemd 242 (242)
+PAM +AUDIT -SELINUX +IMA -APPARMOR -SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=legacy
clear@clr-01 ~/clr-k8s-examples $ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
clear@clr-01 ~/clr-k8s-examples $ kubectl scale --replicas=3 deployment nginx
deployment.extensions/nginx scaled
clear@clr-01 ~/clr-k8s-examples $ for i in $(kubectl get pod -l app=nginx -o jsonpath='{.items[*].status.podIP}'); do curl -s -o /dev/null -w "%{http_code}\n" $i; done
^C
clear@clr-01 ~/clr-k8s-examples $ sysctl -ar '\.rp_filter'
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.cali2b86df72b07.rp_filter = 1
net.ipv4.conf.caliacd620db126.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.enp0s8.rp_filter = 0
net.ipv4.conf.enp0s9.rp_filter = 0
net.ipv4.conf.flannel/1.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 0
from cloud-native-setup.
version 29890 is where it fails and two changes took place here
crio 1.14.1 to 1.14.2
systemd 241 to 242
from cloud-native-setup.
@mcastelino just to confirm, would attempting to access a service in a pod via ingress be a good test for reproduction?
from cloud-native-setup.
@mcastelino @krsna1729 Testing with containerd and cri-o, I am able to create a simple http-echo service and an ingress to route requests to it. From outside the cluster I can use the IP address of the master node with the NodePort for the ingress-nginx service created by our stack. I can get a response. I tried alternative URLs and successfully received 404s. So at this stage it looks like the workaround of:
+ - name: FELIX_IGNORELOOSERPF
+ value: "true"
Is working.
I assume, however, that we don't want this to remain the case and want to enable rp filtering prior to closing this issue, yes?
from cloud-native-setup.
correct. if we can go back to working sysctl rp filter and remove FELIX_IGNORELOOSERPF, you can close then.
from cloud-native-setup.
Here are details on the rp_filter settings:
rp_filter - INTEGER
0 - No source validation.
1 - Strict mode as defined in RFC3704 Strict Reverse Path
Each incoming packet is tested against the FIB and if the interface
is not the best reverse path the packet check will fail.
By default failed packets are discarded.
2 - Loose mode as defined in RFC3704 Loose Reverse Path
Each incoming packet's source address is also tested against the FIB
and if the source address is not reachable via any interface
the packet check will fail.
Current recommended practice in RFC3704 is to enable strict mode
to prevent IP spoofing from DDos attacks. If using asymmetric routing
or other complicated routing, then loose mode is recommended.
The max value from conf/{all,interface}/rp_filter is used
when doing source validation on the {interface}.
Default value is 0. Note that some distributions enable it
in startup scripts.
The difference between setting 1 and 2 is essentially if the source address is reachable via the incoming interface vs any interface, respectively.
There is a discussion here at projectcalico: https://github.com/projectcalico/calicoctl/issues/1082.
In that discussion this snippet is referenced from the Felix code itself:
# For IPv4, we rely on the kernel's reverse path filtering to prevent
# workloads from spoofing their IP addresses.
#
# The RPF check for a particular interface is controlled by several
# sysctls:
#
# - ipv4.conf.all.rp_filter is a global override
# - ipv4.conf.default.rp_filter controls the value that is set on a newly
# created interface
# - ipv4.conf.<interface>.rp_filter controls a particular interface.
#
# The algorithm for combining the global override and per-interface values
# is to take the *numeric* maximum between the two. The values are:
# 0=off, 1=strict, 2=loose. "loose" is not suitable for Calico since it
# would allow workloads to spoof packets from other workloads on the same
# host. Hence, we need the global override to be <=1 or it would override
# the per-interface setting to "strict" that we require.
#
# We bail out rather than simply setting it because setting 2, "loose",
# is unusual and it is likely to have been set deliberately.
Eventually, the discussion ends with the agreement to add FELIX_IGNORELOOSERPF
, although the preferred setup is to use the recommended setting of 1 rather than 2.
Additionally, this recent issue from coreos coreos/fedora-coreos-tracker#219 gets to the cause of the rp filter value change and it is systemd 240. The discussion there indicates a preference for using os defaults (in this case, supporting rp_filter=2). That results in the issue bouncing back to Project Calico just 27 days ago: https://github.com/projectcalico/felix/issues/2082, where the a Project Calico developer (@fasaxc) has said it is not a great default to have set.
Given this, we have two choices: clear linux could change the setting back to 1 or we leave FELIX_IGNORELOOSERPF
in place. In either case, I can ping Project Calico in issue 2082. Thoughts?
from cloud-native-setup.
I believe we have changed rp_filter in the past ( see d31d78c ). If there is a way we can close this out temporarily, I am open to that change.
from cloud-native-setup.
In this case, 1 is actually a more secure setting. The only issue is that systemd seems to default the setting to 2 now.. So whether we change it to 1 in setup_system or clear changes it in the distro, unless we get systemd to change it back to 1 I think our change would need to remain for the foreseeable future. Unless, of course, Project Calico changes to support loose mode.
from cloud-native-setup.
PR #147
from cloud-native-setup.
Closing per discussion and PR #147
from cloud-native-setup.
Related Issues (20)
- kubeadm yaml needs to be updated to match the version of k8s HOT 1
- setup_system.sh fails when OS version is specified and packages have been removed
- kubeadm displaying warnings HOT 2
- cilium CNI does not work with containerd
- Trying to setup k8s with cri-o and kata-fc runtime on bare metal without Vagrant HOT 3
- consider shallow git clone for create `get()` function
- create: should not print out 'Not-documented' in the help
- rook: fails to come up on single node cluster HOT 4
- Systemd level proxy instead of per service HOT 2
- Problems with no_proxy settings HOT 1
- Add information about Rook to readme
- Rook needs updates
- Does Kubernetes actually work now with Clear Linux? HOT 5
- Missing steps in Vagrant install document HOT 3
- create_stack.sh no longer works
- System level proxy breaks containerd HOT 2
- "swapoff -a" not persistent across reboot HOT 3
- crio runnner is not properly installed with setup_system.sh for automated installation of Kubernetes HOT 1
- vfioveth - POD with DPDK HOT 1
- No bundle update? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloud-native-setup.