Giter Club home page Giter Club logo

Comments (14)

ganeshmaharaj avatar ganeshmaharaj commented on August 27, 2024 1

Let us assume that clear is not going to change any of their defaults. I think we can add the change in setup_system with a note to the calico thread and get past this issue for now. We can roll it back later once upstreams settle down on the right fix.

from cloud-native-setup.

mcastelino avatar mcastelino commented on August 27, 2024

Looks like this is still does not fix it.

from cloud-native-setup.

mcastelino avatar mcastelino commented on August 27, 2024

This may be a crio or crio configuration issue. k3s (i.e. containerd with flannel) works fine on Clearlinux.

from cloud-native-setup.

krsna1729 avatar krsna1729 commented on August 27, 2024
clear@clr-01 ~/clr-k8s-examples $ swupd info
Installed version: 29880
clear@clr-01 ~/clr-k8s-examples $ crio --version
crio version 1.14.1
commit: ""
clear@clr-01 ~/clr-k8s-examples $ /lib/systemd/systemd --version
systemd 241 (241)
+PAM +AUDIT -SELINUX +IMA -APPARMOR -SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=legacy
clear@clr-01 ~/clr-k8s-examples $ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
clear@clr-01 ~/clr-k8s-examples $ kubectl scale --replicas=3 deployment nginx
deployment.extensions/nginx scaled
clear@clr-01 ~/clr-k8s-examples $ for i in $(kubectl get pod -l app=nginx -o jsonpath='{.items[*].status.podIP}'); do curl -s -o /dev/null -w "%{http_code}\n" $i;  done
200
200
200
clear@clr-01 ~/clr-k8s-examples $ sysctl -ar '\.rp_filter'
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.enp0s8.rp_filter = 0
net.ipv4.conf.enp0s9.rp_filter = 0
net.ipv4.conf.flannel/1.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 0

from cloud-native-setup.

krsna1729 avatar krsna1729 commented on August 27, 2024
clear@clr-01 ~/clr-k8s-examples $ swupd info
Installed version: 29890
Version URL:       https://cdn.download.clearlinux.org/update
Content URL:       https://cdn.download.clearlinux.org/update
clear@clr-01 ~/clr-k8s-examples $ crio --version
crio version 1.14.2
commit: ""
clear@clr-01 ~/clr-k8s-examples $ /lib/systemd/systemd --version
systemd 242 (242)
+PAM +AUDIT -SELINUX +IMA -APPARMOR -SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=legacy
clear@clr-01 ~/clr-k8s-examples $ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
clear@clr-01 ~/clr-k8s-examples $ kubectl scale --replicas=3 deployment nginx
deployment.extensions/nginx scaled
clear@clr-01 ~/clr-k8s-examples $ for i in $(kubectl get pod -l app=nginx -o jsonpath='{.items[*].status.podIP}'); do curl -s -o /dev/null -w "%{http_code}\n" $i;  done

^C
clear@clr-01 ~/clr-k8s-examples $ sysctl -ar '\.rp_filter'
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.cali2b86df72b07.rp_filter = 1
net.ipv4.conf.caliacd620db126.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.enp0s8.rp_filter = 0
net.ipv4.conf.enp0s9.rp_filter = 0
net.ipv4.conf.flannel/1.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 0

from cloud-native-setup.

krsna1729 avatar krsna1729 commented on August 27, 2024

version 29890 is where it fails and two changes took place here
crio 1.14.1 to 1.14.2
systemd 241 to 242

from cloud-native-setup.

NitinAtIntel avatar NitinAtIntel commented on August 27, 2024

@mcastelino just to confirm, would attempting to access a service in a pod via ingress be a good test for reproduction?

from cloud-native-setup.

NitinAtIntel avatar NitinAtIntel commented on August 27, 2024

@mcastelino @krsna1729 Testing with containerd and cri-o, I am able to create a simple http-echo service and an ingress to route requests to it. From outside the cluster I can use the IP address of the master node with the NodePort for the ingress-nginx service created by our stack. I can get a response. I tried alternative URLs and successfully received 404s. So at this stage it looks like the workaround of:
+ - name: FELIX_IGNORELOOSERPF
+ value: "true"
Is working.
I assume, however, that we don't want this to remain the case and want to enable rp filtering prior to closing this issue, yes?

from cloud-native-setup.

krsna1729 avatar krsna1729 commented on August 27, 2024

correct. if we can go back to working sysctl rp filter and remove FELIX_IGNORELOOSERPF, you can close then.

from cloud-native-setup.

NitinAtIntel avatar NitinAtIntel commented on August 27, 2024

Here are details on the rp_filter settings:

rp_filter - INTEGER
    0 - No source validation.
    1 - Strict mode as defined in RFC3704 Strict Reverse Path
        Each incoming packet is tested against the FIB and if the interface
        is not the best reverse path the packet check will fail.
        By default failed packets are discarded.
    2 - Loose mode as defined in RFC3704 Loose Reverse Path
        Each incoming packet's source address is also tested against the FIB
        and if the source address is not reachable via any interface
        the packet check will fail.

    Current recommended practice in RFC3704 is to enable strict mode
    to prevent IP spoofing from DDos attacks. If using asymmetric routing
    or other complicated routing, then loose mode is recommended.

    The max value from conf/{all,interface}/rp_filter is used
    when doing source validation on the {interface}.

    Default value is 0. Note that some distributions enable it
    in startup scripts.

The difference between setting 1 and 2 is essentially if the source address is reachable via the incoming interface vs any interface, respectively.

There is a discussion here at projectcalico: https://github.com/projectcalico/calicoctl/issues/1082.
In that discussion this snippet is referenced from the Felix code itself:

# For IPv4, we rely on the kernel's reverse path filtering to prevent
# workloads from spoofing their IP addresses.
#
# The RPF check for a particular interface is controlled by several
# sysctls:
#
# - ipv4.conf.all.rp_filter is a global override
# - ipv4.conf.default.rp_filter controls the value that is set on a newly
#   created interface
# - ipv4.conf.<interface>.rp_filter controls a particular interface.
#
# The algorithm for combining the global override and per-interface values
# is to take the *numeric* maximum between the two.  The values are:
# 0=off, 1=strict, 2=loose.  "loose" is not suitable for Calico since it
# would allow workloads to spoof packets from other workloads on the same
# host.  Hence, we need the global override to be <=1 or it would override
# the per-interface setting to "strict" that we require.
#
# We bail out rather than simply setting it because setting 2, "loose",
# is unusual and it is likely to have been set deliberately.

Eventually, the discussion ends with the agreement to add FELIX_IGNORELOOSERPF, although the preferred setup is to use the recommended setting of 1 rather than 2.

Additionally, this recent issue from coreos coreos/fedora-coreos-tracker#219 gets to the cause of the rp filter value change and it is systemd 240. The discussion there indicates a preference for using os defaults (in this case, supporting rp_filter=2). That results in the issue bouncing back to Project Calico just 27 days ago: https://github.com/projectcalico/felix/issues/2082, where the a Project Calico developer (@fasaxc) has said it is not a great default to have set.

Given this, we have two choices: clear linux could change the setting back to 1 or we leave FELIX_IGNORELOOSERPF in place. In either case, I can ping Project Calico in issue 2082. Thoughts?

from cloud-native-setup.

ganeshmaharaj avatar ganeshmaharaj commented on August 27, 2024

I believe we have changed rp_filter in the past ( see d31d78c ). If there is a way we can close this out temporarily, I am open to that change.

from cloud-native-setup.

NitinAtIntel avatar NitinAtIntel commented on August 27, 2024

In this case, 1 is actually a more secure setting. The only issue is that systemd seems to default the setting to 2 now.. So whether we change it to 1 in setup_system or clear changes it in the distro, unless we get systemd to change it back to 1 I think our change would need to remain for the foreseeable future. Unless, of course, Project Calico changes to support loose mode.

from cloud-native-setup.

NitinAtIntel avatar NitinAtIntel commented on August 27, 2024

PR #147

from cloud-native-setup.

NitinAtIntel avatar NitinAtIntel commented on August 27, 2024

Closing per discussion and PR #147

from cloud-native-setup.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.