Comments (30)
krsna1729
commented on July 24, 2024
https://cilium.io/blog/2018/12/10/cilium-14-preview
from cloud-native-setup.
krsna1729
commented on July 24, 2024
/cc @amshinde
from cloud-native-setup.
ganeshmaharaj
commented on July 24, 2024
Can we make it configurable to have both celium and canal?
from cloud-native-setup.
krsna1729
commented on July 24, 2024
cc @mcastelino
from cloud-native-setup.
krsna1729
commented on July 24, 2024
@amshinde @ganeshmaharaj @AntonioMeireles I am seeing this issue when trying out cilium -
MountVolume.SetUp failed for volume "bpf-maps" : mkdir /sys/fs/bpf: operation not permitted
bpf-maps:
Type: HostPath (bare host directory volume)
Path: /sys/fs/bpf
HostPathType: DirectoryOrCreate
$ ls -l /sys/fs
total 0
drwxr-xr-x 13 root root 340 Jan 29 07:52 cgroup
drwxr-xr-x 14 root root 0 Jan 29 07:54 ext4
drwxr-x--- 2 root root 0 Jan 29 07:52 pstore
Any idea if this is a clear kernel issue or clear box issue?
from cloud-native-setup.
AntonioMeireles
commented on July 24, 2024
hi!
can you be a bit more verbose about the specific setup you are using ?
what kernel ?
from cloud-native-setup.
krsna1729
commented on July 24, 2024
@AntonioMeireles your box actually ;)
clear@clr-01 ~ $ cat /etc/os-release
NAME="Clear Linux OS"
VERSION=1
ID=clear-linux-os
ID_LIKE=clear-linux-os
VERSION_ID=27500
PRETTY_NAME="Clear Linux OS"
clear@clr-01 ~ $ uname -a
Linux clr-01 4.19.5-298.kvm #2 SMP Sun Dec 2 06:16:56 UTC 2018 x86_64 GNU/Linux
from cloud-native-setup.
AntonioMeireles
commented on July 24, 2024
under libvirt, right ? on recent revs systemd should had gotten bpf setup automatically... (if it is available by the kernel - 2m double checking)
from cloud-native-setup.
krsna1729
commented on July 24, 2024
That is correct
from cloud-native-setup.
krsna1729
commented on July 24, 2024
On ubuntu 18.04 I have it by default (i'm guessing)
$ ls -l /sys/fs/
total 0
drwxr-xr-x 2 root root 0 Jan 29 02:56 aufs
dr-xr-xr-x 2 root root 0 Jan 29 02:53 bpf
drwxr-xr-x 3 root root 0 Jan 29 02:56 btrfs
drwxr-xr-x 15 root root 380 Nov 1 19:39 cgroup
drwxr-xr-x 2 root root 0 Jan 29 02:56 ecryptfs
drwxr-xr-x 4 root root 0 Jan 29 02:56 ext4
drwxr-xr-x 3 root root 0 Nov 1 19:39 fuse
drwxr-x--- 2 root root 0 Nov 1 19:39 pstore
dr-xr-xr-x 2 root root 0 Jan 24 23:13 resctrl
from cloud-native-setup.
AntonioMeireles
commented on July 24, 2024
- bare-metal ...
[email protected] ~ $ fgrep -ia BPF /usr/src/linux/.config
CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_NET_CLS_BPF=y
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
# CONFIG_BPF_STREAM_PARSER is not set
CONFIG_HAVE_EBPF_JIT=y
# CONFIG_TEST_BPF is not set
[email protected] ~ $ uname -a
Linux blade-01-01.svc.infra 4.14.81-413.lts #1 SMP Thu Nov 15 17:40:10 UTC 2018 x86_64 GNU/Linux
- libvirt
[email protected] ~ $ fgrep -ia BPF /usr/lib/kernel/config-4.19.5-298.kvm
CONFIG_BPF=y
# CONFIG_BPF_SYSCALL is not set
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m
CONFIG_NET_CLS_BPF=y
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
[email protected] ~ $ uname -a
Linux test.clearlinux.local 4.19.5-298.kvm #2 SMP Sun Dec 2 06:16:56 UTC 2018 x86_64 GNU/Linux
the stuff that get's built in kvm kernel not enough :/
from cloud-native-setup.
AntonioMeireles
commented on July 24, 2024
but things not directly comparable - lts vs non lts ... digging more.
Anyway, if only for parity and for production purposes in already suggested in the past that we (well / you (plural)) should think about shipping an lts kernel biased towards kvm ...
from cloud-native-setup.
AntonioMeireles
commented on July 24, 2024
[email protected] ~ $ fgrep -ia BPF /usr/lib/kernel/config-4.19.13-679.native
CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m
CONFIG_NET_CLS_BPF=y
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
# CONFIG_TEST_BPF is not set
ok, late kvm kernels are busted ... for this use case :( they should have above set set ...
from cloud-native-setup.
krsna1729
commented on July 24, 2024
Just for reference ubuntu config
$ uname -r
4.15.0-36-generic
$ grep -i bpf /boot/config-$(uname -r)
CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_TEST_BPF=m
from cloud-native-setup.
AntonioMeireles
commented on July 24, 2024
at sight the deal breaker here is the (missing) CONFIG_CGROUP_BPF . checking here if the kernel-native behaves under libvirt (if so - even if suboptimal - you can move forward :-) )
from cloud-native-setup.
AntonioMeireles
commented on July 24, 2024
[email protected] ~ $ uname -a
Linux test.clearlinux.local 4.19.13-679.native #1 SMP Sat Dec 29 13:28:58 PST 2018 x86_64 GNU/Linux
[email protected] ~ $ ls -al /sys/fs/
total 0
drwxr-xr-x 8 root root 0 Jan 29 08:51 .
dr-xr-xr-x 12 root root 0 Jan 29 08:51 ..
drwx-----T 2 root root 0 Jan 29 08:51 bpf
drwxr-xr-x 3 root root 0 Jan 29 08:52 btrfs
drwxr-xr-x 13 root root 340 Jan 29 08:51 cgroup
drwxr-xr-x 4 root root 0 Jan 29 08:52 ext4
drwxr-x--- 2 root root 0 Jan 29 08:51 pstore
drwxr-xr-x 3 root root 0 Jan 29 08:52 xfs
from cloud-native-setup.
AntonioMeireles
commented on July 24, 2024
so, sudo swupd kernel-native; sudo clr-boot-manager set-kernel org.clearlinux.native.4.19.13-679; sudo reboot on all your nodes should do it in the meantime ... long term fix is to get @fenrus75 do his magic :-)
confirm please, it seems to work here :-)
from cloud-native-setup.
krsna1729
commented on July 24, 2024
Operation successful,
sudo swupd bundle-add kernel-native
sudo clr-boot-manager list-kernels
sudo clr-boot-manager set-kernel org.clearlinux.native.4.20.5-688
sudo reboot
$ ls -l /sys/fs/
total 0
drwx-----T 2 root root 0 Jan 29 08:46 bpf
drwxr-xr-x 3 root root 0 Jan 29 08:46 btrfs
drwxr-xr-x 13 root root 340 Jan 29 08:46 cgroup
drwxr-xr-x 4 root root 0 Jan 29 08:46 ext4
drwxr-x--- 2 root root 0 Jan 29 08:46 pstore
drwxr-xr-x 3 root root 0 Jan 29 08:46 xfs
patient dead
clear@clr-01 ~/clr-k8s-examples $ ./create_stack.sh minimal
[init] Using Kubernetes version: v1.13.2
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR CRI]: container runtime is not running: output: NAME:
clear@clr-01 ~/clr-k8s-examples $ systemctl status crio
● crio.service - Open Container Initiative Daemon
Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/crio.service.d
└─crio-clearlinux.conf
Active: failed (Result: exit-code) since Tue 2019-01-29 08:47:25 UTC; 2min 10s ago
Docs: https://github.com/kubernetes-sigs/cri-o
Process: 1096 ExecStart=/usr/bin/crio $CRIO_STORAGE_OPTIONS $CRIO_NETWORK_OPTIONS $CRIO_METRICS_OPTIONS (code=exited, status=1/FAILURE)
Main PID: 1096 (code=exited, status=1/FAILURE)
from cloud-native-setup.
krsna1729
commented on July 24, 2024
@AntonioMeireles not sure what config in kernel-native messed with crio
from cloud-native-setup.
AntonioMeireles
commented on July 24, 2024
none, afaict. journalctl -u crio logs please
from cloud-native-setup.
AntonioMeireles
commented on July 24, 2024
can you join please ClearLinux's slack and ping me there please ?
from cloud-native-setup.
krsna1729
commented on July 24, 2024
@AntonioMeireles actually let me look into it further. even regular kernel we are facing this issue. Link to slack?
from cloud-native-setup.
AntonioMeireles
commented on July 24, 2024
clearlinux.slack.com
from cloud-native-setup.
krsna1729
commented on July 24, 2024
apparently I need to be invited. Anyhoo, temporarily i moved back to @ganeshmaharaj's box, did the kernel change procedure above and kubeadm init went ahead.
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.3/examples/kubernetes/1.12/cilium-crio.yaml
$ kubectl get po --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cilium-xbxfh 0/1 Running 0 3m48s
kube-system coredns-86c58d9df4-9tpz9 1/1 Running 0 7m59s
kube-system coredns-86c58d9df4-pqzjj 1/1 Running 0 7m59s
kube-system etcd-clr-01 1/1 Running 0 7m7s
kube-system kube-apiserver-clr-01 1/1 Running 0 7m21s
kube-system kube-controller-manager-clr-01 1/1 Running 0 6m58s
kube-system kube-proxy-fxv4g 1/1 Running 0 7m59s
kube-system kube-scheduler-clr-01 1/1 Running 0 7m1s
$ kubectl get node
NAME STATUS ROLES AGE VERSION
clr-01 Ready master 8m35s v1.13.2
Will troubleshoot in the AM
from cloud-native-setup.
bryteise
commented on July 24, 2024
@krsna1729 I think based on the update frequency of the kvm kernel you are better off switching to using the native kernel for these tasks. Sorry about going back and forth between different kernels X(.
from cloud-native-setup.
AntonioMeireles
commented on July 24, 2024
(thinking loud)
if we are going to loose the extra sauce from the -kvm kernel stream then it may make sense to just consume the lts one for this specific use case... as experience tells that k8s not that fast to accommodate (too) recent kernel releases...
from cloud-native-setup.
ganeshmaharaj
commented on July 24, 2024
@AntonioMeireles @krsna1729 tested this issue using both my box and @AntonioMeireles's and they both are working properly now. I saw the same error earlier today.
crio service restart goes through an iteration where it is activating for a long time while it is creating lvm objects in the loopback device and we just need to wait for crio to finish that.
vagrant@alt-clr-01 ~/clr-k8s-examples $ sudo systemctl status crio
? crio.service - Open Container Initiative Daemon
Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/crio.service.d
??crio-clearlinux.conf
/etc/systemd/system/crio.service.d
??proxy.conf
Active: activating (auto-restart) (Result: exit-code) since Wed 2019-01-30 05:58:47 UTC; 9s ago
Docs: https://github.com/kubernetes-sigs/cri-o
Process: 591 ExecStart=/usr/bin/crio $CRIO_STORAGE_OPTIONS $CRIO_NETWORK_OPTIONS $CRIO_METRICS_OPTIONS (code=exited, status=1/FAILURE)
Main PID: 591 (code=exited, status=1/FAILURE)
Jan 30 05:58:47 alt-clr-01 systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE
Jan 30 05:58:47 alt-clr-01 systemd[1]: crio.service: Failed with result 'exit-code'.
Jan 30 05:58:47 alt-clr-01 systemd[1]: Failed to start Open Container Initiative Daemon.
vagrant@alt-clr-01 ~/clr-k8s-examples $ sudo systemctl status crio
? crio.service - Open Container Initiative Daemon
Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/crio.service.d
??crio-clearlinux.conf
/etc/systemd/system/crio.service.d
??proxy.conf
Active: activating (start) since Wed 2019-01-30 05:58:57 UTC; 586ms ago
Docs: https://github.com/kubernetes-sigs/cri-o
Main PID: 641 (crio)
Tasks: 10
Memory: 15.3M
CGroup: /system.slice/crio.service
??641 /usr/bin/crio
??694 lvcreate --wipesignatures y -n thinpoolmeta storage --extents 1%VG
Jan 30 05:58:57 alt-clr-01 systemd[1]: Starting Open Container Initiative Daemon...
Jan 30 05:58:57 alt-clr-01 crio[641]: time="2019-01-30 05:58:57.561940845Z" level=error msg="[VG Name ]"
vagrant@alt-clr-01 ~/clr-k8s-examples $ sudo systemctl status crio
? crio.service - Open Container Initiative Daemon
Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/crio.service.d
??crio-clearlinux.conf
/etc/systemd/system/crio.service.d
??proxy.conf
Active: activating (start) since Wed 2019-01-30 05:58:57 UTC; 2s ago
Docs: https://github.com/kubernetes-sigs/cri-o
Main PID: 641 (crio)
Tasks: 11
Memory: 219.1M
CGroup: /system.slice/crio.service
??641 /usr/bin/crio
??754 /usr/bin/dmeventd
??767 mkfs.ext4 -E nodiscard,lazy_itable_init=0,lazy_journal_init=0 /dev/mapper/container-253:3-268345-base
Jan 30 05:58:57 alt-clr-01 systemd[1]: Starting Open Container Initiative Daemon...
Jan 30 05:58:57 alt-clr-01 crio[641]: time="2019-01-30 05:58:57.561940845Z" level=error msg="[VG Name ]"
Jan 30 05:58:58 alt-clr-01 dmeventd[754]: dmeventd ready for processing.
Jan 30 05:58:58 alt-clr-01 dmeventd[754]: dmeventd libdevmapper-event-lvm2thin.so dlopen failed: libdevmapper-event-lvm2thin.so: cannot>
...skipping...
a bit later you will see that the server is up and running.
vagrant@alt-clr-01 ~/clr-k8s-examples $ sudo systemctl status crio
? crio.service - Open Container Initiative Daemon
Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/crio.service.d
??crio-clearlinux.conf
/etc/systemd/system/crio.service.d
??proxy.conf
Active: active (running) since Wed 2019-01-30 05:59:01 UTC; 2s ago
Docs: https://github.com/kubernetes-sigs/cri-o
Main PID: 641 (crio)
Tasks: 12
Memory: 15.9M
CGroup: /system.slice/crio.service
??641 /usr/bin/crio
??754 /usr/bin/dmeventd
Jan 30 05:58:57 alt-clr-01 systemd[1]: Starting Open Container Initiative Daemon...
Jan 30 05:58:57 alt-clr-01 crio[641]: time="2019-01-30 05:58:57.561940845Z" level=error msg="[VG Name ]"
Jan 30 05:58:58 alt-clr-01 dmeventd[754]: dmeventd ready for processing.
Jan 30 05:58:58 alt-clr-01 dmeventd[754]: dmeventd libdevmapper-event-lvm2thin.so dlopen failed: libdevmapper-event-lvm2thin.so: cannot>
Jan 30 05:59:01 alt-clr-01 crio[641]: time="2019-01-30 05:59:01.949612683Z" level=error msg="error updating cni config: Missing CNI def>
Jan 30 05:59:01 alt-clr-01 systemd[1]: Started Open Container Initiative Daemon.
from cloud-native-setup.
ahsan518
commented on July 24, 2024
Just tested that with Cilium and Crio, and it works fine.
from cloud-native-setup.
mcastelino
commented on July 24, 2024
@ahsan518 can you add the validated cilium manifest to the repo
from cloud-native-setup.
ahsan518
commented on July 24, 2024
Closing this as Cilium is merged.
from cloud-native-setup.
Related Issues (20)
- kubeadm yaml needs to be updated to match the version of k8s HOT 1
- setup_system.sh fails when OS version is specified and packages have been removed
- kubeadm displaying warnings HOT 2
- cilium CNI does not work with containerd
- Trying to setup k8s with cri-o and kata-fc runtime on bare metal without Vagrant HOT 3
- consider shallow git clone for create `get()` function
- create: should not print out 'Not-documented' in the help
- rook: fails to come up on single node cluster HOT 4
- Systemd level proxy instead of per service HOT 2
- Problems with no_proxy settings HOT 1
- Add information about Rook to readme
- Rook needs updates
- Does Kubernetes actually work now with Clear Linux? HOT 5
- Missing steps in Vagrant install document HOT 3
- create_stack.sh no longer works
- System level proxy breaks containerd HOT 2
- "swapoff -a" not persistent across reboot HOT 3
- crio runnner is not properly installed with setup_system.sh for automated installation of Kubernetes HOT 1
- vfioveth - POD with DPDK HOT 1
- No bundle update? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloud-native-setup.