Cilium Pods crashing on Kubernetes Workers about cilium HOT 5 OPEN

cilium-agent level=fatal msg="Unable to initialize service maps" error="Unable to create map /sys/fs/bpf/tc/globals/cilium_lb4_reverse_sk: cannot allocate memory" subsys=daemon

This likely means the Cilium pods are running out of memory on those nodes. Have you tried increasing their memory allocation? Are they smaller nodes than the existing ones? Also, keep in mind you're running old Cilium on an old kernel, YMMV.

from cilium.

I too thought of the same, but these hosts have 1TB of memory (MemTotal: 1055937224 kB).
free -m
total used free shared buff/cache available
Mem: 1031188 6135 1010756 6 14296 1020007
Swap: 0 0 0

w.r.t Memory requests & Limit, I see the same config across other Cilium pods on other nodes...

Working Node:
name: cilium-agent
resources:
limits:
cpu: 500m
memory: 500M
requests:
cpu: 100m
memory: 64M

Non-Working Node:
name: cilium-agent
resources:
limits:
cpu: 500m
memory: 500M
requests:
cpu: 100m
memory: 64M

from cilium.

#23436 also looks like the same.
I tried to reimage the node and scale the cluster, but still the same error.

Normal Created 29m (x3 over 30m) kubelet Created container cilium-agent │
│ Normal Started 29m (x3 over 30m) kubelet Started container cilium-agent │
│ Warning Unhealthy 29m (x4 over 30m) kubelet Startup probe failed: Get "http://127.0.0.1:9879/healthz": dial tcp 127.0 │
│ .0.1:9879: connect: connection refused │
│ Warning BackOff 4m56s (x132 over 29m) kubelet Back-off restarting failed container cilium-agent in pod cilium-58k25_kub │e-system(4af21328-f69d-4af8-b1dd-30b97d8c8f78)

level=info msg="Inheriting MTU from external network interface" device=nm_bond ipAddr=172.30.124.90 mtu=1500 subsys=mtu
level=warning msg="No valid cgroup base path found: socket load-balancing tracing with Hubble will not work.See the kubeproxy-free guide for more details." subsys=cgroup-manager
level=info msg="Envoy: Starting xDS gRPC server listening on /var/run/cilium/xds.sock" subsys=envoy-manager
level=fatal msg="Unable to initialize service maps" error="Unable to create map /sys/fs/bpf/tc/globals/cilium_lb4_reverse_sk: cannot allocate memory" subsys=daemon

from cilium.

On the latest kernel as well.
uname -r
4.18.0-553.8.1.el8_10.x86_64

Unfortuantely, I will not be able to upgrade the cilium version since have issues with just 2 nodes, rest of the 30+ nodes are working as expected

from cilium.

@VishnuParthasarathy
For me fixed by making request and limit 10x more

from cilium.