Comments (5)
cilium-agent level=fatal msg="Unable to initialize service maps" error="Unable to create map /sys/fs/bpf/tc/globals/cilium_lb4_reverse_sk: cannot allocate memory" subsys=daemon
This likely means the Cilium pods are running out of memory on those nodes. Have you tried increasing their memory allocation? Are they smaller nodes than the existing ones? Also, keep in mind you're running old Cilium on an old kernel, YMMV.
from cilium.
I too thought of the same, but these hosts have 1TB of memory (MemTotal: 1055937224 kB).
free -m
total used free shared buff/cache available
Mem: 1031188 6135 1010756 6 14296 1020007
Swap: 0 0 0
w.r.t Memory requests & Limit, I see the same config across other Cilium pods on other nodes...
Working Node:
name: cilium-agent
resources:
limits:
cpu: 500m
memory: 500M
requests:
cpu: 100m
memory: 64M
Non-Working Node:
name: cilium-agent
resources:
limits:
cpu: 500m
memory: 500M
requests:
cpu: 100m
memory: 64M
from cilium.
#23436 also looks like the same.
I tried to reimage the node and scale the cluster, but still the same error.
Normal Created 29m (x3 over 30m) kubelet Created container cilium-agent │
│ Normal Started 29m (x3 over 30m) kubelet Started container cilium-agent │
│ Warning Unhealthy 29m (x4 over 30m) kubelet Startup probe failed: Get "http://127.0.0.1:9879/healthz": dial tcp 127.0 │
│ .0.1:9879: connect: connection refused │
│ Warning BackOff 4m56s (x132 over 29m) kubelet Back-off restarting failed container cilium-agent in pod cilium-58k25_kub │e-system(4af21328-f69d-4af8-b1dd-30b97d8c8f78)
level=info msg="Inheriting MTU from external network interface" device=nm_bond ipAddr=172.30.124.90 mtu=1500 subsys=mtu
level=warning msg="No valid cgroup base path found: socket load-balancing tracing with Hubble will not work.See the kubeproxy-free guide for more details." subsys=cgroup-manager
level=info msg="Envoy: Starting xDS gRPC server listening on /var/run/cilium/xds.sock" subsys=envoy-manager
level=fatal msg="Unable to initialize service maps" error="Unable to create map /sys/fs/bpf/tc/globals/cilium_lb4_reverse_sk: cannot allocate memory" subsys=daemon
from cilium.
On the latest kernel as well.
uname -r
4.18.0-553.8.1.el8_10.x86_64
Unfortuantely, I will not be able to upgrade the cilium version since have issues with just 2 nodes, rest of the 30+ nodes are working as expected
from cilium.
@VishnuParthasarathy
For me fixed by making request and limit 10x more
from cilium.
Related Issues (20)
- add pressure metric for fragmentation map
- Communication from world with lower MTU is failing to LoadBalancer services HOT 2
- NetworkPolicy denies communication with EKS Pod Identity Agent HOT 6
- CiliumNetworkPolicy egress overrides CiliumClusterwideNetworkPolicy egress HOT 4
- Hubble exporter config uses wrong separator
- Layer-2-only link between pods
- Error when using multiple allowRoutes namespaces in Gateway
- Service ID conflict causes existing service IP to be disconnected after upgrade HOT 3
- bnxt_en version number confusing in the cilium documentation HOT 2
- Linux 6.6 support in CI
- Use bpf mcpu v4 for Linux kernel 6.6+
- Phase out support for 5.4 kernel HOT 1
- hubble user provided tls documentation doesn't work
- Envoy Metrics missing 1.16.x HOT 1
- cilium mesh etcd old HOT 2
- v1.16.1 cnp validator invalid memory address or nil pointer difference HOT 2
- Cilium mesh host alias and port wrong HOT 3
- Problem with INGRESS traffic from master-node HOT 2
- CFP: CCNP for endpoint `"reserved:kube-apiserver"` should automatically include host policy as well when host firewall is enabled
- init cilium with multi-pool will create the wrong pod-cidr router with k8s node cidr ,not ciliumnode cidr
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cilium.