Pods can't reach pods from other subnetworks about k0s HOT 9 CLOSED

Related: #4121

from k0s.

I don't see any errors in ..., kube-router, ...

I was about to respond here that kube-router operates on layer 2 (link level) which means that the nodes needs to be in the same layer 2 network. However, while looking for this in the documentation I found out that it looks like kube-router can do ipip overlay tunneling when nodes are not in the same subnet.

Last time I tried this I ended up using calico with vxlan, which worked nicely.

from k0s.

Can the nodes communicate over the internal-ip? eg can you ping 10.0.18.32 from 192.168.0.170?

from k0s.

No, they can't. The AWS EC2 node is behind the public IP 63.32.124.37 and the other 2 instances behind the public IP 91.126.192.82.

I don't use these public IPs as IP Nodes because I understant that I can't use public the same public IPs more than once.

One of the reasons of open the issue is to know if this configuration is possible (connect nodes with diferent public ips/networks and some of them behind the same public IP 🤔)

Anyway, tomorrow I'll try what you say, tunneling overlay ipip with kube-router or use calico.

Thanks!

from k0s.

I don't think that will work at all. It will not be possible for the worker nodes to establish any tunnel or anything if they cannot talk to each other. You need have some sort of routing working between the nodes.

from k0s.

Hello, sorry for the delay in answering, I have been testing different configurations these days.

Right now I am trying to configure the cluster with Wireguard for the network section.

I managed to build the cluster as I wanted by following this document:

https://www.inovex.de/de/blog/how-to-set-up-a-k3s-cluster-on-wireguard/

As you can see, it is in K3S, but I want to use K0S. Once I have managed to replicate the behavior, I am going to expose my final K0S configuration here in case someone else has a similar use case and close the incident.

Many thanks for everything

from k0s.

The issue is marked as stale since no activity has been recorded in 30 days

from k0s.

@ErGallo23 you can try Kilo as CNI, which is based on WireGuard. I think it does exactly what you need. To do it, specify spec.network.provider: custom in your k0s config and install Kilo manually

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: k0s
spec:
  network:
    provider: custom

kubectl apply -f https://raw.githubusercontent.com/squat/kilo/main/manifests/crds.yaml
kubectl apply -f https://raw.githubusercontent.com/squat/kilo/main/manifests/kilo-k3s.yaml

It mentions k3s, but I've tried it with k0s and it worked well in my tests. Basically, there are some paths that should be changed from k3s-specific to k0s-specific

from k0s.

Hi! Thanks @makhov for the answer.

Now I'm using cilium with WG installed in each node and it works fine, but, as you said, after look in Kilo documentation, maybe it's a better approach than run wg on node instead of using Kilo CRDs.

I'll do some tests with kilo and I'll decide the best option between k0s+kilo or k0s+wg in node itself.

Thanks again!

from k0s.