Deployment with private networks support about hcloud-cloud-controller-manager HOT 12 CLOSED

That is just as you want. We use (and recommend) using our networks as pod network too. This just allows direct routing to the pods through our "Network feature".

@LKaemmerling, could you maybe provide an example for pods/service/node subnet? I'm still not getting how to do it right, and which subnet should be part of which other subnet and which not.

Currently I have:

• 10.233.252.0/22 private network with 10.233.252.0/22 as single subnet, used for node IPs
• 10.233.0.0/18 as service network
• 10.233.64.0/18 as pod network

What would be a better configuration here? ("Migration" is not a problem, I'm just playing around, so setting up completely new is fine, too). Thank you very much!

from hcloud-cloud-controller-manager.

This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs.

from hcloud-cloud-controller-manager.

Hi there,
I take it there's no way to actually use private networks with k8s on Hetzner?

from hcloud-cloud-controller-manager.

Sure you can use our Networks. We use them internally too with cilium (like in the docs above). We don't test every possible CNI, so we just test it with cilium.

from hcloud-cloud-controller-manager.

Ok, then pods unable to reach the internet is an intended behavior? Because if I follow the docs on clean kubeadm install that's the behavior I get.

from hcloud-cloud-controller-manager.

Could you provide some logs of the Cloud Controller and maybe have a look into the cloud console, are there any routing within the network?

from hcloud-cloud-controller-manager.

Hi @LKaemmerling
I tried to re-create a new cluster. Unfortunately docs are outdated. With k8s 1.18 I can't get to a working state, coredns stuck in "ContainerCreating".
I'm creating a master + 2 worker Ubuntu 20.04 nodes in:
Private network range: 10.0.0.0/8
Subnet: 10.0.0.0/24
initializing with:

kubeadm init --pod-network-cidr=10.244.0.0/16 --kubernetes-version=v1.18.4 --ignore-preflight-errors=NumCPU --apiserver-advertise-address=10.0.0.2

after joining workers, I fixed tolerations for coredns:

kubectl -n kube-system patch deployment coredns --type json -p '[{"op":"add","path":"/spec/template/spec/tolerations/-","value":{"key":"node.cloudprovider.kubernetes.io/uninitialized","value":"true","effect":"NoSchedule"}}]'

then I deployed cilium:

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.8.0/install/kubernetes/quick-install.yaml

then the controller, after adding secrets:

kubectl apply -f https://raw.githubusercontent.com/hetznercloud/hcloud-cloud-controller-manager/master/deploy/v1.6.1-networks.yaml

The result: I can see routes being created in the cloud console ( same happened before ), but the cluster isn't in a working state. Cilium seems to take over route to 10.0.0.0/24, which makes worker nodes unable to reach control plane.

Downgrading to k8s 1.15 in order to follow the docs isn't an option any longer, kubeadm doesn't support k8s >= 1.17.0.
I don't have time to investigate any further and am ok with closing the issue.

from hcloud-cloud-controller-manager.

@susman after talking to our sysops:

This sounds like a cilium issue. The only thing he said would be adding:

blacklist-conflicting-routes: "false"
auto-direct-node-routes: "false"

to the cilium config. Could you try this?

from hcloud-cloud-controller-manager.

Just to avoid misunderstandings from my side: Why do you need to set --pod-network-cidr=10.244.0.0/16 (assuming that's the subnet you want to use for your network)? Not sure if I understood everything correctly how Hetzner wants us to configure the Kubernetes cluster, but in my opinion pod and services cidr should not overlap with the IPs used for the VMs/internal networking - or did I something wrong?

from hcloud-cloud-controller-manager.

That is just as you want. We use (and recommend) using our networks as pod network too. This just allows direct routing to the pods through our "Network feature".

from hcloud-cloud-controller-manager.

Hi @LKaemmerling ,
Thanks for that!
I tried adding that to cilium manifest, it didn't help either. The problem is the same, coredns' waiting for network plugin.
It did change the behavior though, I wasn't able to get logs from any of cilium's pods:

$ kubectl -n kube-system logs cilium-6tw8n
Error from server: Get https://116.202.182.197:10250/containerLogs/kube-system/cilium-6tw8n/cilium-agent: dial tcp 116.202.182.197:10250: i/o timeout

Looks like it was trying to go over the external network, when conflicting routes got blacklisted.

from hcloud-cloud-controller-manager.

That is just as you want. We use (and recommend) using our networks as pod network too. This just allows direct routing to the pods through our "Network feature".

Not sure how to set that up. Can I prevent the network plugin from creating the routes through the network or is this done automatically/by hcloud-cloud-controller-manager? Is it enough just to have the Pod IP range contained in the Hetzner network subnets?

from hcloud-cloud-controller-manager.