Comments (10)
You might need to set the hostname annotation https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/master/internal/annotation/load_balancer.go#L43 on the LoadBalancer service to make the k8s proxy routing the data correctly. That is a known bug within k8s (but the k8s networking guys don't think it is a bug...).
from hcloud-cloud-controller-manager.
I think I'm hitting a similar issue and did take a deeper look into it. Actually, the issue seems to be "quite known" in the Kubernetes community (metallb/metallb#153, kubernetes/kubernetes#79976, kubernetes/kubernetes#66607, kubernetes/kubernetes#92312, kubernetes/enhancements#1392, kubernetes/kubernetes#79783, kubernetes/kubernetes#59976).
TLDR: I think the problem is the following:
Using hcloud-cloud-controller-manager, LoadBalancer services get to know their external IPs. This IP gets added to the ipvs0 interface to allow cluster-internal access to the LoadBalancer. Also, a route will be created pointing to this IP on all nodes (ip route show table local
). If now the (Hetzner) Load Balancer tries to send a health check packet, the cluster's reply will stay within the cluster, since the route is pointing to the ipvs0 interface instead of the internal network's network card.
There are a lot solutions in discussion, but as far as I know, nothing helpful so far. The only workaround seems to be to use iptables instead of ipvs as kube_proxy mode (didn't try yet with Hetzner Load Balancer). However, this will come with a drawback regarding performance (https://www.projectcalico.org/comparing-kube-proxy-modes-iptables-or-ipvs/).
As a very dirty hack and experiment, I temporarily removed the local route (ip route del local $internal_loadbalancer_ip dev kube-ipvs0 table local
) and health checks started to lighten in green immediatly. However, this ugly workaround will not survive a reboot.
Currently reading about stuff that it might be possbile to replace kube_proxy/ipvs with cilium, but just started with trying to understand things there...For now, I guess, only iptables will "work". But I'm happy to discuss and work with you and Hetzner staff to find a solution.
from hcloud-cloud-controller-manager.
I forgot to mention that I use Cilium without kubeproxy.
helm install cilium cilium/cilium --namespace kube-system \
--set global.kubeProxyReplacement=strict \
--set global.k8sServiceHost=192.168.x.y \
--set global.k8sServicePort=6443
192.168.x.y = private IP of master
from hcloud-cloud-controller-manager.
Have you configured network properly?
After upgrade to 1.6.1, Cillium still works for me.
from hcloud-cloud-controller-manager.
Okay, i have an update on this after i tried with Cilium. I've tested this with deploying a simple nginx-ingress-controller using Helm. Cloud-Controller-Manager is properly set up.
I can see that the LoadBalancer is created and can connect to the service if a use the NodePorts mentioned by the LoadBalancer.
Here is the LoadBalancer-Service Manifest (I've removed some unnecessary fields):
apiVersion: v1
kind: Service
metadata:
annotations:
load-balancer.hetzner.cloud/location: nbg1
load-balancer.hetzner.cloud/name: nginx-ingress-lb
load-balancer.hetzner.cloud/use-private-ip: "true"
name: ingress-nginx-ingress-controller
namespace: kube-system
spec:
clusterIP: 10.233.56.32
externalTrafficPolicy: Local
healthCheckNodePort: 31134
ports:
- name: http
nodePort: 31337
port: 80
protocol: TCP
targetPort: http
- name: https
nodePort: 31719
port: 443
protocol: TCP
targetPort: https
selector:
app: nginx-ingress
app.kubernetes.io/component: controller
release: ingress
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: 167....
- ip: 192.168.0.5
Here are some logs
I0626 22:09:25.753157 1 serving.go:313] Generated self-signed cert in-memory
W0626 22:09:26.203013 1 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0626 22:09:26.207971 1 controllermanager.go:120] Version: v0.0.0-master+$Format:%h$
W0626 22:09:26.620198 1 controllermanager.go:132] detected a cluster without a ClusterID. A ClusterID will be required in the future. Please tag your cluster to avoid any future issues
Hetzner Cloud k8s cloud controller v1.6.1 started
I0626 22:09:26.621026 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0626 22:09:26.621068 1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0626 22:09:26.621108 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0626 22:09:26.621113 1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0626 22:09:26.621270 1 secure_serving.go:178] Serving securely on [::]:10258
I0626 22:09:26.622040 1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0626 22:09:26.622247 1 controllermanager.go:247] Started "service"
I0626 22:09:26.622345 1 controller.go:208] Starting service controller
I0626 22:09:26.622358 1 shared_informer.go:223] Waiting for caches to sync for service
I0626 22:09:26.707120 1 controllermanager.go:247] Started "route"
I0626 22:09:26.708629 1 route_controller.go:100] Starting route controller
I0626 22:09:26.708651 1 shared_informer.go:223] Waiting for caches to sync for route
I0626 22:09:26.710168 1 node_controller.go:110] Sending events to api server.
I0626 22:09:26.710250 1 controllermanager.go:247] Started "cloud-node"
I0626 22:09:26.711945 1 node_lifecycle_controller.go:78] Sending events to api server
I0626 22:09:26.712092 1 controllermanager.go:247] Started "cloud-node-lifecycle"
I0626 22:09:26.723993 1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0626 22:09:26.724076 1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0626 22:09:26.808991 1 shared_informer.go:230] Caches are synced for route
I0626 22:09:26.822637 1 shared_informer.go:230] Caches are synced for service
I0626 22:09:44.591416 1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"ingress-nginx-ingress-controller", UID:"87539d5a-e9ae-420b-bc95-56927466dab6", APIVersion:"v1", ResourceVersion:"9147", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0626 22:09:44.605421 1 load_balancers.go:81] "ensure Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" service="ingress-nginx-ingress-controller" nodes=[k8s-worker-2 k8s-worker-1]
I0626 22:09:46.312257 1 load_balancer.go:360] "add target" op="hcops/LoadBalancerOps.ReconcileHCLBTargets" service="ingress-nginx-ingress-controller" targetName="k8s-worker-2"
I0626 22:09:47.430045 1 load_balancer.go:360] "add target" op="hcops/LoadBalancerOps.ReconcileHCLBTargets" service="ingress-nginx-ingress-controller" targetName="k8s-worker-1"
I0626 22:09:48.593661 1 load_balancer.go:420] "add service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=80 loadBalancerID=35777
I0626 22:09:49.449492 1 load_balancer.go:420] "add service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=443 loadBalancerID=35777
I0626 22:09:50.394766 1 load_balancers.go:117] "reload HC Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" loadBalancerID=35777
I0626 22:09:50.562668 1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"ingress-nginx-ingress-controller", UID:"87539d5a-e9ae-420b-bc95-56927466dab6", APIVersion:"v1", ResourceVersion:"9147", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer
I0626 22:09:50.601969 1 load_balancers.go:81] "ensure Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" service="ingress-nginx-ingress-controller" nodes=[k8s-worker-2 k8s-worker-1]
I0626 22:09:50.602126 1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"ingress-nginx-ingress-controller", UID:"87539d5a-e9ae-420b-bc95-56927466dab6", APIVersion:"v1", ResourceVersion:"9195", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0626 22:09:50.727823 1 load_balancer.go:409] "update service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=80 loadBalancerID=35777
I0626 22:09:51.687194 1 load_balancer.go:409] "update service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=443 loadBalancerID=35777
I0626 22:09:52.624144 1 load_balancers.go:117] "reload HC Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" loadBalancerID=35777
I0626 22:09:52.768408 1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"ingress-nginx-ingress-controller", UID:"87539d5a-e9ae-420b-bc95-56927466dab6", APIVersion:"v1", ResourceVersion:"9195", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer
from hcloud-cloud-controller-manager.
Have you configured network properly?
After upgrade to 1.6.1, Cillium still works for me.
Hi @acim
It seems properly configured. The routes have been created. Its cut out of the log above because i "restarted" the hcloud-controller-manager for a clear log output.
from hcloud-cloud-controller-manager.
@alexzimmer96 does everything work for you now? Actually i can't see a problem based on the logs. Does the health check work (visible within the cloud console)?
from hcloud-cloud-controller-manager.
@LKaemmerling unfortunately i deleted the original cluster to try other installation methods and network plugins beside Calico und Cillium. But indeed, the health checks did not succeed. Although i was able to create a connection using the Ports that were specified as the LoadBalancers target, if used the public ip of one any node.
from hcloud-cloud-controller-manager.
Just found this: https://docs.cilium.io/en/v1.8/gettingstarted/kubeproxy-free/
from hcloud-cloud-controller-manager.
This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs.
from hcloud-cloud-controller-manager.
Related Issues (20)
- feat: Event when Robot Node is removed because name changed HOT 1
- docs: explain and recommend Robot Webservice Users HOT 1
- Taint uninitialized not getting removed HOT 1
- LoadBalancer / Service Enhancements
- Multiple LoadBalancer services not correct in LB services list HOT 2
- Controller does not properly add node metadata HOT 6
- Pod crashes when setting HCLOUD_NETWORK and network: false HOT 10
- docs(load-balancer): explain how to use managed & uploaded certificates HOT 1
- LB: load-balancer.hetzner.cloud/node-selector annotation not working HOT 3
- Node Addresses won't get updated when using Wireguard for Cluster Creation HOT 9
- Calico and HCC HOT 21
- Invalid characters in labele instance-type HOT 1
- CCM failed to get node address from cloud provider that matches IPv6 HOT 4
- Automatically label robot nodes HOT 2
- invalid_input when deleting loadbalancer manually.
- Inconsistency in the naming for load balancer annotation. HOT 2
- CCM with Hetzner network attempts to create IPv6 routes HOT 5
- fix: Too many requests to `GET /v1/servers/{id}`
- Hetzner Cloud Control manager not connecting with Hetzner HOT 5
- CCM installation with microk8s HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hcloud-cloud-controller-manager.