Giter Club home page Giter Club logo

Comments (10)

LKaemmerling avatar LKaemmerling commented on July 17, 2024 2

You might need to set the hostname annotation https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/master/internal/annotation/load_balancer.go#L43 on the LoadBalancer service to make the k8s proxy routing the data correctly. That is a known bug within k8s (but the k8s networking guys don't think it is a bug...).

from hcloud-cloud-controller-manager.

MatthiasLohr avatar MatthiasLohr commented on July 17, 2024 1

I think I'm hitting a similar issue and did take a deeper look into it. Actually, the issue seems to be "quite known" in the Kubernetes community (metallb/metallb#153, kubernetes/kubernetes#79976, kubernetes/kubernetes#66607, kubernetes/kubernetes#92312, kubernetes/enhancements#1392, kubernetes/kubernetes#79783, kubernetes/kubernetes#59976).

TLDR: I think the problem is the following:

Using hcloud-cloud-controller-manager, LoadBalancer services get to know their external IPs. This IP gets added to the ipvs0 interface to allow cluster-internal access to the LoadBalancer. Also, a route will be created pointing to this IP on all nodes (ip route show table local). If now the (Hetzner) Load Balancer tries to send a health check packet, the cluster's reply will stay within the cluster, since the route is pointing to the ipvs0 interface instead of the internal network's network card.

There are a lot solutions in discussion, but as far as I know, nothing helpful so far. The only workaround seems to be to use iptables instead of ipvs as kube_proxy mode (didn't try yet with Hetzner Load Balancer). However, this will come with a drawback regarding performance (https://www.projectcalico.org/comparing-kube-proxy-modes-iptables-or-ipvs/).
As a very dirty hack and experiment, I temporarily removed the local route (ip route del local $internal_loadbalancer_ip dev kube-ipvs0 table local) and health checks started to lighten in green immediatly. However, this ugly workaround will not survive a reboot.

Currently reading about stuff that it might be possbile to replace kube_proxy/ipvs with cilium, but just started with trying to understand things there...For now, I guess, only iptables will "work". But I'm happy to discuss and work with you and Hetzner staff to find a solution.

from hcloud-cloud-controller-manager.

acim avatar acim commented on July 17, 2024 1

I forgot to mention that I use Cilium without kubeproxy.

helm install cilium cilium/cilium --namespace kube-system \
  --set global.kubeProxyReplacement=strict \
  --set global.k8sServiceHost=192.168.x.y \
  --set global.k8sServicePort=6443

192.168.x.y = private IP of master

from hcloud-cloud-controller-manager.

acim avatar acim commented on July 17, 2024

Have you configured network properly?

After upgrade to 1.6.1, Cillium still works for me.

from hcloud-cloud-controller-manager.

alexzimmer96 avatar alexzimmer96 commented on July 17, 2024

Okay, i have an update on this after i tried with Cilium. I've tested this with deploying a simple nginx-ingress-controller using Helm. Cloud-Controller-Manager is properly set up.

I can see that the LoadBalancer is created and can connect to the service if a use the NodePorts mentioned by the LoadBalancer.

Here is the LoadBalancer-Service Manifest (I've removed some unnecessary fields):

apiVersion: v1
kind: Service
metadata:
  annotations:
    load-balancer.hetzner.cloud/location: nbg1
    load-balancer.hetzner.cloud/name: nginx-ingress-lb
    load-balancer.hetzner.cloud/use-private-ip: "true"
  name: ingress-nginx-ingress-controller
  namespace: kube-system
spec:
  clusterIP: 10.233.56.32
  externalTrafficPolicy: Local
  healthCheckNodePort: 31134
  ports:
  - name: http
    nodePort: 31337
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 31719
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app: nginx-ingress
    app.kubernetes.io/component: controller
    release: ingress
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 167....
    - ip: 192.168.0.5

Here are some logs

I0626 22:09:25.753157       1 serving.go:313] Generated self-signed cert in-memory
W0626 22:09:26.203013       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0626 22:09:26.207971       1 controllermanager.go:120] Version: v0.0.0-master+$Format:%h$
W0626 22:09:26.620198       1 controllermanager.go:132] detected a cluster without a ClusterID.  A ClusterID will be required in the future.  Please tag your cluster to avoid any future issues
Hetzner Cloud k8s cloud controller v1.6.1 started
I0626 22:09:26.621026       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0626 22:09:26.621068       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0626 22:09:26.621108       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0626 22:09:26.621113       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0626 22:09:26.621270       1 secure_serving.go:178] Serving securely on [::]:10258
I0626 22:09:26.622040       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0626 22:09:26.622247       1 controllermanager.go:247] Started "service"
I0626 22:09:26.622345       1 controller.go:208] Starting service controller
I0626 22:09:26.622358       1 shared_informer.go:223] Waiting for caches to sync for service
I0626 22:09:26.707120       1 controllermanager.go:247] Started "route"
I0626 22:09:26.708629       1 route_controller.go:100] Starting route controller
I0626 22:09:26.708651       1 shared_informer.go:223] Waiting for caches to sync for route
I0626 22:09:26.710168       1 node_controller.go:110] Sending events to api server.
I0626 22:09:26.710250       1 controllermanager.go:247] Started "cloud-node"
I0626 22:09:26.711945       1 node_lifecycle_controller.go:78] Sending events to api server
I0626 22:09:26.712092       1 controllermanager.go:247] Started "cloud-node-lifecycle"
I0626 22:09:26.723993       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I0626 22:09:26.724076       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
I0626 22:09:26.808991       1 shared_informer.go:230] Caches are synced for route 
I0626 22:09:26.822637       1 shared_informer.go:230] Caches are synced for service 
I0626 22:09:44.591416       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"ingress-nginx-ingress-controller", UID:"87539d5a-e9ae-420b-bc95-56927466dab6", APIVersion:"v1", ResourceVersion:"9147", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0626 22:09:44.605421       1 load_balancers.go:81] "ensure Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" service="ingress-nginx-ingress-controller" nodes=[k8s-worker-2 k8s-worker-1]
I0626 22:09:46.312257       1 load_balancer.go:360] "add target" op="hcops/LoadBalancerOps.ReconcileHCLBTargets" service="ingress-nginx-ingress-controller" targetName="k8s-worker-2"
I0626 22:09:47.430045       1 load_balancer.go:360] "add target" op="hcops/LoadBalancerOps.ReconcileHCLBTargets" service="ingress-nginx-ingress-controller" targetName="k8s-worker-1"
I0626 22:09:48.593661       1 load_balancer.go:420] "add service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=80 loadBalancerID=35777
I0626 22:09:49.449492       1 load_balancer.go:420] "add service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=443 loadBalancerID=35777
I0626 22:09:50.394766       1 load_balancers.go:117] "reload HC Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" loadBalancerID=35777
I0626 22:09:50.562668       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"ingress-nginx-ingress-controller", UID:"87539d5a-e9ae-420b-bc95-56927466dab6", APIVersion:"v1", ResourceVersion:"9147", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer
I0626 22:09:50.601969       1 load_balancers.go:81] "ensure Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" service="ingress-nginx-ingress-controller" nodes=[k8s-worker-2 k8s-worker-1]
I0626 22:09:50.602126       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"ingress-nginx-ingress-controller", UID:"87539d5a-e9ae-420b-bc95-56927466dab6", APIVersion:"v1", ResourceVersion:"9195", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0626 22:09:50.727823       1 load_balancer.go:409] "update service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=80 loadBalancerID=35777
I0626 22:09:51.687194       1 load_balancer.go:409] "update service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=443 loadBalancerID=35777
I0626 22:09:52.624144       1 load_balancers.go:117] "reload HC Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" loadBalancerID=35777
I0626 22:09:52.768408       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"ingress-nginx-ingress-controller", UID:"87539d5a-e9ae-420b-bc95-56927466dab6", APIVersion:"v1", ResourceVersion:"9195", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer

from hcloud-cloud-controller-manager.

alexzimmer96 avatar alexzimmer96 commented on July 17, 2024

Have you configured network properly?

After upgrade to 1.6.1, Cillium still works for me.

Hi @acim
It seems properly configured. The routes have been created. Its cut out of the log above because i "restarted" the hcloud-controller-manager for a clear log output.

from hcloud-cloud-controller-manager.

LKaemmerling avatar LKaemmerling commented on July 17, 2024

@alexzimmer96 does everything work for you now? Actually i can't see a problem based on the logs. Does the health check work (visible within the cloud console)?

from hcloud-cloud-controller-manager.

alexzimmer96 avatar alexzimmer96 commented on July 17, 2024

@LKaemmerling unfortunately i deleted the original cluster to try other installation methods and network plugins beside Calico und Cillium. But indeed, the health checks did not succeed. Although i was able to create a connection using the Ports that were specified as the LoadBalancers target, if used the public ip of one any node.

from hcloud-cloud-controller-manager.

MatthiasLohr avatar MatthiasLohr commented on July 17, 2024

Just found this: https://docs.cilium.io/en/v1.8/gettingstarted/kubeproxy-free/

from hcloud-cloud-controller-manager.

github-actions avatar github-actions commented on July 17, 2024

This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs.

from hcloud-cloud-controller-manager.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.