Missing providerID, labels and annotations about hcloud-cloud-controller-manager HOT 4 CLOSED

Hey @morsik,

Is the taint node.cloudprovider.kubernetes.io/uninitialized still set on your nodes? This taint is added by the kubelet when it first starts and creates the Node, but only if the initial start is done with the flag --cloud-provider=external. We only set the Provider ID & Taints on the first reconcile, if this taint is set. After that we (or rather the kubernetes/cloud-provider library we use) considers the node "initialized" and this is not being checked again.

Could you post the logs of hcloud-cloud-controller-manager during the first few minutes after adding a new node?

from hcloud-cloud-controller-manager.

@apricote nope, I never saw such taint. The only taint I had was standard node-role.kubernetes.io/control-plane:NoSchedule which I removed in my use-case.

morsik@Star-Destroyer ~ $ kubectl describe nodes | grep Taint -A2
Taints:             <none>
Unschedulable:      false
Lease:
--
Taints:             <none>
Unschedulable:      false
Lease:
--
Taints:             <none>
Unschedulable:      false
Lease:

Also:

Could you post the logs of hcloud-cloud-controller-manager during the first few minutes after adding a new node?

Does this mean, that if node was in "wrong" state when adding, then it'll be never fixed and I have to reinitialize nodes?

BUT... I will verify all of this because:

• Regarding master node: while it was initially started with --cloud-provider=external - there was hostname mismatch (FQDN vs. hostname between what was on host and what name was in Hetzner) so maybe that's why it wasnt catched (but why? it should be fixed automatically later when I fixed stuff)
• Regarding 2 nodes I added later: I forgot to configure Kubelet with --cloud-provider=external so initially it started without. I added it later after node initialization, restarted kubelet and even rebooted node - didn't fix :(

from hcloud-cloud-controller-manager.

You should be able to manually add the taint to the affected nodes, which should retrigger the initialization by us, but I have not tested this before, if this is a production cluster, you might want to test it in another cluster first.

kubectl taint node your-node-name node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

from hcloud-cloud-controller-manager.

@apricote I just created new node with all settings correct, and it indeed created missing labels/annotations/providerID so it looks like it works properly when everything is correct, but after you do any kind of mistake, it doesn't recover...

But thank you for this tip!
Tainting node indeed retriggered initialization and now all missing stuff appears and everything is configured properly!

I'm closing this then since it's not a bug in software, but it would be good to add this to documentation.

from hcloud-cloud-controller-manager.