Issue with StatefulSet Rolling Update Strategy about libcluster HOT 10 OPEN

I think you would have the same issue even if you use deployment, new pods would crash-loop anyways.

The way how we solved this is: we have our own implementation of the Kubernetes strategy that polls k8s API and only joins nodes of the same version (from the version label) into a cluster. This makes it impossible to handoff state between application versions but makes sure that code that was never tested to co-live in a cluster would end up crashing in production.

from libcluster.

@AndrewDryga do you think this custom k8s strategy is worth a PR or can be shared ? Because i feel like how are more people not running into this same issue ? Is everyone else using this library only every deploying libcluster 1 time and then never adding new features into its registry from that point forward ?

from libcluster.

@amacciola the problem with our strategy is that it is very opinionated (uses specific labels named for our environment, uses node names from k8s labels, etc). I will think about open-sourcing it but it's a relatively easy change: just leverage labelSelector in get_nodes/4 callback implementation and query only the pods of a specific version.

from libcluster.

@AndrewDryga okay i will try this. So if i am trying to extend the k8s dns strategy here:
https://github.com/bitwalker/libcluster/blob/main/lib/strategy/kubernetes_dns.ex

you are suggestion that i need to tweak the get_nodes method here:

to also additionally query for a specific version or at least only matching version numbers ?

from libcluster.

@amacciola you can't extract that information from DNS server, instead you should modify that function in lib/strategy/kubernetes.ex. K8s API returns a lot of information about the pod including labels that you need to use to store the version.

from libcluster.

@AndrewDryga i see. So its changing

so include additional params to only return info for certain version info

from libcluster.

@amacciola yes, you want to query for pods and return only the ones that match your current version

from libcluster.

@AndrewDryga i am working on testing this new strategy out now so thanks for the insight. But i just wanted to make sure i understood how some of the Libcluster combined with Horde registry code is working under the hood.

If we have 3 pods running for the same application. Each of these pods have lets say server_1 registered in the HordeRegistry.

• pod_1 with version_1
• pod_2 with version_1 -> running server_1
• pod_3 with version_1

If we then trigger an update for version_2, which contains a new service that gets registered in the Horde Registry, server_2. Lets say the update starts with pod_3. When pod_3 comes online with the new version and finds pod_1 and pod_2 .

Does it just pick a process_id from one of the 3 pods to to try and start the new service on ? So you will have a 2 in 3 chance it tries to start the new service on a pod with version_1 and not version_2. ???

from libcluster.

I'm not using Horde but the pods with version 1 would not see pods with version 2 in the Erlang cluster, so basically, for each of the islands (one per version), everything would behave like it's a cluster with the same codebase. If you have globally unique jobs it also means that you will have two of the workers started (one per island).

from libcluster.

For now i have just created a separate HordeRegistry for each Genserver we would want to be leveraging the Libcluster strategies. As long as we dont have too many its a minor annoyance to fix this issue

from libcluster.