Losing quorum as soon as a node goes down about piraeus HOT 4 OPEN

Have you checked with drbdsetup status on the remaining nodes that they indeed have quorum? If they do have it, it seems like a bug in the HA controller.

from piraeus.

Yes, they do lose quorum. For example just now :

pvc-e57930e5-6772-41e4-8c98-99105b77970a role:Secondary suspended:quorum
  disk:UpToDate quorum:no blocked:upper
  talos-00r-fu9 role:Secondary
    peer-disk:Diskless
  talos-813-fn2 connection:Connecting

It has an UpToDate and a Diskless node, and yet it thinks it lost quorum. That's the only volume that lost quorum, the other ones look the same but with quorum, and the local node became Primary, maybe it's something to do with that specific volume somehow

from piraeus.

Very weird. Probably something for the DRBD folks to look at.

If you just want to disable the taints, you can disable the HA Controller since 2.3.0: https://github.com/piraeusdatastore/piraeus-operator/blob/v2/docs/reference/linstorcluster.md#spechighavailabilitycontroller

from piraeus.

It looks like the TieBreaker / Diskless node doesn't count towards the quorum when changing primary, so if the Primary for a volume goes down (even cleanly, it appears) the other one can't become primary anymore, and goes into a lost quorum state.

That is probably a drbd issue, but when the primary goes down cleanly I wonder if the operator could make sure the secondary switches first, while it has quorum ?
Or maybe I should just go to a placement count of 3 to avoid this

from piraeus.