Split brain issue about puppet-couchbase HOT 9 OPEN

Hi sorry for the long response time. I've had some personal stuff recently. In theory if you could start literally 20 nodes at once that could theoretically result in a split brain in the configuration, yes. At the same time though I don't know of a way that would work in practice. As long as you have existing nodes in the cluster they will pick up the new members and add them and migrate, but that won't happen all at the same time just due to the way that puppet would not be able to run them all with the same timing.

Have you run into an issue specifically with this? I can also try to test something myself.

from puppet-couchbase.

I tested myself and split brain happens in 99% of cases :(

On 4 Mar 2016 01:21 +0100, Justice [email protected], wrote:

Hi sorry for the long response time. I've had some personal stuff recently. In theory if you could start literally 20 nodes at once that could theoretically result in a split brain in the configuration, yes. At the same time though I don't know of a way that would work in practice. As long as you have existing nodes in the cluster they will pick up the new members and add them and migrate, but that won't happen all at the same time just due to the way that puppet would not be able to run them all with the same timing.

Have you run into an issue specifically with this? I can also try to test something myself.

—
Reply to this email directly orview it on GitHub(#34 (comment)).

from puppet-couchbase.

Huh. Weird. I'll look into it some more.

from puppet-couchbase.

I think this is the key point:

As long as you have existing nodes in the cluster

In the case of spawning a completely new cluster (not adding to an existing one with nodes already) with 20 new VMs this is very likely to happen as the VMs come up simultaneously.

from puppet-couchbase.

@dfairhurst Fair enough. I'll work on engineering a solution for that particular problem.

from puppet-couchbase.

Thoughts on waiting random T seconds (i.e. sleep $(/usr/bin/shuf -i 1000-10000 -n 1)) in the module before starting/joining the cluster?

from puppet-couchbase.

Good idea! I'll consider how to best implement this that won't fall afoul of timeouts for exec, etc.

from puppet-couchbase.

Well, assuming this actually works, what about adding it to the couchbasenode.erb template such that subsequent entries would render as follows:

#!/bin/bash

touch /opt/couchbase/var/.installed

#Server node configurations below
/opt/couchbase/bin/couchbase-cli rebalance -c localhost -u couchbase -p 'password' --server-add=couchbase01.example.com --server-add-username=couchbase --server-add-password='password'
/usr/bin/sleep $(/usr/bin/shuf -i 500-10000 -n 1)

/opt/couchbase/bin/couchbase-cli rebalance -c localhost -u couchbase -p 'password' --server-add=couchbase02.example.com --server-add-username=couchbase --server-add-password='password'
/usr/bin/sleep $(/usr/bin/shuf -i 500-10000 -n 1)

from puppet-couchbase.

Also worth noting:

DEPRECATED: Adding server from the rebalance command is deprecated and will be removed in future release, use the server-add command to add servers instead.

I was originally looking for a --wait option like they have for bucket-create but now I'm curious if server-add behaves any differently in mitigating this same issue more natively.

from puppet-couchbase.