Need Process to Update Software Safely about terraform-aws-quorum-cluster HOT 4 CLOSED

One idea I've formed is to send a signal to each piece of software that needs upgrading, tell it to shutdown, then either

• replace it with the upgraded software. Replacing the older version directly with the newer version assumes that the upgraded software can work with and/or upgrade the data from the older version, or
• add the new software to another path, then upgrade the scripts that launch these software. This means
  • supporting 2 binary paths, ie, an A/B path, one for the older version, and one for the newer version.
  • supporting 2 data paths, one for the older version, and one for the newer version.
  • once upgrade is deemed successful, the older binary and data can either be kept/archived, or deleted. This option of having an A/B path takes into account that upgrade can fail.

This means one of the following:

• deploying an agent (like a software upgrade agent) on every node, which listens on a port for commands, then execute these commands, or
• use ssh to contact every node, and send a command to shutdown Quorum, Constellation, Vault, etc, or
• a combination of the above.

Ensure the following:

• each node's security, so that its software doesn't just listen to any Tom, Dick, Harry's command to shutdown, otherwise, the nodes could suffer from a DDOS attack.
• less than 1/2 of all validator nodes are down in 900 seconds, in order not to risk consensus failure. Is there a way to get node statistics? This can help in deciding to bring nodes down for upgrading.
• PATH in environment is upgraded, if necessary.
• launch scripts are upgraded.
• optionally, trigger a backup after the upgrade is completed.

Need to investigate whether each software can support a soft signal that handles an orderly shutdown, or if each software can recover from a fatal "kill" command.

The custom scripts (refer to Blockchain Lecture, pg 8-9 under Tech Stack) can be replaced directly using cp, or mv.

from terraform-aws-quorum-cluster.

@EximChua once you have determined the design you're going with (which you may or may not have already done) can you write up the details of your design?

Remember the program is run with supervisord, so you will be able to leverage that for starting and stopping the software that needs updating.

from terraform-aws-quorum-cluster.

Implemented as SoftwareUpgrade.

from terraform-aws-quorum-cluster.

I can't seem to close this issue.

from terraform-aws-quorum-cluster.