'dirty' nodes need /var/lib/rook deleting about cloud-native-setup HOT 13 CLOSED

@krsna1729 @ganeshmaharaj - adding to reset_stack.sh is the ideal, because that is notionally when you want to do that cleanup... but, that is harder as you have to get all the ssh setup etc.
That's why I was suggesting adding it to the setup_system.sh as a sort of 'pre-sanity clean check`, as the time this presents as a real problem is when re-installing a node, so that script would be run, and can clean up any old rook dirs before trying to re-install anew. Much easier to code up.

from cloud-native-setup.

@grahamwhaley
Is this already addressed with the reset script?
https://github.com/clearlinux/cloud-native-setup/blob/master/clr-k8s-examples/reset_stack.sh#L40

from cloud-native-setup.

from cloud-native-setup.

Yup added complexity. kubectl, awk, xargs, ssh should be enough

from cloud-native-setup.

I was thinking just last night, maybe we can add it to the setup_system.sh script, as that is run on every node. And, although you don't technically need to run it a second time on a node to re-join a cluster, in my ansible scripts I do, and we could mandate that you must run the setup_system.sh script before each join.

Or, yes, we could write a small clean_node.sh or unjoin_node.sh script that does the kubeadm reset -f on the node followed by removal of the rook lib dir. (I have an ansible that does a host side drain followed by those actions on the node(s)).

from cloud-native-setup.

@mcastelino

from cloud-native-setup.

@grahamwhaley we can just run the reset_stack.sh right? Can we close this?

from cloud-native-setup.

@krsna1729 would be nice to add logic to clean /var/lib/rook from all the nodes. Maybe find list of nodes when reset_stack.sh is called and do the ssh + delete at the end of the script after kube is reset?

from cloud-native-setup.

@grahamwhaley add it to setup_system.sh with a TODO to remove it in future. From rook documentation -

Connect to each machine and delete /var/lib/rook, or the path specified by the dataDirHostPath.
In the future this step will not be necessary when we build on the K8s local storage feature.

from cloud-native-setup.

Im not set against but seems like it should be in reset_stack.sh and called by something like Ansible for each node. IMO we should not implement multi-node management in bash just to remove /var/lib/rook. It also leaves it decoupled for user to use the tool of their choice.

from cloud-native-setup.

Should we add this in Ansible, or make the changes in the script ?

from cloud-native-setup.

We have this https://github.com/clearlinux/cloud-native-setup/blob/master/clr-k8s-examples/reset_stack.sh#L43 in reset_stack script already.

from cloud-native-setup.

closing this, please feel free to open this if still active.

from cloud-native-setup.