How to run elastically on kubernetes (nnodes vs worker replicas) about elastic HOT 3 CLOSED

Hi Matthew, thank you for your question.
When a new node joins the cluster, the amount of nodes changes in the system and we need to redistribute ranks. The safes way to do that is to stop all the workers that run the user training scripts and perform rank distribution.
Please look at the example https://github.com/pytorch/elastic/blob/master/examples/imagenet/main.py to see how you can implement checkpointing now.

from elastic.

@Tierex thanks for your answer! I actually was using the imagenet code as a sanity check.

Just to make sure I understand you, when a node joins and we redistribute ranks, we use the most recent checkpoint to restore state everywhere. If there is no checkpoint yet, but we were partially through an epoch we would lose the partial progress of that epoch when ranks are redistributed.

Is that accurate?

from elastic.

Yes, @mttcnnff this is correct. Currently, torchelastic is agnostic about user scripts, as a result we cannot wait while epoch finishes when the cluster membership change occurs. Eventually, we are planning to fix it, I created a feature request to keep track of this issue: #98

from elastic.