aliyuncontainerservice / et-operator Goto Github PK

Hi！Is it stable now? Can it be applied to a production environment? thanks！

Now launcher use ssh when attach to workers, there are some problem:

1. it needs workers open sshd
2. controller needs to create a keyPair as secret for every job
3. it makes controller hard to know whether worker training process is shutdown when scaleIn job

A better way is using kubectl exec to replace ssh.

The total available resources in cluster may change over time, it would be nice to support auto-scale if user want their training jobs to use as most resources as possible.

Common case will be:

• create a et-job with --np, --min-np and --max-np, --np tend to be small since launcher won't start if not at least np workers ready
• auto scale-out if extra worker could be created until reach --max-np
• [optional] auto scale-in if some worker been preempted or failed unexpected

If this is not the default behavior, it can be controlled by something like a scalePolicy:auto.

When testing horovod elastic worker preemption, the whole job may fail in some cases with uncertain reason.
It'd be better if operator support job level restart(restartpolicy + backofflimit).