how to setup slurm in a cluster about lightning HOT 8 CLOSED

Glad you found it useful! I'll put a SLURM tutorial together tomorrow.
In the meantime, check out the multi-node example:

https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/examples/new_project_templates/multi_node_cluster_template.py

from lightning.

Thanks. Looking forward to see SLURM tutorial.

from lightning.

Put the tutorial together here.
Try it out and lmk what you think.

https://link.medium.com/Bs6NxJyjRY

from lightning.

Closing this because the tutorial has been created :)

But, feel free to add a comment if the tutorial is missing something or needs clarification.

from lightning.

Closing this because the tutorial has been created :)

But, feel free to add a comment if the tutorial is missing something or needs clarification.

In the blog, there is a comment:

Disclaimer: This tutorial assumes your cluster is managed by SLURM.

In fact, I have no idea about how to manage cluster using slurm...

When cluster is managed by slurm, I can try PTL in my cluster.

from lightning.

SLURM is installed by your cluster administrator. I guess here I’m using cluster to refer to an academic or corporate cluster.

If it’s literally your own cluster with your own machines then PTL may not support that case because it uses NCCL to communicate and a few flags SLURM sets.

But if it’s an academic or corporate cluster, it most likely has SLURM installed.

If you use srun or sbatch to run jobs, then you’re using SLURM.

from lightning.

Yes, now I want to try slurm by myself...

from lightning.

I did not feel SLURM convenient than Horovod.

from lightning.