This profile configures Snakemake to run on the SLURM Workload Manager
To deploy this profile, run
mkdir -p ~/.config/snakemake
cd ~/.config/snakemake
cookiecutter https://github.com/Snakemake-Profiles/slurm.git
Then, you can run Snakemake with
snakemake --profile slurm ...
profile_name
: A name to address the profile via the--profile
Snakemake option.sbatch_defaults
: List of default arguments to sbatch, e.g.:qos=short time=60
. This is a convenience argument to avoidcluster_config
for a few aruments.cluster_config
: Path to a YAML or JSON configuration file analogues to the Snakemake--cluster-config
option. Path may relative to the profile directory or absolute including environment variables (e.g.$PROJECT_ROOT/config/slurm_defaults.yaml
).advanced_argument_conversion
: If True, try to adjust/constrain mem, time, nodes and ntasks (i.e. cpus) to parsed or default partition after converting resources. This may fail due to heterogeneous slurm setups, i.e. code adjustments will likely be necessary.
Default arguments to snakemake
maybe adjusted in the <profile path>/config.yaml
file.
Arguments are overridden in the following order and must be named according to sbatch long option names:
sbatch_defaults
cookiecutter option- Profile
cluster_config
file__default__
entries - Snakefile threads and resources (time, mem)
- Profile
cluster_config
file entries --cluster-config
parsed to Snakemake (deprecated since Snakemake 5.10)- Any other argument conversion (experimental, currently time, ntasks and mem) if
advanced_argument_conversion
is True.
Resources specified in Snakefiles must all be in the correct unit/format as expected by sbatch
.
The implemented resource names are given (and may be adjusted) in the slurm_utils.RESOURCE_MAPPING
global.
This is intended for system agnostic resources such as time and memory.
The profile supports setting default and per-rule options in either the cluster_config
file and
the --cluster-config
file parsed to snakemake
(the latter is deprecated since snakemake 5.10). The __default__
entry will apply to all jobs. Both may be YAML (see example
below) or JSON files.
__default__:
account: staff
mail-user: [email protected]
large_memory_requirement_job:
constraint: mem2000MB
ntasks: 16
Test-driven development is enabled through the tests folder. Provided
that the user has installed docker and enabled docker
swarm (docker swarm init
), the SLURM tests will
download two images:
and testing of the cookiecutter template itself is enabled through the pytest plugin for Cookiecutters. You can run the tests by issuing
pytest -v -s
- major rewrite and merge of the
slurm-submit.py
script to support any sbatch argument - parse any argument via the
sbatch_defaults
option and - enable per-profile cluster (YAML/JSON) config file
- make experimental sbatch argument adjustments optional via the
advanced_argument_conversion
option
- add qos option
- replace pytest_namespace with pytest_configure
- make days optional (#18)
- add cookiecutter options to set sbatch output and error defaults
- add support for mem_mb in resources
- add support for cluster configuration file
- add advanced slurm-submit file
- adjust resource requirements if they exceed partition configuration settings (#11)