Support using Singularity to run builds in the container image about cli HOT 5 CLOSED

Hi there

I'll look into adding it to the wrapper, but if you want to use Singularity with nextstrain you can:

1. build a singularity image (I used a specific tagged version for this, see Docker Hub for list of tags:

singularity build nextstrain-base-build-20200309T035736Z.simg 'docker://nextstrain/base:build-20200309T035736Z'

2. running the ncov pipeline from within a cloned copy of the ncov github repository, this does nextstrain build .

singularity exec -B $(pwd):/nextstrain/build nextstrain-base-build-20200309T035736Z.simg  snakemake --printshellcmds

For reference, the Docker command that this nextstrain build . runs is:

docker run --rm --tty --interactive --user=$(id -u):$(id -g) --volume=$(pwd):/nextstrain/build --workdir=/nextstrain/build --env=AUGUR_RECURSION_LIMIT --env=AUGUR_MINIFY_JSON --env=AWS_ACCESS_KEY_ID --env=AWS_SECRET_ACCESS_KEY --env=AWS_SESSION_TOKEN --env=ID3C_URL --env=ID3C_USERNAME --env=ID3C_PASSWORD --env=RETHINK_HOST --env=RETHINK_AUTH_KEY nextstrain/base snakemake --printshellcmds

from cli.

Thanks for looking into this @pvanheus! If you have any questions about how the CLI works, feel free to ask them here.

from cli.

Kim Lauer shared their experience running Nextstrain with the CLI and Docker image via Singularity. I'm copying the bulk of their message here for our future reference when thinking about how (or whether) to support Singularity:

I thought I would share how we are running the github snakemake pipeline using singularity on a cluster that does not support docker.

1. Install singularity (singularity pull nextstrain.sif docker://nextstrain/base:latest).
2. Pull latest snakemake pipeline repo (git clone https://github.com/nextstrain/ncov.git)
3. Update data files, configs, etc
4. run (singularity exec --cleanenv nextstrain.sif snakemake --profile my_profiles/<my_user_profile>)

It doesn’t use docker, but the Singularity environment + Nextstrain CLI.
The only limitations I have found so far is I can’t update the image and need to re-install when new software versions come out (when I run nextstrain update inside the container it fails) and pangolin isn’t installed inside the container so I have not been able to take advantage of the run_pangolin feature that looks like it was added in April.

from cli.

This works fairly similarly for ncov-ingest:

SINGULARITYENV_GISAID_API_ENDPOINT=https://www.epicov.org/BLAH
SINGULARITYENV_GISAID_USERNAME_AND_PASSWORD=USER:PASSWORD
singularity exec /tools/containers/nextstrain/ncov-ingest.sif snakemake --configfile config/local_gisaid.yaml

Note that the above does not make use of multiple cores and is thus slower than is necessary. Use the --cores option of snakemake to set the number of cores to use.

from cli.

@pvanheus FWIW, the just-released Nextstrain CLI 6.1.0 now supports Singularity directly. If you want to give it a go, please do report back any issues, suggestions, etc. :-)

from cli.