repronim / neurodocker Goto Github PK

View Code? Open in Web Editor NEW

321.0 18.0 96.0 6.08 MB

Generate custom Docker and Singularity images, and minimize existing containers

Home Page: https://www.repronim.org/neurodocker/

License: Apache License 2.0

Python 97.63% Shell 2.17% Dockerfile 0.20%

neuroscience docker nipype reproducible-research singularity-container science containers linux dockerfile apptainer

neurodocker's Introduction

Neurodocker

Neurodocker is a command-line program that generates custom Dockerfiles and Singularity recipes for neuroimaging and minifies existing containers.

Please see our website https://www.repronim.org/neurodocker for more information.

See our list of supported software

Installation

Use the Neurodocker Docker image (recommended):

docker run --rm repronim/neurodocker:latest --help

The Docker images were moved to repronim/neurodocker from kaczmarj/neurodocker.

This project can also be installed with pip:

pip install neurodocker
neurodocker --help

If the pip install command above gives a permissions error, install as a non-root user:

pip install --user neurodocker

Note: it is not yet possible to minimize Docker containers using the Neurodocker Docker image.

Developer installation

Clone the repository and install in editable mode.

git clone https://github.com/ReproNim/neurodocker
cd neurodocker
python -m pip install --no-cache-dir --editable .[all]

Before committing changes, initialize pre-commit with pre-commit install. This will format code with each commit to keep the style consistent. Neurodocker uses black for formatting.

Build status

You can check the status of the build of the Docker images for several of the neuroimaging software packages that are supported by Neurodocker on this page.

neurodocker's People

Contributors

Stargazers

Watchers

Forkers

djarecka smpadhy yarikoptic schahid sulantha2006 gkiar shotgunosine zhenfenghan nagyistge ross-mitchell arokem nih-fmrif giovtorres mvdoc mjtravers afni-rickr dhinkris kdestasio tjhrad augix effigies ericearl ingo-m andrewbeers araikes jona-sassenhagen ipadawan spolex tknapen ltirrell mattcieslak tjhendrickson paul-asvb satra ucsb-vrl msharrock dddd1007 daelsaid cdla timvanmourik vinferrer jjleewustledu devhliu tclose peerherholz wendy1013 tashrifbillah nagedem m9h jdkent skalia05 gllmflndn handsofthecommunity gdevenyi jsmentch juliaschraeder pwighton aswinnarayanan stebo85 jishanling marcelfarres acpuche aecelaya seedpcseed pennsive georgebuzzell andysworth jakepalmer stijnimaging ghisvail a3sha2 akemisetti celstark noxtoby tomemotion alexisthual djoerch edilbertoamorim spisakt jennydaman remi-gau testingiacwithnewaccount ajaykurani sooyounga hpf200326 dnkennedy hjbockholt phcerdan alafia-ai vbitz thechymera

neurodocker's Issues

Zenodo DOI

Could you please create a Zenodo DOI? I would like to be able to cite neurodocker.

Freesurfer pipeline fails inside neurodocker image because of missing perl

Hi,
I generated a docker image with this command:
neurodocker generate -b centos:7 -p yum --freesurfer version=6.0.0 > Dockerfile

another command to generate Dockerfile with miniconda (2 with python2.7) and pasted the docker commands again in the docker file generated with the first command. At the end, installed my freesurfer pipeline built with nipype.

The freesurfer command nu_correct failed because it could not find perl.
Therefore, I added perl in the yum commands to install perl. Then everything went fine.

The generated-full.Dockerfile in examples folder is also missing perl.

So I think perl should be installed as a dependency before installing freesurfer.

Thanks a lot,

bug: activate=false still activates conda environment

It seems like activate=false is being interpreted by neurodocker as activate=true.

allow creation of conda environment from yml file

This command will create a conda environment from a yml file. This is not possible right now, except with a custom RUN instruction.

conda env create --file path/to/env.yml

add support for minimizing a container for particular commands

see freesurfer/freesurfer#70 for creating a reprozip pack file.

add support for latest ants release

https://github.com/stnava/ANTs/releases

skip some tests when external dependency is not present

may be it was already implemented and intended, I am just presenting my observation:

$> python -m pytest -v -k "not test_build_image" neurodocker 
================================================================= test session starts ==================================================================
platform linux -- Python 3.5.4, pytest-3.2.1, py-1.4.34, pluggy-0.4.0 -- /home/yoh/deb/perspect/neurodocker/venvs/dev/bin/python
cachedir: .cache
rootdir: /home/yoh/deb/perspect/neurodocker, inifile:
plugins: localserver-0.3.7, hypothesis-3.44.1
collected 66 items                                                                                                                                      

neurodocker/docker/tests/test_docker.py::test_docker_is_running PASSED
neurodocker/docker/tests/test_docker.py::TestBuildOutputLogger::test_start PASSED
neurodocker/docker/tests/test_docker.py::TestBuildOutputLogger::test_get_logs PASSED
neurodocker/docker/tests/test_docker.py::TestDockerImage::test___init__ PASSED
neurodocker/docker/tests/test_docker.py::TestDockerImage::test_build PASSED
neurodocker/docker/tests/test_docker.py::TestDockerContainer::test_start_cleanup PASSED
neurodocker/docker/tests/test_docker.py::TestDockerContainer::test_exec_run PASSED
neurodocker/docker/tests/test_docker.py::TestDockerContainer::test_cleanup PASSED
neurodocker/docker/tests/test_docker.py::test_copy_file_from_container PASSED
neurodocker/docker/tests/test_docker.py::test_copy_file_to_container PASSED
neurodocker/interfaces/tests/test_afni.py::TestAFNI::test_invalid_binaries PASSED
neurodocker/interfaces/tests/test_ants.py::TestANTs::test_invalid_binaries PASSED
neurodocker/interfaces/tests/test_ants.py::TestANTs::test_build_from_source_github PASSED
neurodocker/interfaces/tests/test_freesurfer.py::TestFreeSurfer::test_copy_license PASSED
neurodocker/interfaces/tests/test_mrtrix.py::TestMRtrix3::test_build_from_source PASSED
neurodocker/reprozip/tests/test_merge.py::test_merge_pack_files FAILED
neurodocker/reprozip/tests/test_trace.py::test_ReproZipMinimizer_no_ptrace SKIPPED
neurodocker/reprozip/tests/test_trace.py::test_ReproZipMinimizer PASSED
neurodocker/tests/test_generate.py::test__add_add PASSED
neurodocker/tests/test_generate.py::test__add_to_entrypoint PASSED
neurodocker/tests/test_generate.py::test__add_arg PASSED
neurodocker/tests/test_generate.py::test__add_base PASSED
neurodocker/tests/test_generate.py::test__add_cmd PASSED
neurodocker/tests/test_generate.py::test__add_copy PASSED
neurodocker/tests/test_generate.py::test__add_entrypoint PASSED
neurodocker/tests/test_generate.py::test__add_env_vars FAILED
neurodocker/tests/test_generate.py::test__add_exposed_ports PASSED
neurodocker/tests/test_generate.py::test__add_install PASSED
neurodocker/tests/test_generate.py::test__add_arbitrary_instruction PASSED
neurodocker/tests/test_generate.py::test__add_label PASSED
neurodocker/tests/test_generate.py::test_add_run PASSED
neurodocker/tests/test_generate.py::test__add_run_bash PASSED
neurodocker/tests/test_generate.py::test__add_volume PASSED
neurodocker/tests/test_generate.py::test__add_workdir PASSED
neurodocker/tests/test_generate.py::test_DockerfileUsers PASSED
neurodocker/tests/test_generate.py::TestDockerfile::test___repr__ PASSED
neurodocker/tests/test_generate.py::TestDockerfile::test___str__ PASSED
neurodocker/tests/test_generate.py::TestDockerfile::test__create_cmd PASSED
neurodocker/tests/test_generate.py::TestDockerfile::test_save PASSED
neurodocker/tests/test_neurodocker.py::test_generate PASSED
neurodocker/tests/test_neurodocker.py::test_generate_opts FAILED
neurodocker/tests/test_neurodocker.py::test_generate_from_json PASSED
neurodocker/tests/test_neurodocker.py::test_generate_no_print PASSED
neurodocker/tests/test_neurodocker.py::test_generate_save PASSED
neurodocker/tests/test_parser.py::TestSpecsParser::test__validate_keys PASSED
neurodocker/tests/test_utils.py::test_manage_pkgs PASSED
neurodocker/tests/test_utils.py::test_check_url PASSED
neurodocker/tests/test_utils.py::test_indent PASSED
neurodocker/tests/test_utils.py::test_save_load_json PASSED
neurodocker/tests/test_utils.py::test_set_log_level PASSED
neurodocker/tests/test_version.py::test_get_gitversion PASSED

================================================== FAILURES ===================================================
____________________________________________ test_merge_pack_files ____________________________________________

    def _check_deps():
        """Raise RuntimeError if a dependency is not found. These dependencies are
        not included in `requirements.txt`.
        """
        import shutil
    
        msg = "Dependency '{}' not found."
    
        if shutil.which('rsync') is None:
            raise RuntimeError(msg.format('rsync'))
        try:
>           import reprozip
E           ImportError: No module named 'reprozip'

neurodocker/reprozip/merge.py:47: ImportError

During handling of the above exception, another exception occurred:

    def test_merge_pack_files():
        tmpdir = tempfile.mkdtemp()
    
        cmd = ["du -sh /usr", "rm --help"]
        packpath = _create_packfile(cmd, tmpdir)
        new_name = "first-pack.rpz"
        os.rename(packpath, os.path.join(tmpdir, new_name))
    
        cmd = ["ls -l /", "grep --help"]
        _create_packfile(cmd, tmpdir)
    
        pattern = os.path.join(tmpdir, '*.rpz')
        packfiles = glob(pattern)
        assert packfiles, "packfiles not found"
    
        outfile = os.path.join(tmpdir, 'merged.rpz')
>       merge_pack_files(outfile=outfile, packfiles=packfiles)

neurodocker/reprozip/tests/test_merge.py:48: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
neurodocker/reprozip/merge.py:165: in merge_pack_files
    _check_deps()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def _check_deps():
        """Raise RuntimeError if a dependency is not found. These dependencies are
        not included in `requirements.txt`.
        """
        import shutil
    
        msg = "Dependency '{}' not found."
    
        if shutil.which('rsync') is None:
            raise RuntimeError(msg.format('rsync'))
        try:
            import reprozip
        except Exception:
>           raise RuntimeError(msg.format('reprozip'))
E           RuntimeError: Dependency 'reprozip' not found.

neurodocker/reprozip/merge.py:49: RuntimeError
_____________________________________________ test__add_env_vars ______________________________________________

    def test__add_env_vars():
        env = {'THIS': 'THAT'}
        truth = 'ENV THIS="THAT"'
        assert truth == DF._add_env_vars(env)
    
        env['A'] = 'B'
        truth = ('ENV THIS="THAT" \\\n    A="B"')
>       assert truth == DF._add_env_vars(env)
E       assert 'ENV THIS="TH...\\\n    A="B"' == 'ENV A="B" \\\n    THIS="THAT"'
E         - ENV THIS="THAT" \
E         -     A="B"
E         + ENV A="B" \
E         +     THIS="THAT"

neurodocker/tests/test_generate.py:72: AssertionError
_____________________________________________ test_generate_opts ______________________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x7fd553b11ba8>

    def test_generate_opts(capsys):
        args = "generate -b ubuntu:17.04 -p apt --no-check-urls {}"
        main(args.format('--user=neuro').split())
        out, _ = capsys.readouterr()
        assert "USER neuro" in out
    
        main(args.format('--add path/to/file.txt /tmp/file.txt').split())
        out, _ = capsys.readouterr()
        assert 'ADD ["path/to/file.txt", "/tmp/file.txt"]' in out
    
        main(args.format('--copy path/to/file.txt /tmp/file.txt').split())
        out, _ = capsys.readouterr()
        assert 'COPY ["path/to/file.txt", "/tmp/file.txt"]' in out
    
        main(args.format('--env KEY=VAL KEY2=VAL').split())
        out, _ = capsys.readouterr()
>       assert 'ENV KEY="VAL" \\' in out
E       assert 'ENV KEY="VAL" \\' in '# Generated by Neurodocker v0.3.2-5-gf1376bf.\n#\n# Thank you for using Neurodocker. If you discover any issues\n# or...bf", \\\n    \\n  "generation_timestamp": "2018-01-12 10:42:29" \\\n    \\n}\' > /neurodocker/neurodocker_specs.json\n'

neurodocker/tests/test_neurodocker.py:71: AssertionError
============================================= 15 tests deselected =============================================
======================= 3 failed, 47 passed, 1 skipped, 15 deselected in 236.14 seconds =======================
python -m pytest -v -k "not test_build_image" neurodocker  12.78s user 0.19s system 5% cpu 3:56.79 total

it is on a Debian system in a virtualenv with python 3.5

ANTs 2.2.0 version string

The current ANTs version string is 2.1.0.post789-g0740f. I did ensure that the ANTs version in nipype will accept that string as equivalent to 2.2.0, but it would probably be a good idea to fix this, if this framework permits patching.

Add ability to install hcpre

Hello,

I am very interested in neurodocker, however, I have interest in utilzing nipype in order to run my data through the human connectome project minimal preprocessing pipeline. Another developer generated a package named hcpre to piggyback off of nipype's existing infrastructure.

-Tim

add dcm2niix

FreeSurfer 6.0.1

FreeSurfer just released 6.0.1, which is just 6.0.0 with the ability to specify the location of a LICENSE file as an environment variable (freesurfer/freesurfer@release_6_0_0...stable6), which was added to make containerized FreeSurfer easier to use without baking in a license.

I know NeuroDocker's done some work on handling licenses already, but I wanted to know if you were aware of this release and planning to add support in the near future?

ADD: ability to lock versions

When installing new software, a package manager might upgrade or downgrade software on the system. This is not ideal for reproducibility. It is possible to "lock" versions of software with apt-get and yum.

apt-get: https://askubuntu.com/a/18656
yum: https://access.redhat.com/solutions/98873

dockerfile searches for a centos fsl while the base is ubuntu / missing libmng1 during building

Hello,
I am using neurodocker to make an image with the fsl as follows:
sudo docker run --rm kaczmarj/neurodocker:master generate --base ubuntu:16.04 --pkg-manager apt --fsl version=5.0.10

Even though my base is ubuntu 16.04 I 've noticed that the dockerfile asks for the centos version of fsl.
as seen below:

# Generated by Neurodocker v0.3.1-21-ged92d36.
#
# Thank you for using Neurodocker. If you discover any issues
# or ways to improve this software, please submit an issue or
# pull request on our GitHub repository:
#     https://github.com/kaczmarj/neurodocker
#
# Timestamp: 2017-11-13 16:48:47

FROM ubuntu:16.04

ARG DEBIAN_FRONTEND=noninteractive

#----------------------------------------------------------
# Install common dependencies and create default entrypoint
#----------------------------------------------------------
ENV LANG="en_US.UTF-8" \
    LC_ALL="C.UTF-8" \
    ND_ENTRYPOINT="/neurodocker/startup.sh"
RUN apt-get update -qq && apt-get install -yq --no-install-recommends  \
    	apt-utils bzip2 ca-certificates curl locales unzip \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \
    && localedef --force --inputfile=en_US --charmap=UTF-8 C.UTF-8 \
    && chmod 777 /opt && chmod a+s /opt \
    && mkdir -p /neurodocker \
    && if [ ! -f "$ND_ENTRYPOINT" ]; then \
         echo '#!/usr/bin/env bash' >> $ND_ENTRYPOINT \
         && echo 'set +x' >> $ND_ENTRYPOINT \
         && echo 'if [ -z "$*" ]; then /usr/bin/env bash; else $*; fi' >> $ND_ENTRYPOINT; \
       fi \
    && chmod -R 777 /neurodocker && chmod a+s /neurodocker
ENTRYPOINT ["/neurodocker/startup.sh"]

#-----------------------------------------------------------
# Install FSL v5.0.10
# FSL is non-free. If you are considering commerical use
# of this Docker image, please consult the relevant license:
# https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Licence
#-----------------------------------------------------------
RUN apt-get update -qq && apt-get install -yq --no-install-recommends bc dc libfontconfig1 libfreetype6 libgl1-mesa-dev libglu1-mesa-dev libgomp1 libice6 libmng1 libxcursor1 libxft2 libxinerama1 libxrandr2 libxrender1 libxt6 \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \
    && echo "Downloading FSL ..." \
    && curl -sSL https://fsl.fmrib.ox.ac.uk/fsldownloads/fsl-5.0.10-centos6_64.tar.gz \
    | tar zx -C /opt \
    && /bin/bash /opt/fsl/etc/fslconf/fslpython_install.sh -q -f /opt/fsl \
    && sed -i '$iecho Some packages in this Docker container are non-free' $ND_ENTRYPOINT \
    && sed -i '$iecho If you are considering commercial use of this container, please consult the relevant license:' $ND_ENTRYPOINT \
    && sed -i '$iecho https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Licence' $ND_ENTRYPOINT \
    && sed -i '$isource $FSLDIR/etc/fslconf/fsl.sh' $ND_ENTRYPOINT
ENV FSLDIR=/opt/fsl \
    PATH=/opt/fsl/bin:$PATH

#--------------------------------------
# Save container specifications to JSON
#--------------------------------------
RUN echo '{ \
    \n  "pkg_manager": "apt", \
    \n  "check_urls": true, \
    \n  "instructions": [ \
    \n    [ \
    \n      "base", \
    \n      "ubuntu:16.04" \
    \n    ], \
    \n    [ \
    \n      "fsl", \
    \n      { \
    \n        "version": "5.0.10" \
    \n      } \
    \n    ] \
    \n  ], \
    \n  "generation_timestamp": "2017-11-13 16:48:47", \
    \n  "neurodocker_version": "0.3.1-21-ged92d36" \
    \n}' > /neurodocker/neurodocker_specs.json

When I try to build the image I get the following error for libmng1 that cannot be found


sudo docker run --rm kaczmarj/neurodocker:master generate --base ubuntu:16.04 --pkg-manager apt --fsl version=5.0.10 | sudo docker build -
Sending build context to Docker daemon   5.12kB
Step 1/8 : FROM ubuntu:16.04
 ---> dd6f76d9cc90
Step 2/8 : ARG DEBIAN_FRONTEND=noninteractive
 ---> Using cache
 ---> f94a0784a4bc
Step 3/8 : ENV LANG "en_US.UTF-8" LC_ALL "C.UTF-8" ND_ENTRYPOINT "/neurodocker/startup.sh"
 ---> Using cache
 ---> ac4dd3777e7f
Step 4/8 : RUN apt-get update -qq && apt-get install -yq --no-install-recommends      	apt-utils bzip2 ca-certificates curl locales unzip     && apt-get clean     && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*     && localedef --force --inputfile=en_US --charmap=UTF-8 C.UTF-8     && chmod 777 /opt && chmod a+s /opt     && mkdir -p /neurodocker     && if [ ! -f "$ND_ENTRYPOINT" ]; then          echo '#!/usr/bin/env bash' >> $ND_ENTRYPOINT          && echo 'set +x' >> $ND_ENTRYPOINT          && echo 'if [ -z "$*" ]; then /usr/bin/env bash; else $*; fi' >> $ND_ENTRYPOINT;        fi     && chmod -R 777 /neurodocker && chmod a+s /neurodocker
 ---> Using cache
 ---> 9803b908162c
Step 5/8 : ENTRYPOINT /neurodocker/startup.sh
 ---> Using cache
 ---> 3608f3650b39
Step 6/8 : RUN apt-get update -qq && apt-get install -yq --no-install-recommends bc dc libfontconfig1 libfreetype6 libgl1-mesa-dev libglu1-mesa-dev libgomp1 libice6 libmng1 libxcursor1 libxft2 libxinerama1 libxrandr2 libxrender1 libxt6     && apt-get clean     && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*     && echo "Downloading FSL ..."     && curl -sSL https://fsl.fmrib.ox.ac.uk/fsldownloads/fsl-5.0.10-centos6_64.tar.gz     | tar zx -C /opt     && /bin/bash /opt/fsl/etc/fslconf/fslpython_install.sh -q -f /opt/fsl     && sed -i '$iecho Some packages in this Docker container are non-free' $ND_ENTRYPOINT     && sed -i '$iecho If you are considering commercial use of this container, please consult the relevant license:' $ND_ENTRYPOINT     && sed -i '$iecho https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Licence' $ND_ENTRYPOINT     && sed -i '$isource $FSLDIR/etc/fslconf/fsl.sh' $ND_ENTRYPOINT
 ---> Running in bb5c40141f2e
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package libmng1
The command '/bin/sh -c apt-get update -qq && apt-get install -yq --no-install-recommends bc dc libfontconfig1 libfreetype6 libgl1-mesa-dev libglu1-mesa-dev libgomp1 libice6 libmng1 libxcursor1 libxft2 libxinerama1 libxrandr2 libxrender1 libxt6     && apt-get clean     && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*     && echo "Downloading FSL ..."     && curl -sSL https://fsl.fmrib.ox.ac.uk/fsldownloads/fsl-5.0.10-centos6_64.tar.gz     | tar zx -C /opt     && /bin/bash /opt/fsl/etc/fslconf/fslpython_install.sh -q -f /opt/fsl     && sed -i '$iecho Some packages in this Docker container are non-free' $ND_ENTRYPOINT     && sed -i '$iecho If you are considering commercial use of this container, please consult the relevant license:' $ND_ENTRYPOINT     && sed -i '$iecho https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Licence' $ND_ENTRYPOINT     && sed -i '$isource $FSLDIR/etc/fslconf/fsl.sh' $ND_ENTRYPOINT' returned a non-zero code: 100

Is this missing library a necessary library or I can just remove it from the dockerfile?
How can I generate the dockerfile that will have the correct ubuntu version of fsl instead of the centos that the neurodocker creates?

Thank you

add mrtrix3

Add support for mrtrix.

Binaries are available here (immediate download). Compiled on CentOS 6.6 without GUI libraries. Packaged with package_mrtrix.

Consider uploading those binaries to DropBox, in case the original uploader decides to move or delete them.

Also consider building mrtrix on CenOS 5.11 to support older systems.

consider ways of automatically building containers

It would be nice to push a generated (Singularity) file to so Singularity Hub can build the image automatically.

Prepend apt-get install and yum install with eatmydata

eatmydata should speed up system package installations.

looks like it is not available through yum. try building from source on a centos image and prepending it to yum install.

no latest tag on Dockerhub

None of the images on dockerhub are tagged latest, so the commands in the readme starting: docker run --rm kaczmarj/neurodocker fail, at least on Docker 17.09.0-ce-mac32 (19506).

add labels to Dockerfiles

Use the Dockerfile LABEL instruction to attach metadata to the image. At a minimum, include software and versions. Consider adding a key in the dictionary of specifications to allow user to add custom labels.

LABEL replaces the MAINTAINER instruction.

Download shared dependencies at beginning of Dockerfile

Certain packages are required to download and install most of the software (e.g., curl). Download the common dependencies at the beginning of the Dockerfile to avoid having to re-download them for each software package.

map keywords to available flags

For example, add a keyword for the flag --no-install-recommends in apt-get install.

It would be good to implement a solution that does not require hard-coding of the flags.

Add the option flags to all relevant cli options.

neurodocker generate -b debian:stretch -p apt --install git flags="--no-install-recommends --reinstall"

Pipe logs to file on host

Unable to locate package libmng1 in ubuntu

Unable to locate package libmng1 error when trying to install fsl using ubuntu 17.04 as a base image

add: --run-bash option to neurodocker generate

Add option --run-bash (or --bash-instruction) to run some bash code. The only way to do this now is with --instruction "RUN bash -c \" ... \"", which is clunky.

add git hash to dev version image

$ docker run --rm kaczmarj/neurodocker -V
neurodocker version 0.2.0-dev

also make versioned releases on docker hub with appropriate lables.

kaczmarj/neurodocker:0.2.0
kaczmarj/neurodocker:master

if you haven't already, you can automate the builds on pushes to master.

Run all docker image tests as non-root user.

currently, tests within docker images are being run as root. instead, run them as a non-root user.

Readme example fails

The example under "Generate Dockerfile (full)" fails with:

neurodocker: error: unrecognized arguments: --dcm2niix version=latest

add ability to install R

kwargs_from_env() got an unexpected keyword argument 'timeout'

decided to try a sample from README.md but didn't quite work out:

$> python sample.py
Traceback (most recent call last):
  File "sample.py", line 1, in <module>
    from neurodocker import Dockerfile, SpecsParser
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/__init__.py", line 18, in <module>
    import neurodocker.docker
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/docker/__init__.py", line 9, in <module>
    from neurodocker.docker.docker import client, DockerContainer, DockerImage
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/docker/docker.py", line 33, in <module>
    client = docker.from_env(timeout=30)
  File "/usr/lib/python2.7/dist-packages/docker/client.py", line 39, in from_env
    return Client.from_env(**kwargs)
  File "/usr/lib/python2.7/dist-packages/docker/client.py", line 113, in from_env
    return cls(**kwargs_from_env(**kwargs))
TypeError: kwargs_from_env() got an unexpected keyword argument 'timeout'

$> cat sample.py
from neurodocker import Dockerfile, SpecsParser
from neurodocker.docker import DockerImage, DockerContainer

specs = {
    'base': 'ubuntu:17.04',
    'pkg_manager': 'apt',
    'check_urls': False,  # Verify communication with URLs used in build.
    'miniconda': {
        'python_version': '3.5.1',
        'conda_install': 'traits',
        'pip_install': 'https://github.com/nipy/nipype/archive/master.tar.gz'},
    'mrtrix3': {'use_binaries': False},
    'ants': {'version': '2.2.0', 'use_binaries': True},
    'fsl': {'version': '5.0.10', 'use_binaries': True},
    'spm': {'version': '12', 'matlab_version': 'R2017a'},
}

parser = SpecsParser(specs)
df = Dockerfile(parser.specs)
df.save('path/to/Dockerfile')
print(df)

problem with the full docker image (readme)

Trying to run examples from readme, I created the full Dockerfile with [this command] (https://github.com/kaczmarj/neurodocker#generate-dockerfile-full), but when I try to create a container I'm getting this:

/neurodocker/startup.sh: line 12: myfile.sh: No such file or directory
/neurodocker/startup.sh: line 13: arg1: command not found
``

add fsl eddy 5.0.11 pre-release

add ability to enter arbitrary Dockerfile instructions

A class like ANTs would benefit from the option of supplying a URL with the latest binaries.

AFNI image doesn't include python

I ran this command:
docker run --rm kaczmarj/neurodocker:master generate -b ubuntu:17.04 -p apt --afni version=latest | docker build -t afni_nd -

started the container:
docker run -it afni_nd /bin/bash

then tried to run:
afni_system_check.py --check_all

and received the error message:
/usr/bin/env: ‘python’: No such file or directory

conda is deleted while installing Miniconda

The conda program in non-root conda environments is a symlink to the root environment's conda. The root conda is deleted during the installation of Miniconda.

should raise an error when run with py2

socket.gaierror while running the "generate dockerfile" example

Summary

When I run the example
run --rm kaczmarj/neurodocker:v0.3.1 generate -b ubuntu:16.04 -p apt --ants version=2.2.0
I got into the socket.gaierror

Any ideas on what causes this error?
Thank you

Platform details:

Ubuntu 16.04 ( inside VMbox )

###Error details


sudo docker run --rm kaczmarj/neurodocker:v0.3.1 generate -b ubuntu:16.04 -p apt --ants version=2.2.0
Unable to find image 'kaczmarj/neurodocker:v0.3.1' locally
v0.3.1: Pulling from kaczmarj/neurodocker
88286f41530e: Pull complete 
cbf6c3c6bd80: Pull complete 
2592e13ad4e6: Pull complete 
Digest: sha256:d5acfc788c670f0d597e34e60d78fb93c276e2cd803a31b5ddbf225981c1ae83
Status: Downloaded newer image for kaczmarj/neurodocker:v0.3.1
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.6/socket.py", line 743, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f2dda72ccc0>: Failed to establish a new connection: [Errno -3] Try again

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dl.dropbox.com', port=443): Max retries exceeded with url: /s/2f4sui1z6lcgyek/ANTs-Linux-centos5_x86_64-v2.2.0-0740f91.tar.gz (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f2dda72ccc0>: Failed to establish a new connection: [Errno -3] Try again',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/neurodocker", line 11, in <module>
    load_entry_point('neurodocker', 'console_scripts', 'neurodocker')()
  File "/opt/neurodocker/neurodocker/neurodocker.py", line 278, in main
    subparser_functions[namespace.subparser_name](namespace)
  File "/opt/neurodocker/neurodocker/neurodocker.py", line 225, in generate
    df = Dockerfile(specs)
  File "/opt/neurodocker/neurodocker/generate.py", line 456, in __init__
    self.cmd = self._create_cmd()
  File "/opt/neurodocker/neurodocker/generate.py", line 477, in _create_cmd
    chunks = _get_dockerfile_chunks(self.specs)
  File "/opt/neurodocker/neurodocker/generate.py", line 435, in _get_dockerfile_chunks
    chunk = _get_dockerfile_chunk(instruction, options, specs)
  File "/opt/neurodocker/neurodocker/generate.py", line 413, in _get_dockerfile_chunk
    chunk = callable_(**options).cmd
  File "/opt/neurodocker/neurodocker/interfaces/ants.py", line 74, in __init__
    self.cmd = self._create_cmd()
  File "/opt/neurodocker/neurodocker/interfaces/ants.py", line 83, in _create_cmd
    chunks = [comment, self.install_binaries()]
  File "/opt/neurodocker/neurodocker/interfaces/ants.py", line 98, in install_binaries
    check_url(url)
  File "/opt/neurodocker/neurodocker/utils.py", line 141, in check_url
    request = requests.head(url, timeout=timeout, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 98, in head
    return request('head', url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='dl.dropbox.com', port=443): Max retries exceeded with url: /s/2f4sui1z6lcgyek/ANTs-Linux-centos5_x86_64-v2.2.0-0740f91.tar.gz (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f2dda72ccc0>: Failed to establish a new connection: [Errno -3] Try again',))

Add different versions of SPM

Currently, the dev version of SPM is installed by default. Users should be able to choose the version of SPM they want, including dev.

Todo

Determine which MCR version goes with the different versions of SPM.

add: json file of specs in container

json serialize the neurodocker specs to /neurodocker_spec.json and store it in the container.

add setup.py and simplify SpecsParser

connect to docker container with x2go

Error: Cannot assign requested address - with jupyter notebook

Hi,

I'm not sure if it is a bug or if I'm doing something wrong. But I have an issue with using jupyter notebook within a neurodocker image. If I run jupyter notebook I get the following error:

neuro@9f1ad00fd1b6:~$ jupyter notebook
[W 09:05:04.084 NotebookApp] server_extensions is deprecated, use nbserver_extensions
Traceback (most recent call last):
  File "/opt/conda/envs/neuro/bin/jupyter-notebook", line 11, in <module> sys.exit(main())
  File "/opt/conda/envs/neuro/lib/python3.6/site-packages/jupyter_core/application.py", line 267, in launch_instance return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/envs/neuro/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance app.initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "/opt/conda/envs/neuro/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error return method(app, *args, **kwargs)
  File "/opt/conda/envs/neuro/lib/python3.6/site-packages/notebook/notebookapp.py", line 1366, in initialize self.init_webapp()
  File "/opt/conda/envs/neuro/lib/python3.6/site-packages/notebook/notebookapp.py", line 1188, in init_webapp self.http_server.listen(port, self.ip)
  File "/opt/conda/envs/neuro/lib/python3.6/site-packages/tornado/tcpserver.py", line 142, in listen sockets = bind_sockets(port, address=address)
  File "/opt/conda/envs/neuro/lib/python3.6/site-packages/tornado/netutil.py", line 197, in bind_sockets sock.bind(sockaddr)
OSError: [Errno 99] Cannot assign requested address

But if I start jupyter notebook with jupyter notebook --ip 127.0.0.1 it works. In the same way, if I start the docker image with docker run -it --rm -p 8888:8888 my_docker_image jupyter notebook --ip 127.0.0.1 it works also.

I used the following docker command to create my neurodocker image:

docker run --rm kaczmarj/neurodocker generate -b neurodebian:stretch-non-free -p apt \
    --instruction "RUN curl -sL https://deb.nodesource.com/setup_6.x | bash -" \
    --install dcm2niix convert3d ants graphviz tree git-annex-standalone vim emacs-nox nano less ncdu tig git-annex-remote-rclone xvfb mesa-utils build-essential nodejs \
    --afni version=latest \
    --fsl version=5.0.10 \
    --freesurfer version=6.0.0 min=True \
    --spm version=12 matlab_version=R2017a \
    --install psmisc libapparmor1 sudo \
    --instruction "RUN bash -c \"curl http://download2.rstudio.org/rstudio-server-\$(curl https://s3.amazonaws.com/rstudio-server/current.ver)-amd64.deb >> rstudio-server-amd64.deb && dpkg -i rstudio-server-amd64.deb && rm rstudio-server-amd64.deb\" " \
    --instruction "RUN curl -sSL https://dl.dropbox.com/s/lfuppfhuhi1li9t/cifti-data.tgz?dl=0 | tar zx -C / " \
    --user=neuro \
    --miniconda python_version=3.6 \
                conda_install="jupyter jupyterlab traits pandas matplotlib scikit-learn seaborn swig reprozip reprounzip altair traitsui apptools configobj vtk jupyter_contrib_nbextensions bokeh scikit-image" \
                env_name="neuro" \
                pip_install="https://github.com/nipy/nibabel/archive/master.zip https://github.com/nipy/nipype/tarball/master nilearn https://github.com/INCF/pybids/archive/master.zip datalad dipy nipy duecredit pymvpa2 mayavi git+https://github.com/jupyterhub/nbrsessionproxy.git" \
    --instruction "RUN bash -c \"source activate neuro && python -m ipykernel install --user --name neuro --display-name Py3-neuro \" " \
    --instruction "RUN bash -c \"source activate neuro && pip install --pre --upgrade ipywidgets pythreejs \" " \
    --instruction "RUN bash -c \"source activate neuro && pip install  --upgrade https://github.com/maartenbreddels/ipyvolume/archive/23eb91685dfcf200ee82f89ab6f7294f9214db8c.zip && jupyter nbextension install --py --sys-prefix ipyvolume && jupyter nbextension enable --py --sys-prefix ipyvolume \" " \
    --instruction "RUN bash -c \"source activate neuro && jupyter nbextension enable rubberband/main && jupyter nbextension enable exercise2/main && jupyter nbextension enable spellchecker/main \" " \
    --instruction "RUN bash -c \"source activate neuro && jupyter serverextension enable --sys-prefix --py nbrsessionproxy && jupyter nbextension install --sys-prefix --py nbrsessionproxy && jupyter nbextension enable --sys-prefix --py nbrsessionproxy \" " \
    --miniconda python_version=2.7 \
                env_name="afni27" \
                conda_install="ipykernel" \
                add_to_path=False \
    --instruction "RUN bash -c \"source activate afni27 && python -m ipykernel install --user --name afni27 --display-name Py2-afni \" " \
    --instruction "RUN bash -c \"source activate neuro && python -c 'from nilearn import datasets; haxby_dataset = datasets.fetch_haxby()' \" " \
    --workdir /home/neuro \
    --no-check-urls > Dockerfile

Now, one workaround to this issue is to create the file ~/.jupyter/jupyter_notebook_config.py with the content c.NotebookApp.ip = '*'. Therefore, I could just add the following command to my docker command:

--instruction "RUN bash -c \"echo c.NotebookApp.ip = '*' > ~/.jupyter/jupyter_notebook_config.py\" "

But is there a more straight forward way to handle this issue?

Thanks,
Michael

REQUEST: optional flag for curl retries

I'd love to have

standardized curl (or wget, etc) command across supported software
--curl_args or something similar that allows universally setting flags, such as --retry across the build

However, I'd be happy with just hardcoding --retry 5 into missing lines, especially: https://github.com/kaczmarj/neurodocker/blob/c713d17ae0bae70aad7d8a03195ba73458b4e0d8/neurodocker/interfaces/freesurfer.py#L177

Allow creation of an arbitrary number of conda environments

Right now, neurodocker can only install one conda environment.

neurodebian - ubuntu installation error

with ubuntu base installation of neurodebian fails with gpg keyserver error.

https://dl.dropbox.com/s/zxs209o955q6vkg/neurodebian.gpg gpgkeys: protocol https' not supported
gpg: no handler for keyserver scheme https' gpg: WARNING: unable to fetch URI https://dl.dropbox.com/s/zxs209o955q6vkg/neurodebian.gpg: keyserver error Removing intermediate container 17b51386ea90 The command '/bin/sh -c apt-get update -qq && apt-get install -yq --no-install-recommends dirmngr gnupg && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && curl -sSL http://neuro.debian.net/lists/jessie.us-nh.full > /etc/apt/sources.list.d/neurodebian.sources.list && apt-key adv --fetch-keys https://dl.dropbox.com/s/zxs209o955q6vkg/neurodebian.gpg && (apt-key adv --refresh-keys --keyserver hkp://pool.sks-keyservers.net:80 0xA5D32F012649A5A9 || true) && apt-get update' returned a non-zero code: 2 ERROR: Build failed: The command '/bin/sh -c apt-get update -qq && apt-get install -yq --no-install-recommends dirmngr gnupg && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && curl -sSL http://neuro.debian.net/lists/jessie.us-nh.full > /etc/apt/sources.list.d/neurodebian.sources.list && apt-key adv --fetch-keys https://dl.dropbox.com/s/zxs209o955q6vkg/neurodebian.gpg && (apt-key adv --refresh-keys --keyserver hkp://pool.sks-keyservers.net:80 0xA5D32F012649A5A9 || true) && apt-get update' returned a non-zero code: 2 ERROR: Build failed with exit code 2

Can not build AFNI using Neurodocker

Hi:

I am getting the following error during the build of AFNI using a dockerfile generated by neurodocker:

configure: Need to download and build NLopt
trying URL 'http://ab-initio.mit.edu/nlopt/nlopt-2.4.2.tar.gz'
Error in download.file(url = "http://ab-initio.mit.edu/nlopt/nlopt-2.4.2.tar.gz", :
cannot open URL 'http://ab-initio.mit.edu/nlopt/nlopt-2.4.2.tar.gz'
Execution halted
/bin/tar: This does not look like a tar archive

gzip: stdin: unexpected end of file
/bin/tar: Child returned status 1
/bin/tar: Error is not recoverable: exiting now
Warning message:
In untar(tarfile = "nlopt-2.4.2.tar.gz") :
'/bin/tar -xf 'nlopt-2.4.2.tar.gz'' returned error code 2
configure: Starting to install library to /tmp/RtmpqWSb5X/R.INSTALL4236c845677/nloptr/nlopt-2.4.2
./configure: line 3325: cd: nlopt-2.4.2: No such file or directory

The docker file was generated using the following command:

docker run --rm kaczmarj/neurodocker:master generate -b ubuntu -p apt --afni version=latest use_binaries=true > afni_docker

I get the same error if I do not include the 'use_binaries' flag.

The build is then initiated using:

docker build -f afni_docker -t afni .

My docker version info:

docker version
Client:
Version: 17.06.2-ce
API version: 1.30
Go version: go1.8.3
Git commit: cec0b72
Built: Tue Sep 5 20:12:06 2017
OS/Arch: darwin/amd64

Server:
Version: 17.06.2-ce
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: cec0b72
Built: Tue Sep 5 19:59:19 2017
OS/Arch: linux/amd64
Experimental: true

The build proceeds for quite a while, but appears to get stuck while installing R packages, at this line in the generated dockerfile:

/opt/afni/rPkgsInstall -pkgs ALL

The error is strange, because if you copy / paste the troublesome url into your browser:

http://ab-initio.mit.edu/nlopt/nlopt-2.4.2.tar.gz

the download starts and completes.

command-line interface cannot build packages from source

For example, the command neurodocker -b debian -p apt --ants version=2.2.0 use_binaries=0 returns a Dockerfile that downloads the ANTs binaries.

Probably because the argument to use_binaries is being converted to a string. In the main function or in the SpecsParser, add a method that converts boolean options to bools.

BUG: cannot install specific versions of pip/conda packages.

the command below does not work

neurodocker generate -b debian:stretch -p apt --miniconda env_name=py35 python_version=3.5 conda_install=pyqt=5

ADD: options to specify user, environment variables, ports to expose, etc.

This can be done with the--instruction argument, but it would be better to have dedicated arguments for the various Dockerfile instructions.

Run tests in singularity

At the moment, tests are being run only in Docker.

Example uses base debian stretch but neurodebian jessie

This can cause problems: I tried to install mrtrix, but it would fail because couldn't find the correct dependencies. I'm happy to send a PR, but just wanted to make sure that it wasn't intended for some other reason.

State somewhere in the README a minimal Python version needed

It seems that NeuroDocker is not compatible with python2, but also not clear which of the python3 versions should it work with? .travis.yml tests only with 3.6 -- is that the minimal supported version? if not, would be great if it was tested on all supported versions on travis.
Cheers!