apptainer / singularity Goto Github PK

Singularity has been renamed to Apptainer as part of us moving the project to the Linux Foundation. This repo has been persisted as a snapshot right before the changes.

Home Page: https://github.com/apptainer/apptainer

License: Other

Shell 2.51% C 2.96% Makefile 0.49% Python 0.17% Go 93.30% Awk 0.57%

cloud-native container containers hpc linux parallel portability portable reproducible reproducible-science rootless-containers science singularity singularity-container

singularity's Introduction

Apptainer

What is Apptainer?

Apptainer is an open source container platform designed to be simple, fast, and secure. Many container platforms are available, but Apptainer is designed for ease-of-use on shared systems and in high performance computing (HPC) environments. It features:

An immutable single-file container image format, supporting cryptographic signatures and encryption.
Integration over isolation by default. Easily make use of GPUs, high speed networks, parallel filesystems on a cluster or server.
Mobility of compute. The single file SIF container format is easy to transport and share.
A simple, effective security model. You are the same user inside a container as outside, and cannot gain additional privilege on the host system by default.

Apptainer is open source software, distributed under the BSD License.

Apptainer was formerly known as Singularity and is now a part of the Linux Foundation. When migrating from Singularity see the admin migration documentation and user compatibility documentation.

Check out talks about Apptainer and some use cases of Apptainer on our website.

Getting Started with Apptainer

To install Apptainer from source, see the installation instructions. For other installation options, see our guide.

System administrators can learn how to configure Apptainer, and get an overview of its architecture and security features in the administrator guide.

For users, see the user guide for details on how to run and build containers with Apptainer.

Contributing to Apptainer

Community contributions are always greatly appreciated. To start developing Apptainer, check out the guidelines for contributing.

Please note we have a code of conduct. Please follow it in all your interactions with the project members and users.

Our roadmap, other documents, and user/developer meeting information can be found in the apptainer community page.

We also welcome contributions to our user guide and admin guide.

Support

To get help with Apptainer, check out the Apptainer Help web page.

Go Version Compatibility

Apptainer aims to maintain support for the two most recent stable versions of Go. This corresponds to the Go Release Maintenance Policy and Security Policy, ensuring critical bug fixes and security patches are available for all supported language versions.

Citing Apptainer

Apptainer can be cited using its former name Singularity.

The Singularity software may be cited using our Zenodo DOI 10.5281/zenodo.1310023:

Singularity Developers (2021) Singularity. 10.5281/zenodo.1310023 https://doi.org/10.5281/zenodo.1310023

This is an 'all versions' DOI for referencing Singularity in a manner that is not version-specific. You may wish to reference the particular version of Singularity used in your work. Zenodo creates a unique DOI for each release, and these can be found in the 'Versions' sidebar on the Zenodo record page.

Please also consider citing the original publication describing Singularity:

Kurtzer GM, Sochat V, Bauer MW (2017) Singularity: Scientific containers for mobility of compute. PLoS ONE 12(5): e0177459. https://doi.org/10.1371/journal.pone.0177459

License

Unless otherwise noted, this project is licensed under a 3-clause BSD license found in the license file.

singularity's People

Contributors

Stargazers

Watchers

Forkers

aronr donkirkby jjohnson42 kmuriki zashary estabroo remyd1 gitter-badger kellyrowland ctlipping loveshack nl384 truatpasteurdotfr yfxia yarikoptic aduffy19 bbockelm aculich emun2 czka nirmalasrjn zonca cequencer akkornel tin6150 bauerm97 ch741 boegel vsoch gijzelaerr q-leap-networks soichih dthain citarulm sbutcher dyndna flx42 rjeschmi acaprez jtriley oschulz hubiepisteme eshelman trcook mppmu satra henktillman alvarosimon quanteek zixuanhe stdweird wpoely86 nekel-seyew alexxnica basvandervlies duke-translational-bioinformatics cclerget robermorales guowei-he paulo7777 nih-hpc victorgregoriopenguin obsidiansystems genomicsnx djw8605 lkurz crbaird hcrat tacc freesky-edward trophime khurtado it4innovations moskalenko mrmhodak votavap tristan0x edmondac karlnewell paulhopkins strategist922 brianv0 bertl4398 ityonemo tabaer hartzell yhcote tkuennen umitsamima ryanolson nagyistge rajaajar87 swidnikk rowhit ikaneshiro raonyguimaraes armenr bmwiedemann olifre ilmagico

singularity's Issues

obsolete executables

Are ftrace and ftype still required? They are still built, but I think they are obsolete. (ftrace prevents building on non-x86 architectures.)

Cannot run yum in centos container

Hi,

Since the update to singularity 2.0 I can't run yum in containers

[root@localhost ~]# singularity create container.img
[root@localhost ~]# singularity bootstrap container.img centos.def
[root@localhost ~]#  singularity shell --writable container.img
Singularity/container.img> yum install top
Loaded plugins: fastestmirror
Could not create lock at /var/run/yum.pid: [Errno 30] Read-only file system: '/var/run/yum.pid'


Can't create lock file; exiting
Singularity/container.img> whoami
root
Singularity/container.img> exit
exit

[root@localhost ~]# singularity --version
2.1

[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

Thanks for your help,

Martin

Build problem on Fedora 20

I have the following header conflict when building 9f27c61 on Fedora 20:

gcc -DHAVE_CONFIG_H -I.  -DARCH_x86_64  -Wall -g -O2 -MT ftrace-ftrace.o -MD -MP -MF .deps/ftrace-ftrace.Tpo -c -o ftrace-ftrace.o `test -f 'ftrace.c' || echo './'`ftrace.c
In file included from ftrace.c:28:0:
/usr/include/linux/ptrace.h:58:8: error: redefinition of ‘struct ptrace_peeksiginfo_args’
 struct ptrace_peeksiginfo_args {
        ^
In file included from ftrace.c:27:0:
/usr/include/sys/ptrace.h:191:8: note: originally defined here
 struct ptrace_peeksiginfo_args

solved commenting the line

#include <linux/ptrace.h>

in ftrace.c. Is the fix correct?

avoid the need for bc

I tried to build an image in a fairly clean centos7 VM and it failed or want of bc. That either needs requiring in the spec file or, probably better, avoiding:

diff --git a/libexec/cli/image.exec b/libexec/cli/image.exec
index 56974c4..d38299d 100644
--- a/libexec/cli/image.exec
+++ b/libexec/cli/image.exec
@@ -66,7 +66,7 @@ shift

 case "$SUBCOMMAND" in
     create)
-        IMAGE_BYTES=`echo "$IMAGE_SIZE * (1024 * 1024)" | bc`
+        IMAGE_BYTES=$(($IMAGE_SIZE * 1024 * 1024))
         IMAGE_FILE="$1"
         shift

'/usr/local/libexec/singularity/sexec-prep': No such file or directory

"sudo make install-perms" gives an error. The file name should be ".../sexec_prep"

resolv.conf

It seems that sapps use a default resolv.conf with the google DNS.
What if you must use a custom one ?
I tried overriding this default resolv.conf using %files but it does not work.

In fact, if a sapp is to work as any local executable, it should probably use its host network config, not a default one.

strange mount/unmount behavior

seen in setup meeting on 5/31/16 on ubuntu 14.

Cannot run yum in Centos container

I get the following error:

Could not create lock at /var/run/yum.pid: [Errno 30] Read-only file system: '/var/run/yum.pid'

Relocatable RPM (or instructions)

Hey,

I'd like specify the prefix when building an RPM or have the RPM be relocatable. Even if this was just an "edit here" example that would be sufficient.

Thanks

Segfault

When opening image as writable when file permission doesn't allow.

No usable temporary directory found

Hello,

On SL6.7, when I do yum install within a CentOS 7 image I get:

--> Finished Dependency Resolution

[Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp']

Any ideas?

Many thanks,
Sebastian.

PS: I am using the code at master

platform restrictions not documented

I couldn't see it noted anywhere that singularity is (currently?) x86_64-specific, as I realized when trying to build a range of Fedora packages.

Also it might be useful to have information on the versions of Linux that will/won't support it for common distributions. Specifically RHEL5's isn't good enough -- we still have such a cluster... -- but I don't know about the oldest supported Debian and SuSE.

Local installation of singularity?

Is it (technically) possible to have a completely local (non-root) installation of singularity for running containers? Creating and bootstrapping images requires sudo, but I've noticed that unless singularity is installed with sudo make install, I run into the following error when trying to run containers (via singularity shell)

ABORT: Could not escalate effective user privledges!

EXT3 unsupported optional features (2c0)

Hi!

I just installed v2.0 and created a Debian container image (on a Debian machine), which works great:

debian $ singularity exec container.img cat /etc/debian_version
8.4

Now, I transferred that container image and try to "run" it on a Centos machine, and there seem to be a problem with the way the filesystem image is formatted:

centos $ singularity exec container.img cat /etc/debian_version
ERROR: Failed to mount (ro) '/dev/loop0' at '/tmp/.singularity-215845.2032753612.144117941686781743/mnt': Invalid argument

And the kernel logs on the Centos host say:

JBD: Ignoring recovery information on journal
JBD: Unrecognised features on journal
EXT4-fs (loop0): error loading journal
JBD: Ignoring recovery information on journal
JBD: Unrecognised features on journal
EXT4-fs (loop0): error loading journal
EXT3-fs (loop0): error: couldn't mount because of unsupported optional features (2c0)

Any idea what's wrong here?

Thanks!

Add start/stop for holding namespace open

8:08 what that needs is a feature I've been thinking about
18:09 which holds open the namespace
18:10 usage would be a bit different
18:10 it would have to include a "start" and "stop"
18:10 which will be in addition to the shell/run/exec options
18:11 so when you "start" the container, it will launch a daemon process which will hold open the namespace.
18:11 then all subsequent commands will join into that existing container

CentOS-5 issues (echo/runtime)

[tru@sillage singularity]$ cat find.sspec
Name: find
Exec: /usr/bin/find

echo does not look right ;)

[tru@sillage singularity]$ singularity build find.sspec
\e[0;39mBuilding: find
\e[0;39m\e[0;39mChecking paths...
\e[0;39m\e[0;39mChecking writable...
\e[0;39m\e[0;39mEvaluating: %packages
\e[0;39m\e[0;39mCreating template configs
\e[0;39m\e[0;39m  /etc/nsswitch.conf
\e[0;39m\e[0;39m  /etc/resolv.conf
\e[0;39m\e[0;39m  /etc/hosts
\e[0;39m\e[0;39mEvaluating: hard requirements
\e[0;39m\e[0;39mInstalling file: /bin/sh
\e[0;39m\e[0;39mEvaluating: %exec
\e[0;39m\e[0;39mInstalling file: /usr/bin/find
\e[0;39m\e[0;39mEvaluating: %files
\e[0;39m\e[0;39mInvoking dependency check
\e[0;39m\e[0;39mInstalling file: /lib64/libselinux.so.1
\e[0;39m\e[0;39mInstalling file: /lib64/libc.so.6
\e[0;39m\e[0;39mInstalling file: /lib64/libdl.so.2
\e[0;39m\e[0;39mInstalling file: /lib64/libsepol.so.1
\e[0;39m\e[0;39mInstalling file: /lib64/ld-linux-x86-64.so.2
\e[0;39m\e[0;39mInstalling file: /lib64/libtermcap.so.2
\e[0;39m\e[0;39mInstalling file: /lib64/libnss_files.so.2
\e[0;39m\e[0;39mInstalling file: /lib64/libnss_dns.so.2
\e[0;39m\e[0;39mInstalling file: /lib64/libresolv.so.2
\e[0;39m\e[0;39mIncluding SAPPFILE
\e[0;39m\e[0;39mCalculating container size
\e[0;39m\e[0;39mCalculating checksums
\e[0;39m\e[0;39mWriting metadata
\e[0;39mdirname: extra operand `./lib64/libresolv.so.2'
Try `dirname --help' for more information.
\e[0;39mBuilding primary singularity exec driver
\e[0;39m\e[0;39mBuilding singularity test runscript
\e[0;39m\e[0;39mBuilding singularity main runscript
\e[0;39m\e[0;39mRunning %test...
\e[0;39m\e[0;39mCleaning up unpackaged files
\e[0;39m\e[0;39mCreating output SAPP container
\e[0;39m\e[0;39mWROTE: find.sapp

runtime:

[tru@sillage singularity]$ ./find.sapp 
ERROR: Could not stat /home/tru/.sapp-cache/./find/c!
/usr/libexec/singularity/run.exec: line 79: /home/tru/.sapp-cache/./find/c/pids: No such file or directory
sed: can't read /home/tru/.sapp-cache/./find/c/pids: No such file or directory

[tru@sillage singularity]$ singularity run find
ERROR: Could not stat /home/tru/.sapp-cache/find/c!
/usr/libexec/singularity/run.exec: line 79: /home/tru/.sapp-cache/find/c/pids: No such file or directory
sed: can't read /home/tru/.sapp-cache/find/c/pids: No such file or directory

[tru@sillage singularity]$ find /home/tru/.sapp-cache/ -ls
486539395    0 drwxr-xr-x   3 tru      Bis            17 Nov 30 12:38 /home/tru/.sapp-cache/
499539116    0 drwxr-xr-x   2 tru      Bis            16 Nov 30 12:38 /home/tru/.sapp-cache/find
499539121    4 -rw-r--r--   1 tru      Bis            33 Nov 30 12:38 /home/tru/.sapp-cache/find/sum

Binaries may not expect shared libraries in /lib{,64}

All shared libraries get copied into /lib or /lib64 in the "container".
This does not work with binaries from Ubuntu, as they expect shared libraries (have rpath set) in "/lib/x86_64-linux-gnu/", even for "/bin/sh". Explicitly setting LD_LIBRARY_PATH in src/sexec.c with setenv(2) helps.

format error

With current master:

mount.c: In function ‘main’:
mount.c:141:9: warning: too many arguments for format [-Wformat-extra-args]
         snprintf(prompt, strlen(containerimage) + strlen(mountpoint) + 15, "Singularity/%s> ", containername, mountpoint);
         ^

loose matching in perlmods.dep

perlmods.dep currently does

egrep "^\s*(use|require|no)\s*"

To reduce false positives, I think it should be

egrep "^\s*(use|require|no)\s"

failure to find next loop device

At least in centos 6, if I have an existing loop mount, bootstrap fails:

$ sudo losetup -f
/dev/loop1
$ sudo singularity bootstrap c6.img singularity/examples/centos.def 
ERROR: Failed to associate image to loop
ERROR: Could not associate c6.img to loop device /dev/loop0

If it's not obvious, I can probably debug it eventually.

curl and dns

Trying to investigate dns related issues, I made a curl sapp.

I can not pinpoint what's needed to make it work, but here's my findings:

Name: curl
Exec: /usr/bin/curl

does not work:
./curl.sapp google.fr
curl: (6) Could not resolve host: google.fr

Name: curl
Exec: /usr/bin/curl
%files:
/lib/x86_64-linux-gnu/

this works !

But from here, I used singularity strace to get the exact list of used libs, I pasted them in the %files section, but never managed to get it to work:

singularity strace 2>curl.err curl google.fr
parsing it gives me the following list of files:

/lib/i386-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libcom_err.so.2
/lib/x86_64-linux-gnu/libcrypt.so.1
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
/lib/x86_64-linux-gnu/libdl.so.2
/lib/x86_64-linux-gnu/libgcrypt.so.11
/lib/x86_64-linux-gnu/libgpg-error.so.0
/lib/x86_64-linux-gnu/libkeyutils.so.1
/lib/x86_64-linux-gnu/libpthread.so.0
/lib/x86_64-linux-gnu/libresolv.so.2
/lib/x86_64-linux-gnu/libssl.so.1.0.0
/lib/x86_64-linux-gnu/libz.so.1
/usr/lib/x86_64-linux-gnu/libasn1.so.8
/usr/lib/x86_64-linux-gnu/libcurl.so.4
/usr/lib/x86_64-linux-gnu/libffi.so.6
/usr/lib/x86_64-linux-gnu/libgnutls.so.26
/usr/lib/x86_64-linux-gnu/libgssapi.so.3
/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
/usr/lib/x86_64-linux-gnu/libhcrypto.so.4
/usr/lib/x86_64-linux-gnu/libheimbase.so.1
/usr/lib/x86_64-linux-gnu/libheimntlm.so.0
/usr/lib/x86_64-linux-gnu/libhx509.so.5
/usr/lib/x86_64-linux-gnu/libidn.so.11
/usr/lib/x86_64-linux-gnu/libk5crypto.so.3
/usr/lib/x86_64-linux-gnu/libkrb5.so.26
/usr/lib/x86_64-linux-gnu/libkrb5.so.3
/usr/lib/x86_64-linux-gnu/libkrb5support.so.0
/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0
/usr/lib/x86_64-linux-gnu/libroken.so.18
/usr/lib/x86_64-linux-gnu/librtmp.so.0
/usr/lib/x86_64-linux-gnu/libsasl2.so.2
/usr/lib/x86_64-linux-gnu/libsqlite3.so.0
/usr/lib/x86_64-linux-gnu/libtasn1.so.6
/usr/lib/x86_64-linux-gnu/libwind.so.0

What could be the problem ?

Thanks.

ABORT: Failed creating template password file

I created a demo.img on a CentOS host, and I am trying to run singularity shell demo.img on an Arch machine. I am met with the following error:

ERROR: Template passwd not found: /tmp/.singularity-1000.2049.19923486/mnt//etc/passwd
ABORT: Failed creating template password file

Any suggestions as to what this might be about?

Edit: demo.img is created using centos.def found in the examples directory. I am using the master branch

/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

Hi,

The wrapper that starts my app tries to perform some setlocale commands, and I get these warnings:
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Warning message:
In Sys.setlocale("LC_COLLATE", "en_US.UTF-8") :
OS reports request to set locale to "en_US.UTF-8" cannot be honored
During startup - Warning messages:
1: Setting LC_CTYPE failed, using "C"
2: Setting LC_COLLATE failed, using "C"
3: Setting LC_TIME failed, using "C"
4: Setting LC_MESSAGES failed, using "C"
5: Setting LC_MONETARY failed, using "C"
6: Setting LC_PAPER failed, using "C"
7: Setting LC_MEASUREMENT failed, using "C"

I tried to add the following files (from a strace log):
/usr/share/locale/locale.alias
/usr/lib/locale/locale-archive
/usr/share/locale-langpack/en/LC_MESSAGES

But I still can'get rid of this error:
In Sys.setlocale("LC_COLLATE", "en_US.UTF-8") :
OS reports request to set locale to "en_US.UTF-8" cannot be honored

libexec/singularity/functions: line 73: syntax error: unexpected redirection (running singularity on alpine linux)

alpine:~/singularity$ sudo singularity bootstrap a.img examples/debian.def
/usr/local/libexec/singularity/bootstrap.sh: /usr/local/libexec/singularity/functions: line 73: syntax error: unexpected redirection

alpine:~/singularity$ sed -n 69,78p /usr/local/libexec/singularity/functions
get_key_from_conf() {
KEY="$1"
FILE="$2"
if OUT=egrep -i "^$KEY:\s*" $FILE; then
head -n 1 <<< "$OUT" | sed -e "s@^$KEY:\s*@@i"
return 0
fi
return 1
}

alpine linux provides debootstrap, hence my test of bootstrap.

resolving fails with some versions of file(1)

Scripts fail on RHEL6 because of what file returns. This fixes it. (I wonder if the match should be more specific.)

index d462b92..4c3aec5 100644
--- a/libexec/mods/install.smod
+++ b/libexec/mods/install.smod
@@ -53,7 +53,7 @@ dep_resolver() {
                     eval $i "$1"
                 done
             ;;
-            *ASCII*)
+            *ASCII* | *script*)
                 for i in $TXT_RESOLVERS; do
                     message 3 "calling txt_resolvers on: $1 : $i\n"
                     eval $i "$1"

Script files expect bash, but run /bin/sh

On Ubuntu, /bin/sh actually is "dash", resulting in strange failures.

Further, in your containers, you only copy "/bin/sh".
Better install /bin/bash (in mods/install_files), and create a link "sh -> bash"?

PS: Nice project.

pymods.dep copies too much

Resolving python files ends up copying the whole site-packages directory which appears in the ftrace output, which isn't "most minimal"... Using install_single_file, instead of install_file fixes that. Then it still seems to copy too much, which I haven't checked, but is much more reasonable. (Also, it does too much work recursively checking the copied files, and it might be nice to avoid installing .py if there's a corresponding current .pyc.)

I guess the perl version should be changed similarly, but I don't know if it will make a difference.

#! matching is too strict

script_resolver doesn't catch something like

#! /bin/env fred

I generalized it like this:

diff --git a/libexec/mods/interpreter.dep b/libexec/mods/interpreter.dep
index 25d4d81..a5a545a 100644
--- a/libexec/mods/interpreter.dep
+++ b/libexec/mods/interpreter.dep
@@ -26,16 +26,16 @@ TXT_RESOLVERS="script_resolver $BIN_RESOLVERS"

 script_resolver() {
     for file in $@; do
-        LINE=`head -n 1 "$file" | grep "^#\!/"`
-        if check_pattern "#!/usr/bin/env *" "$LINE"; then
-            INT=`echo "$LINE" | sed -e 's@#![^ ]*[ ]*@@'`
-            INT_PATH=`singularity_which "$INT"`
+        read <"$file"
+        if [[ $REPLY =~ \#![[:space:]]*((/[^[:space:]]+)?/bin/env)[[:space:]]+([^[:space:]]+) ]]; then
+            env=${BASH_REMATCH[1]}
+            INT_PATH=`singularity_which "${BASH_REMATCH[3]}"`
             if [ -n "$INT_PATH" ]; then
-                install_file "/usr/bin/env"
+                install_file $env
                 install_file "$INT_PATH"
             fi
-        else
-            INT=`echo "$LINE" | sed -e 's@#!\([^ ]*\).*@\1@'`
+        elif [[ $REPLY =~ \#![[:space:]]*(/[^[:space:]]+) ]]; then
+            INT=${BASH_REMATCH[1]}
             if [ -f "$INT" -a ! -f "$INSTALLDIR/c/$INT" ]; then
                 install_file "$INT"
             fi

packaging warnings

The following warnings are issued by checks required for Fedora review, but I think they're not real issues. Could confirm that, and maybe consider either inserting an initgroups and chdir to pacify the check or adding comments about them? (That isn't actually necessary for the review, providing the warning is deemed spurious.)

$ rpmlint -I missing-call-to-chdir-with-chroot -I missing-call-to-setgroups-before-setuid
missing-call-to-chdir-with-chroot:
This executable appears to call chroot without using chdir to change the
current directory. This is likely an error and permits an attacker to break
out of the chroot by using fchdir. While that's not always a security issue,
this has to be checked.

missing-call-to-setgroups-before-setuid:
This executable is calling setuid and setgid without setgroups or initgroups.
There is a high probability this means it didn't relinquish all groups, and
this would be a potential security issue to be fixed. Seek POS36-C on the web
for details about the problem.

[POS36-C is https://www.securecoding.cert.org/confluence/display/c/POS36-C.+Observe+correct+revocation+order+while+relinquishing+privileges]

RFE distributing container images

Hi,

provide the container.def allong with the container.img
naming: dot release and dot release with updates as of YYYY-MM-DD rolloed in?
example CentOS-6.8-GA.img and CentOS-6.8-2016-06-07.img
tar -CjSf container.tbz2 container.img container.def
one needs to preserver the "sparse" file state, bzip2/gzip don't, but tar does!, otherwise expanding a compressed container.img yields a non sparse file, (possible workaroud is to cp --sparse=always expanded-non-sparse.img expanded-sparse-converted.img, but ymmv).

Cheers

build fails on RHEL 6

This is on RHEL 6 with current master source (13ee029):

gcc -DHAVE_CONFIG_H -I.  -DSYSCONFDIR=\"/usr/local/etc\" -DLOCALSTATEDIR=\"/usr/local/var\" -DLIBEXECDIR=\"/usr/local/libexec\" -DNS_CLONE_NEWPID -DNS_CLONE_FS -DNS_CLONE_FILES -DNS_CLONE_NEWNS  -Wall -g -O2 -MT sexec-loop-control.o -MD -MP -MF .deps/sexec-loop-control.Tpo -c -o sexec-loop-control.o `test -f 'loop-control.c' || echo './'`loop-control.c
loop-control.c: In function ‘obtain_loop_dev’:
loop-control.c:55: error: ‘LOOP_CTL_GET_FREE’ undeclared (first use in this function)
loop-control.c:55: error: (Each undeclared identifier is reported only once
loop-control.c:55: error: for each function it appears in.)
make[2]: *** [sexec-loop-control.o] Error 1
make[2]: Leaving directory `/home/dlove/rpmbuild/SOURCES/singularity/src'

installation issue on ubuntu 14.04 LTS

krowland@WARP-Drive:~/singularity$ sh ./autogen.sh 
+ autoreconf -i
configure.ac:32: error: possibly undefined macro: AC_PROG_LIBTOOL
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
autoreconf: /usr/bin/autoconf failed with exit status: 1
+ [ -z  ]
+ ./configure
configure: error: cannot find install-sh, install.sh, or shtool in "." "./.." "./../.."

This seems to be fixed by removing the AC_PROG_LIBTOOL line in configure.ac. I can PR that change in; I'm not sure if it requires more investigation.

No module named site error for a python-based package

Tried building a singularity container of a python-based GUI application (xpra) from xpra.org from an rpm package installed under RHEL-7.0 from the upstream repository. Singularity built xpra.sapp without errors, but when I try running the container on the same system I built it on I get the following error

$ singularity run xpra
ImportError: No module named site

Here's the spec file:

$ cat spec/xpra
Name: xpra
Maintainer: local
Exec: /usr/bin/xpra

%packages
xpra

It looks like a few python-related fixes went into singularity recently, so this build could either be triggering another bug or I'm just not doing it right.

Integrate ftrace for specgen

Need to complete ftrace and integrate with specgen

problem with configure prefix

Hi,

I found another issue. When I try to install singularity in another directory than the default one (/opt/singularity), I have issues with the command "/opt/singularity/bin/singularity build" once it is installed :

/opt/singularity/libexec/singularity/mods/install.smod: line 47: /usr/libexec/singularity/ftype: No such file or directory
... a lot of that kind of message ...

A little fix is to add a symbolic link from where singularity is searching :

mkdir -p /usr/libexec && cd $_
ln -s /opt/singularity/libexec/singularity

I think that this problem appears since the last update in the master branch.

Best,
Remy

bootstrap fails on RHEL

Do you want me to try to fix this? If so, should it always look for the corresponding CentOS? (I think that won't generally have quite the same package set available, and definitely not the same as SL.)

$ sudo singularity bootstrap /tmp/c6.img examples/centos.def 
Loaded plugins: auto-update-debuginfo, changelog, etckeeper, fastestmirror,
              : filter-data, merge-conf, post-transaction-actions, priorities,
              : product-id, protectbase, security, subscription-manager, verify,
              : versionlock
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Setting up Install Process
http://mirror.centos.org/centos-6Server/6Server/os/x86_64/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
To address this issue please refer to the below knowledge base article 

https://access.redhat.com/articles/1320623

If above article doesn't help to resolve this issue please open a ticket with Red Hat Support.

Error: Cannot retrieve repository metadata (repomd.xml) for repository: os-base. Please verify its path and try again

issue with having .sapp-cache in home directory in NFS context

Just a note in case this hadn't come up as something you might address at some point.

I'm trying singularity on an Ubuntu 14.04 machine on which my home directory is NFS-mounted. I'm getting this error:
paciorek@smeagol:/tmp> ./echo.sapp
ERROR: Could not bind mount /accounts/gen/vis/paciorek/.sapp-cache/./echo/c/dev

Channeling a conversation about this with Ryan Lovett - it seems that the issue is that root can't do the bind mount because it can access a user's home directory on an NFS client.

using singularity shell and strace: ERROR: Singularity container is not installed

Seems to do the same thing for all SAPPs I produced:

%singularity run blastn.sapp -h
=> works OK

but

%singularity shell blastn.sapp
ERROR: Singularity container is not installed: blastn.sapp

%singularity shell $(pwd)/blastn.sapp
ERROR: Singularity container is not installed: blastn.sapp

%singularity shell $./blastn.sapp
ERROR: Singularity container is not installed: blastn.sapp

%singularity strace ./blastn.sapp 
ERROR: Singularity container is not installed: ./blastn.sapp

Did I miss something ?!

RFE: add a logic for non glibc linux distribution providing debootrap

just bitten by a debootstrap feature on non debian distribution:

debootstrap will try to guess the current architecture (via dpkg or anything else)
on alpine linux (musl based, not glibc), "dpkg --print-architecture" gives "musl-linux-amd64"
instead of the usual "amd64"

alpine:~/singularity$ sudo singularity bootstrap b.img examples/debian.def
I: Retrieving Release
I: Retrieving Release.gpg
I: Checking Release signature
I: Valid Release signature (key id 75DDC3C4A499F1A18CB5F3C8CBF8D6FD518E17E1)
E: Invalid Release file, no entry for main/binary-musl-linux-amd64/Packages

./libexec/mods/linux_build_debian.smod:
one might need to had a check and force a "$DEBOOTSTRAP_PATH --arch amd64 ..."

Output a better error message

$ singularity run /path/to/file.sapp

Gives a bad error message. Fixor it.

Bind config

Create a configuration file which must be root owned to configure what directories/paths to bind mount into the container.

Could not escalate privledges

Hi,

Firstly, your project sounds really interesting.

However, I am not able to generate any sapp execution file. If I try to generate one while I am root, I get:
"ERROR: Do not run singularities as root!"

But if I try to do the same as a simple user I get:
"ERROR: Could not escalate privledges!"

About documentation:

By the way, you shoud update your documentation (README file), because apparently you changed the way spec files should be written:

name echo -> Name: echo
program /bin/echo -> Exec: /bin/echo

PS: please do not write <sapp specfile> in your README, because it could also lead to some misunderstanding in your file definition (you could use 3*backquote to escape your text (markdown)).

Best,
Remy

infinite loop

r.spec:

Name: R302
Exec: /usr/bin/R

singularity build r.spec: it loops forever

Evaluating: Exec
Installing file: /bin/R
Installing file: /usr/lib/locale/locale-archive
Creating dir   : /usr/lib/R/bin
Installing file: /usr/lib/R/bin/mkinstalldirs
Installing file: /bin/bash
Installing file: /lib/x86_64-linux-gnu/libtinfo.so.5
Installing file: /usr/lib/R/bin/Rd2pdf
Installing file: /usr/lib/R/bin/Rdconv
Installing file: /usr/lib/R/bin/Stangle
Installing file: /usr/lib/R/bin/COMPILE
Installing file: /usr/lib/R/bin/LINK
Installing file: /usr/lib/R/bin/Rcmd
Installing file: /usr/lib/R/bin/R
Installing file: /usr/lib/locale/locale-archive
Installing file: /usr/lib/R/bin/mkinstalldirs
Installing file: /usr/lib/R/bin/Rd2pdf
Installing file: /usr/lib/R/bin/Rdconv
Installing file: /usr/lib/R/bin/Stangle
Installing file: /usr/lib/R/bin/COMPILE
Installing file: /usr/lib/R/bin/LINK
Installing file: /usr/lib/R/bin/Rcmd
Installing file: /usr/lib/R/bin/R
Installing file: /usr/lib/locale/locale-archive
Installing file: /usr/lib/R/bin/mkinstalldirs
Installing file: /usr/lib/R/bin/Rd2pdf
Installing file: /usr/lib/R/bin/Rdconv
Installing file: /usr/lib/R/bin/Stangle
Installing file: /usr/lib/R/bin/COMPILE
Installing file: /usr/lib/R/bin/LINK
Installing file: /usr/lib/R/bin/Rcmd
Installing file: /usr/lib/R/bin/R
...`

R and tcl/tk

Hi,

Continuing on my attempt to build a SAPP for R, I have trouble setting up tcl/tk and probably more generally X11 display.

When you run R, and then the install.packages() function, it tries to display a graphical window with a list of repositories, using tk bindings.
So I tried to add what seems to be missing. Currently my spec is:

Name: R302
Exec: /usr/bin/R
%files:
/bin/sed
/usr/share/R/
/usr/share/tcltk/
/etc/fonts/fonts.conf
/etc/fonts/conf.d

The problem is that it crashes with a segfault:

./R302.sapp -q
> install.packages("dplyr")
Installing package into ‘/home/karl/R/x86_64-pc-linux-gnu-library/3.0’
(as ‘lib’ is unspecified)
--- Please select a CRAN mirror for use in this session ---

 *** caught segfault ***
address 0x500000000, cause 'memory not mapped'

Traceback:
 1: structure(.External(.C_dotTclObjv, objv), class = "tclObj")
 2: .Tcl.objv(.Tcl.args.objv(...))
 3: tcl(type, win, ...)
 4: tkwidget(parent, "ttk::label", ...)
 5: ttklabel(dlg, text = title, foreground = "blue")
 6: tcltk::tk_select.list(choices, preselect, multiple, title)
 7: select.list(choices, multiple = FALSE, title = title, graphics = TRUE)
 8: menu(m[, 1L], graphics, "CRAN mirror")
 9: chooseCRANmirror()
10: contrib.url(repos, type)
11: grep("^file:", contriburl)
12: install.packages("dplyr")

How could I debug this ?
thanks.

/proc should not be bind mounted

should mount new proc. The container running in a new pid namespace would see inaccurate content in /proc for all the processes that "don't exist".

mount a new proc. This would give the same data, except for things like processes, which would be tailored to the container environment rather than reflect the parent system.

There may be similar considerations for /sys, but I'm not 100% sure. Might err on the side of doing new mount for sys if I had to choose personally.

bootstrap problems on SL6.7

Hello,

I am using singularity v2.0 and I had the following errors when bootstraping the examples:

[root@test singularity]# singularity bootstrap /tmp/container.img examples/centos.def
Setting RELEASE=7
Loaded plugins: refresh-packagekit, security
Setting up Install Process
os-base                                                                                        | 3.6 kB     00:00
Not using downloaded repomd.xml because it is older than what we have:
  Current   : Fri Feb 12 16:54:21 2016
  Downloaded: Wed Dec  9 22:35:45 2015
Error: xz compression not available

[root@test singularity]# singularity bootstrap /tmp/container.img examples/scientific.def
Setting RELEASE=7
Loaded plugins: refresh-packagekit, security
Setting up Install Process
os-base                                                                                        | 3.7 kB     00:00
Error: xz compression not available

[root@test singularity]# singularity bootstrap /tmp/container.img examples/debian.def
ERROR: debootstrap is not in PATH... Perhaps 'apt-get install' it?

[root@test singularity]# singularity bootstrap /tmp/container.img examples/ubuntu.def
ERROR: debootstrap is not in PATH... Perhaps 'apt-get install' it?

Is there something that I am doing wrong?

Thanks,
Sebastian.

image-building example fails

I'll give up at this point -- I probably should have batched things, apologies.

I tried the image-building example in the readme on centos7. Those two functions definitely aren't defined anywhere:

[vagrant@localhost ~]$ cat centos.def
VERSION=7
PackageRepo "http://mirror.centos.org/centos-${VERSION}/${VERSION}/os/\$basearch/"
Initalize
InstallPkgs centos-release coreutils python strace vim-minimal
Finalize
[vagrant@localhost ~]$ sudo singularity bootstrap /tmp/Centos-7.img centos.def
Checking for yum...
./centos.def: line 2: PackageRepo: command not found
./centos.def: line 3: Initalize: command not found
CRITICAL:yum.cli:Config error: Error accessing file for config file:///mnt//yum.conf

I wondered if test.sh would help, but that looks obsolete now.

By the way, on the readme: it says that Red Hat doesn't provide debootstrap. However it is in EPEL and Fedora https://apps.fedoraproject.org/packages/debootstrap, as yum is in Debian.

Support multiple MirrorURL entries in the image definition file

For a richer bootstrap please support multiple MirrorURL entries in the image def file.

2 issues while trying to build pydmesg

Hi,

I tried to create a sapp of a python script ( https://gist.githubusercontent.com/dopuskh3/9908ac6b31133eb54331/raw/74beaa1f2c64e09b6152b1ac8849af172c12b6e1/pydmesg ), but I had some issues :

/opt/singularity/singularity-1.0/bin/singularity specgen /media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg
...
WROTE: pydmesg.sspec

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/singularity build pydmesg.sspec
....
Running post scriptlet
Defining container shell environment
Building singularity main runscript
Building singularity test runscript
Building singularity exec runscript
Running test...
Including specfile
Writing metadata
Creating output SAPP container
WROTE: pydmesg.sapp
Cleaning up temporary files...

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/singularity install pydmesg.sapp
Installing: pydmesg (5681a8f5-31f9-49b5-986e-61f06dac48a0)
Done...

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/singularity check pydmesg.sapp
ERROR: Singularity container is not installed: pydmesg.sapp

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/singularity install pydmesg.sapp
Singularity container is already installed: pydmesg (5681a8f5-31f9-49b5-986e-61f06dac48a0)

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/singularity run pydmesg.sapp
/.singularity.exec: 2: exec: /media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg: not found

remy@atlas:/tmp$ grep /media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg pydmesg.sspec 
exec /media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg "$@"
/media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg
/media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg

remy@atlas:/tmp$/opt/singularity/singularity-1.0/bin/singularity strace pydmesg.sapp
ERROR: Singularity container is not installed: pydmesg.sapp

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/singularity list |grep 5681a8f5-31f9-49b5-986e-61f06dac48a0
pydmesg              5681a8f5-31f9-49b5-986e-61f06dac48a0           33M Generated by 'singularity specgen'

remy@atlas:/tmp$ ./pydmesg.sapp 
-ne \e[0;31mERROR: File checksums not found for this SAPP
\e[0;39m

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/sapprun  pydmesg.sapp 
-ne \e[0;31mERROR: SAPP directory not found
\e[0;39m
-ne \e[0;31mNot launching SAPP container
\e[0;39m

I tried also with relative path, same error.

After editing sspec file, it seems it missed /var/log/dmesg and /proc/uptime. I added it and reproduce the creation of the sapp, but obviously, I still get that error.

I also tried with another "pydmesg" piece of code but I had the same issue.

BTW, I wonder how singularity analyze python code (import stuff, open, or when we append a path to PYTHONPATH (e.g. : sys.path.append('/path/to/whatever')...), or if it just add the whole PYTHONPATH directory to sapp file.

Regards,
Remy

bug importing directory

Hello,
Using singularity 0f5452b,
the build imports only a subset of the files and put them in wrong places.

For instance, it installs /usr/local/R-3.0.2/lib/R/library/base/help/scoping.R which does not exist,
it is actually /usr/local/R-3.0.2/lib/R/library/base/demo/scoping.R

If I change the %files spec to /usr/local/R-3.0.2/lib/R/library/base/*, it imports all files but flatten the directory.

r.spec:
Name: R302
Exec: /usr/local/bin/R
%files
/usr/local/R-3.0.2/lib/R/library/base

the directory base looks like:

drwxr-xr-x   7 root root  4096 Feb  5  2014 ./
drwxr-xr-x 258 root root 12288 May 15  2015 ../
-rw-r--r--   1 root root   954 Feb  5  2014 CITATION
drwxr-xr-x   2 root root  4096 Feb  5  2014 demo/
-rw-r--r--   1 root root   267 Feb  5  2014 DESCRIPTION
drwxr-xr-x   2 root root  4096 Feb  5  2014 help/
drwxr-xr-x   2 root root  4096 Feb  5  2014 html/
-rw-r--r--   1 root root 22491 Feb  5  2014 INDEX
drwxr-xr-x   2 root root  4096 Feb  5  2014 Meta/
drwxr-xr-x   2 root root  4096 Feb  5  2014 R/

/usr/local/R-3.0.2/lib/R/library/base/demo:
total 28
drwxr-xr-x 2 root root 4096 Feb  5  2014 ./
drwxr-xr-x 7 root root 4096 Feb  5  2014 ../
-rw-r--r-- 1 root root 1179 Feb  5  2014 error.catching.R
-rw-r--r-- 1 root root 4252 Feb  5  2014 is.things.R
-rw-r--r-- 1 root root 2115 Feb  5  2014 recursion.R
-rw-r--r-- 1 root root 1560 Feb  5  2014 scoping.R

/usr/local/R-3.0.2/lib/R/library/base/help:
total 1892
drwxr-xr-x 2 root root    4096 Feb  5  2014 ./
drwxr-xr-x 7 root root    4096 Feb  5  2014 ../
-rw-r--r-- 1 root root   11185 Feb  5  2014 aliases.rds
-rw-r--r-- 1 root root   31346 Feb  5  2014 AnIndex
-rw-r--r-- 1 root root 1867084 Feb  5  2014 base.rdb
-rw-r--r-- 1 root root    8536 Feb  5  2014 base.rdx
-rw-r--r-- 1 root root    2868 Feb  5  2014 paths.rds
...

when I build:

singularity build ./r.spec
Creating temporary container path...
Singularity App Container Name: R302.sapp
Running build scriptlet
Evaluating: packages
Building the runtime level: 1
Installing file: /bin/sh
Installing file: /home/karl/bin/libstderred.so
Installing file: /lib/x86_64-linux-gnu/libdl.so.2
Installing file: /lib/x86_64-linux-gnu/libc.so.6
Installing file: /lib64/ld-linux-x86-64.so.2
Installing file: /bin/strace
Installing file: /lib64/libnss_files.so.2
Installing file: /lib/i386-linux-gnu/libc.so.6
Installing file: /lib/ld-linux.so.2
Evaluating: Exec
Installing file: /bin/R
Installing file: /bin/bash
Installing file: /lib/x86_64-linux-gnu/libtinfo.so.5
Evaluating: files
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/base.rdb
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/AnIndex
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/base.rdx
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/paths.rds
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/aliases.rds
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/scoping.R
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/is.things.R
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/error.catching.R
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/recursion.R
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/00Index.html
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/R.css
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/links.rds
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/Rd.rds
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/hsearch.rds
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/demo.rds
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/package.rds
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/base.rdb
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/base
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/Rprofile
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/base.rdx
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/INDEX
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/CITATION
Installing file: /usr/local/R-3.0.2/lib/R/library/base/help/DESCRIPTION
Running install scriptlet
Creating template configs
  /etc/nsswitch.conf
  /etc/resolv.conf
  /etc/hosts
Checking for files that need santizing
Cleaning up device files
Running post scriptlet
Defining container shell envrionment
Building singularity main runscript
Building singularity test runscript
Building singularity exec runscript
Running test...
Hello from within the container... (no test code defined)
Including specfile
Writing metadata
Creating output SAPP container
WROTE: R302.sapp