singularityhub / docker2singularity Goto Github PK

Hello,
I'm trying to create a writable image:
sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v /opt/SingularityImages/test:/output --privileged -t --rm singularityware/docker2singularity -w fcpindi/c-pac

but it errors out with:
/docker2singularity.sh: line 110: $1: unbound variable

It works fine when not using the -w or -f flags, but I need to make a modification before converting to an .simg.

I've tried escaping the slash in the image name, and using quotes.

Can you give men some insight as to what the error is indicating?

Many thanks,
Louis

Hi! For silly reasons involving permissions and firewalls, I have a CI workflow that uses docker2singularity to convert a local Docker image to Singularity format, which then gets handed off to the production system via an intermediate file server that both can connect to. For other silly reasons, the CI wipes its Docker layer cache nightly, so I always work with whatever happens to be tagged as singularityware/docker2singularity:v2.6.

Last week, however, the images produced by the CI stopped working with Singularity 2.6.1, printing ERROR : Unknown image format/type when I try to run it. I can reproduce this with any of the subcommands. For example, on a working image:

singularity -v inspect ampel-v0.6.12.simg
Increasing verbosity level (2)
Singularity version: 2.6.1-dist
Exec'ing: /usr/libexec/singularity/cli/inspect.exec
VERBOSE: Set messagelevel to: 2
VERBOSE: Initialize configuration file: /etc/singularity/singularity.conf
VERBOSE: Initializing Singularity Registry
VERBOSE: No autofs bug path in configuration, skipping
VERBOSE: Found available loop device: /dev/loop37
VERBOSE: Using loop device: /dev/loop37
VERBOSE: Mounting squashfs image: /dev/loop37 -> /var/singularity/mnt/final
VERBOSE: Running command: /bin/bash
{
    "org.label-schema.singularity.build": "squashfs",
    "org.label-schema.docker.version": "18.09.6",
    "org.label-schema.schema-version": "1.0",
    "org.label-schema.singularity.build-type": "docker2singularity",
    "org.label-schema.docker.id": "sha256:bfcb5fd8d67e2c1d5ebb9e81840095cce5a17753179b4eb5697a117c3805dfdd",
    "org.label-schema.build-date": "2019-05-15-14:47:48",
    "org.label-schema.singularity.version": "2.6.0-dist",
    "org.label-schema.docker.created": "2019-05-15"
}

while on a recent image, I get:

singularity -v inspect ampel-v0.6.simg
Increasing verbosity level (2)
Singularity version: 2.6.1-dist
Exec'ing: /usr/libexec/singularity/cli/inspect.exec
VERBOSE: Set messagelevel to: 2
VERBOSE: Initialize configuration file: /etc/singularity/singularity.conf
VERBOSE: Initializing Singularity Registry
VERBOSE: No autofs bug path in configuration, skipping
VERBOSE: File is not a valid SquashFS image
VERBOSE: File is not a valid EXT3 image

Any idea what's going wrong here?

Singularity: sexec (U=0,P=189)> Command=exec, Container=/tmp/jinja-2016-11-01-68a7b91a1cb0.img, CWD=/, Arg1=/bin/sh
find: unrecognized: -executable
BusyBox v1.24.2 (2016-11-01 12:56:05 GMT) multi-call binary.

Usage: find [-HL] [PATH]... [OPTIONS] [ACTIONS]

Failure at: https://github.com/singularityware/docker2singularity/blob/master/docker2singularity.sh#L149

BusyBox's find has the following options:

BusyBox v1.24.2 (2016-11-01 12:56:05 GMT) multi-call binary.

Usage: find [-HL] [PATH]... [OPTIONS] [ACTIONS]

Search for files and perform actions on them.
First failed action stops processing of current file.
Defaults: PATH is current directory, action is '-print'

	-L,-follow	Follow symlinks
	-H		...on command line only
	-xdev		Don't descend directories on other filesystems
	-maxdepth N	Descend at most N levels. -maxdepth 0 applies
			actions to command line arguments only
	-mindepth N	Don't act on first N levels
	-depth		Act on directory *after* traversing it

Actions:
	( ACTIONS )	Group actions for -o / -a
	! ACT		Invert ACT's success/failure
	ACT1 [-a] ACT2	If ACT1 fails, stop, else do ACT2
	ACT1 -o ACT2	If ACT1 succeeds, stop, else do ACT2
			Note: -a has higher priority than -o
	-name PATTERN	Match file name (w/o directory name) to PATTERN
	-iname PATTERN	Case insensitive -name
	-path PATTERN	Match path to PATTERN
	-ipath PATTERN	Case insensitive -path
	-regex PATTERN	Match path to regex PATTERN
	-type X		File type is X (one of: f,d,l,b,c,...)
	-perm MASK	At least one mask bit (+MASK), all bits (-MASK),
			or exactly MASK bits are set in file's mode
	-mtime DAYS	mtime is greater than (+N), less than (-N),
			or exactly N days in the past
	-mmin MINS	mtime is greater than (+N), less than (-N),
			or exactly N minutes in the past
	-newer FILE	mtime is more recent than FILE's
	-inum N		File has inode number N
	-user NAME/ID	File is owned by given user
	-group NAME/ID	File is owned by given group
	-size N[bck]	File size is N (c:bytes,k:kbytes,b:512 bytes(def.))
			+/-N: file size is bigger/smaller than N
	-links N	Number of links is greater than (+N), less than (-N),
			or exactly N
	-prune		If current file is directory, don't descend into it
If none of the following actions is specified, -print is assumed
	-print		Print file name
	-print0		Print file name, NUL terminated
	-exec CMD ARG ;	Run CMD with all instances of {} replaced by
			file name. Fails if CMD exits with nonzero
	-exec CMD ARG + Run CMD with {} replaced by list of file names
	-delete		Delete current file/directory. Turns on -depth option

Hi,
I was trying to convert a local docker image(xxx.tar) to singularity one, but I got the following error on my linux server. I am new to docker and might make some really basic mistakes since I failed to find any information related to this output.

docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: exec: "tail": executable file not found in $PATH: unknown.

Any info on how to convert a docker image to singularity image would also be highly appreciated.
And if there is any additional information I could provide to help, please just let me know.
Thanks.

When I run this, I get an image file with a random string appended, like this:

stevekm_phoenix-demo_demo1-2018-03-15-94a3b28d5358.img

I would rather be able to tell it what to name the output file, or at the least turn off the random string, so that if I run the same command repeatedly (e.g. in a Makefile) the same file gets produced & overwritten every time.

In this case, I am building it off the container here it would be better if the output was named something like stevekm_phoenix-demo_demo1.img

I'm not sure why this is, but when I run "grep -q Buildroot /etc/issue" on my Busybox container, there is no exit code, so the fix in #10 doesn't work. However, if I run that command and then check $?, it contains the expected '0' exit code. So 'if $? ; then ...' it works fine.

10-docker2singularity.sh currently contains:

#!/bin/sh
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

It's copied to /.singularity.d/env/10-docker2singularity.sh in the resulting singularity image.

However, since 10-docker2singularity.sh sorts (ASCII) after 10-docker.sh, any PATH set by a Dockerfile ENV directive (represented in 10-docker.sh) will be overwritten with the PATH set in 10-docker2singularity.sh.

Is 10-docker2singularity.sh still necessary (naive question)? If so, could it perhaps be renamed to, e.g., 9-docker2singularity.sh, so any PATH that is set in the source Docker image via Dockerfile ENV directive will still apply?

docker2singularity must run in a privileged containers due to its use of /dev/loop0 (and possibly other reasons). This makes it impossible to run in circleCI, which explicitly runs its containers without privileges. It would be awesome if a workaround could be found to remove the need on privilege so that we can do singularity deployments from circleCI.

I am tying to import an alpine images (Alpine is based on busybox and musl libc ):

docker run  -v /var/run/docker.sock:/var/run/docker.sock  -v $PWD:/output  --privileged -t --rm   singularityware/docker2singularity  alpine:latest

However, it fails

Size: 6 MB for the singularity container
(1/9) Creating an empty image...
Creating a sparse image with a maximum size of 6MiB...
Using given image size of 6
Formatting image (/sbin/mkfs.ext3)
Done. Image can be found at: /tmp/alpine_latest-2017-03-03-ee74f4f51a7e.img
(2/9) Importing filesystem...
(3/9) Bootstrapping...
(4/9) Adding run script...
(5/9) Setting ENV variables...
Singularity: sexec (U=0,P=144)> Command=exec, Container=/tmp/alpine_latest-2017-03-03-ee74f4f51a7e.img, CWD=/, Arg1=/bin/sh
(6/9) Adding mount points...
Singularity: sexec (U=0,P=150)> Command=exec, Container=/tmp/alpine_latest-2017-03-03-ee74f4f51a7e.img, CWD=/, Arg1=/bin/sh
(7/9) Fixing permissions...
Singularity: sexec (U=0,P=156)> Command=exec, Container=/tmp/alpine_latest-2017-03-03-ee74f4f51a7e.img, CWD=/, Arg1=/bin/sh
We're not running on BusyBox/Buildroot
Singularity: sexec (U=0,P=191)> Command=exec, Container=/tmp/alpine_latest-2017-03-03-ee74f4f51a7e.img, CWD=/, Arg1=/bin/sh
find: unrecognized: -executable
BusyBox v1.25.1 (2016-10-26 16:15:20 GMT) multi-call binary.

Usage: find [-HL] [PATH]... [OPTIONS] [ACTIONS]

Search for files and perform actions on them.
First failed action stops processing of current file.
Defaults: PATH is current directory, action is '-print'

	-L,-follow	Follow symlinks
	-H		...on command line only
	-xdev		Don't descend directories on other filesystems
	-maxdepth N	Descend at most N levels. -maxdepth 0 applies
			actions to command line arguments only
	-mindepth N	Don't act on first N levels
	-depth		Act on directory *after* traversing it

Actions:
	( ACTIONS )	Group actions for -o / -a
	! ACT		Invert ACT's success/failure
	ACT1 [-a] ACT2	If ACT1 fails, stop, else do ACT2
	ACT1 -o ACT2	If ACT1 succeeds, stop, else do ACT2
			Note: -a has higher priority than -o
	-name PATTERN	Match file name (w/o directory name) to PATTERN
	-iname PATTERN	Case insensitive -name
	-path PATTERN	Match path to PATTERN
	-ipath PATTERN	Case insensitive -path
	-regex PATTERN	Match path to regex PATTERN
	-type X		File type is X (one of: f,d,l,b,c,...)
	-perm MASK	At least one mask bit (+MASK), all bits (-MASK),
			or exactly MASK bits are set in file's mode
	-mtime DAYS	mtime is greater than (+N), less than (-N),
			or exactly N days in the past
	-mmin MINS	mtime is greater than (+N), less than (-N),
			or exactly N minutes in the past
	-newer FILE	mtime is more recent than FILE's
	-inum N		File has inode number N
	-user NAME/ID	File is owned by given user
	-group NAME/ID	File is owned by given group
	-size N[bck]	File size is N (c:bytes,k:kbytes,b:512 bytes(def.))
			+/-N: file size is bigger/smaller than N
	-links N	Number of links is greater than (+N), less than (-N),
			or exactly N
	-prune		If current file is directory, don't descend into it
If none of the following actions is specified, -print is assumed
	-print		Print file name
	-print0		Print file name, NUL terminated
	-exec CMD ARG ;	Run CMD with all instances of {} replaced by
			file name. Fails if CMD exits with nonzero
	-exec CMD ARG + Run CMD with {} replaced by list of file names
	-delete		Delete current file/directory. Turns on -depth option

Specifically, we want the docker2singularity image to provide a Dockerized access to native Singularity, which uses the Docker Registry API to generate images. This is to guarantee that images created are equivalent, which (I don't believe) is currently the case! See apptainer/singularity#396 (comment) for details, and this likely will fix #10

I need to create a Singularity container for Summit, a ppc system. I do not have root on power hardware, so I am building the container in docker with qemu, and then trying to convert via docker2singuarity:

sudo docker run -v /var/run/docker.sock:/var/run/docker.sock \
  -v /tmp/test:/output --privileged -t --rm \
  singularityware/docker2singularity \
  my-image

I have done this successfully in the past (~2 years ago), but it no longer works as I expect. The Singularity image is amd64, even though it contains ppc64le software. My best guess is that this is due to a change in Singularity itself, although I don't actually know what is going on.

Using docker2singularity, is there any way to force the Singularity image to be ppc64le?

Running Docker 19.03.5 on macOS 10.14.6. When I run docker2singularity with the following code, everything looks right, but no Singularity image is output anywhere on my system:

docker run \ -v /var/run/docker.sock:/var/run/docker.sock \ -v \Users\my\path\to\folder:/output \ --privileged -t --rm \ singularityware\docker2singularity \ mypackage:version

I get the following output:
`Image Format: squashfs
Docker Image: brainkey/brainage-standalone:bk-brainage-preproc

Inspected Size: 1712 MB

(1/10) Creating a build sandbox...
(2/10) Exporting filesystem...
(3/10) Creating labels...
(4/10) Adding run script...
(5/10) Setting ENV variables...
(6/10) Adding mount points...
(7/10) Fixing permissions...
(8/10) Stopping and removing the container...
(9/10) Building squashfs container...
INFO: Starting build...
INFO: Creating SIF file...
INFO: Build complete: /tmp/brainkey_brainage-standalone_bk-brainage-preproc-2019-12-11-b1bd81913114.simg
(10/10) Moving the image to the output folder...
765,931,520 100% 247.94MB/s 0:00:02 (xfr#1, to-chk=0/1)
Final Size: 731MB`

A search through my full system doesn't turn up a .simg. What's the issue?

I'm getting this error when I try to convert the bids/cpac from docker to singularity:

sudo docker run --privileged -ti --rm -v /var/run/docker.sock:/var/run/docke
r.sock -v /home/egarza/MRI/apps/singularity_images:/output singularityware/docker2singularity bids/cpac
Size: 5533 MB for the singularity container
(1/9) Creating an empty image...
Creating a sparse image with a maximum size of 5533MiB...
Using given image size of 5533
Formatting image (/sbin/mkfs.ext3)
Done. Image can be found at: /tmp/bids_cpac-2017-03-29-71120146c4f4.img
(2/9) Importing filesystem...
(3/9) Bootstrapping...
(4/9) Adding run script...
(5/9) Setting ENV variables...
Singularity: sexec (U=0,P=146)> Command=exec, Container=/tmp/bids_cpac-2017-03-29-71120146c4f4.img, CWD=/, Arg1=/bin/sh
(6/9) Adding mount points...
Singularity: sexec (U=0,P=152)> Command=exec, Container=/tmp/bids_cpac-2017-03-29-71120146c4f4.img, CWD=/, Arg1=/bin/sh
(7/9) Fixing permissions...
Singularity: sexec (U=0,P=158)> Command=exec, Container=/tmp/bids_cpac-2017-03-29-71120146c4f4.img, CWD=/, Arg1=/bin/sh
We're running on BusyBox/Buildroot
Singularity: sexec (U=0,P=201)> Command=exec, Container=/tmp/bids_cpac-2017-03-29-71120146c4f4.img, CWD=/, Arg1=/bin/sh
find: `/proc/2/task/2/fd/11': No such file or directory
find: `/proc/2/task/2/fd/11': No such file or directory
find: `/proc/2/task/2/fdinfo/11': No such file or directory
find: `/proc/2/task/2/fdinfo/11': No such file or directory
find: `/proc/2/fd/11': No such file or directory
find: `/proc/2/fd/11': No such file or directory
find: `/proc/2/fdinfo/11': No such file or directory
find: `/proc/2/fdinfo/11': No such file or directory

Any help is appreciated,

Thanks

Eduardo

Hi,

After pulling the most recent docker2singularity version, it now breaks when trying to convert various images. Could this be related to the changes made in the following PR? 04c465d

See below for error messages.

docker run -v /var/run/docker.sock:/var/run/docker.sock -v ~/Desktop:/output --privileged -t --rm singularityware/docker2singularity continuumio/miniconda
Size: 655 MB for the singularity container
(1/9) Creating an empty image...
Creating a sparse image with a maximum size of 655MiB...
Using given image size of 655
Formatting image (/sbin/mkfs.ext3)
Done. Image can be found at: /tmp/continuumio_miniconda-2017-06-04-656a1fe1d99e.img
(2/9) Importing filesystem...
(3/9) Bootstrapping...
(4/9) Adding run script...
(5/9) Setting ENV variables...
Singularity: sexec (U=0,P=146)> Command=exec, Container=/tmp/continuumio_miniconda-2017-06-04-656a1fe1d99e.img, CWD=/, Arg1=/bin/sh
(6/9) Adding mount points...
Singularity: sexec (U=0,P=152)> Command=exec, Container=/tmp/continuumio_miniconda-2017-06-04-656a1fe1d99e.img, CWD=/, Arg1=/bin/sh
(7/9) Fixing permissions...
Singularity: sexec (U=0,P=158)> Command=exec, Container=/tmp/continuumio_miniconda-2017-06-04-656a1fe1d99e.img, CWD=/, Arg1=/bin/sh
We're running on BusyBox/Buildroot
Singularity: sexec (U=0,P=199)> Command=exec, Container=/tmp/continuumio_miniconda-2017-06-04-656a1fe1d99e.img, CWD=/, Arg1=/bin/sh
find: /proc/2/task/2/fd/11': No such file or directory find: /proc/2/task/2/fd/11': No such file or directory
find: /proc/2/task/2/fdinfo/11': No such file or directory find: /proc/2/task/2/fdinfo/11': No such file or directory
find: /proc/2/fd/11': No such file or directory find: /proc/2/fd/11': No such file or directory
find: /proc/2/fdinfo/11': No such file or directory find: /proc/2/fdinfo/11': No such file or directory

Docker info:

Kernel Version: 4.9.31-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64

Hi friend, what is your question?

as title says.

Hello,

I plan on building Singularity images for our HPC cluster. There are some that we will maintain on Docker hub or some that are maintained by others. I plan producing Singularity recipes that fetch these docker images and build corresponding Singularity containers. Is this tool docker2singularity a proper way to do it? E.g., if I updated and maintain https://hub.docker.com/r/openiss/rt-mocap-vfx and then create a Singularity file that transforms to the corresponding container here?

• Possibly related to #98
• The example here uses Bootstrap:docker, not sure if it's relevant

Thank you!

Did

> docker run -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/output --privileged -t --rm singularityware/docker2singularity:1.11 centos:7

and then

> sudo singularity exec -w ./centos_7-2016-09-06-a66c9b92ef09.img /bin/bash
Singularity.centos_7-2016-09-06-a66c9b92ef09.img> yum update

which fails due to non space available

...
  32:bind-license-9.9.4-29.el7_2.4.noarch: Insufficient space in download directory /var/cache/yum/x86_64/7/updates/packages
    * free   0 
    * needed 82 k

Singularity.centos_7-2016-09-06-a66c9b92ef09.img> df /var/cache/yum/x86_64/7/updates/packages
Filesystem     1K-blocks   Used Available Use% Mounted on
singularity       284317 283750         0 100% /

My Docker images are huge, and I keep getting out of space errors when I try to build with the basic docker2singularity commandline.

I know I can probably (on my own machine anyway) allocate more space for my Docker VM so it would have a larger /tmp slice, but that's not always possible.

This has lead me to write a bash script to setup various mountpoints for docker2singularity.

It would be nice if some of these things could be better documented, it would also be nice if the docker2singularity.sh script had more control over where its temporary directories go.

Here is the most important part of what I have so far, but it still doesn't work.

docker run -v /var/run/docker.sock:/var/run/docker.sock \
-v ${OUTPUT_DIRECTORY}:/output \
-v ${TMP_FOR_BUILD}:/tmp -e SINGULARITY_TMPDIR=/tmp/buildtmp \
-v ${TMP_FOR_BUILD}/singcache:/tmp/singcache -e SINGULARITY_CACHEDIR=/tmp/singcache \
-v ${TMP_FOR_BUILD}/tmpother:/tmp/tmpother -e TMP=/tmp/tmpother -e TEMP=/tmp/tmpother -e TEMPDIR=/tmp/tmpother \
--privileged -t --rm \
quay.io/singularity/docker2singularity \
${BUILD_IMAGE}

Hi, I am getting a tar error while running singularityware/docker2singularity on a mac, running macOS Seirra. The first post I came across on the issue suggest I need to install bdstar. Can anyone confirm or provide another recommendation?

docker run -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/mriqc_singularity --privileged -t --rm singularityware/docker2singularity poldracklab/mriqc

Image Format: squashfs
Docker Image: poldracklab/mriqc

Inspected Size: 7627 MB

(1/10) Creating a build sandbox...
(2/10) Exporting filesystem...
tar: usr/local/miniconda/pkgs/ncurses-5.9-10/share/terminfo/X: Directory renamed before its status could be extracted

Hello:

I created a docker image rstudio-4.0.4-20211017, I wanted to converted it to a singularity file. I used the following method and it worked perfect for me in past, but today it failed with the error message:

docker run -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/test:/output --privileged -t --rm quay.io/singularity/docker2singularity rstudio-4.0.4-20211017

Image Format: squashfs
Docker Image: rstudio-4.0.4-20211017

Inspected Size: 2883 MB

(1/10) Creating a build sandbox...
(2/10) Exporting filesystem...
(3/10) Creating labels...
(4/10) Adding run script...
(5/10) Setting ENV variables...
(6/10) Adding mount points...
(7/10) Fixing permissions...
(8/10) Stopping and removing the container...
(9/10) Building squashfs container...
INFO: Starting build...
INFO: Creating SIF file...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xa8 pc=0x5635685b337a]

goroutine 1 [running]:
github.com/sylabs/sif/v2/pkg/sif.CreateContainerAtPath(0x7ffccd453e35, 0x37, 0xc000586b80, 0x1, 0x1, 0xc0004fef00, 0x2, 0x4)
github.com/sylabs/sif/[email protected]/pkg/sif/create.go:269 +0x17a
github.com/sylabs/singularity/internal/pkg/build/assemblers.createSIF(0x7ffccd453e35, 0x37, 0xc0003b0000, 0xc00016c690, 0x2d, 0x0, 0x563568cd707e, 0x5, 0x0, 0x0)
github.com/sylabs/[email protected]/internal/pkg/build/assemblers/sif.go:121 +0x809
github.com/sylabs/singularity/internal/pkg/build/assemblers.(*SIFAssembler).Assemble(0xc000388de0, 0xc0003b0000, 0x7ffccd453e35, 0x37, 0x0, 0x0)
github.com/sylabs/[email protected]/internal/pkg/build/assemblers/sif.go:212 +0x600
github.com/sylabs/singularity/internal/pkg/build.(*stage).Assemble(...)
github.com/sylabs/[email protected]/internal/pkg/build/stage.go:42
github.com/sylabs/singularity/internal/pkg/build.(*Build).Full(0xc0001aa180, 0x5635694e2980, 0xc000042058, 0x0, 0x0)
github.com/sylabs/[email protected]/internal/pkg/build/build.go:466 +0xb85
github.com/sylabs/singularity/cmd/internal/cli.runBuildLocal(0x5635694e2980, 0xc000042058, 0x563569d6e8a0, 0x7ffccd453e35, 0x37, 0x7ffccd453e6d, 0x39)
github.com/sylabs/[email protected]/cmd/internal/cli/build_linux.go:348 +0x840
github.com/sylabs/singularity/cmd/internal/cli.runBuild(0x563569d6e8a0, 0xc000264140, 0x2, 0x2)
github.com/sylabs/[email protected]/cmd/internal/cli/build_linux.go:134 +0x34a
github.com/spf13/cobra.(*Command).execute(0x563569d6e8a0, 0xc00003c0e0, 0x2, 0x2, 0x563569d6e8a0, 0xc00003c0e0)
github.com/spf13/[email protected]/command.go:860 +0x2ac
github.com/spf13/cobra.(*Command).ExecuteC(0x563569d792a0, 0x563569495578, 0xc0000a3860, 0xc00031dfb0)
github.com/spf13/[email protected]/command.go:974 +0x35c
github.com/spf13/cobra.(*Command).Execute(...)
github.com/spf13/[email protected]/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
github.com/spf13/[email protected]/command.go:895
github.com/sylabs/singularity/cmd/internal/cli.ExecuteSingularity()
github.com/sylabs/[email protected]/cmd/internal/cli/singularity.go:430 +0x20e
main.main()
github.com/sylabs/[email protected]/cmd/singularity/cli.go:19 +0x52

Today, before seeing the above error message, I tried the above command and hit an error:
FATAL: While performing build: while creating SIF: while creating container: writing data object for SIF file: copying data object file to SIF file: write /tmp/rstudio-4.0.4-20211017-2021-10-18-ae6bf9b2ebd6.sif: no space left on device

So, I used the command "docker system prune -a" to remove exited containers and unused images. I also restarted docker.

Thank you very much for your suggestion!

Regards,
Rong

The documentation at "http://singularity.lbl.gov/docs-docker" indicates that docker2singularity will produce a singularity image in the current directory

To produce a Singularity image of “ubuntu:latest” in the present working directory.

But docker2singularity outputs to /output/

Would it be a bad idea to use docker2singularity to build Singularity images from Singularity definition files with singularity build? Lightly modifying the example in the Singularity 3.7 User Guide
, modifying the %files section to copy some file ("test.py") in the current working directory, and using the docker engine from Docker Desktop for Mac 20.10.3:

$ cat singularity.def 
Bootstrap: library
From: ubuntu:18.04
Stage: build

%setup
    touch /file1
    touch ${SINGULARITY_ROOTFS}/file2

%files
    test.py /opt

%environment
    export LISTEN_PORT=12345
    export LC_ALL=C

%post
    apt-get update && apt-get install -y netcat
    NOW=`date`
    echo "export NOW=\"${NOW}\"" >> $SINGULARITY_ENVIRONMENT

%runscript
    echo "Container was created $NOW"
    echo "Arguments received: $*"
    exec echo "$@"

%startscript
    nc -lp $LISTEN_PORT

%test
    grep -q NAME=\"Ubuntu\" /etc/os-release
    if [ $? -eq 0 ]; then
        echo "Container base is Ubuntu as expected."
    else
        echo "Container base is not Ubuntu."
        exit 1
    fi

%labels
    Author [email protected]
    Version v0.0.1

%help
    This is a demo container used to illustrate a def file that uses all
    supported sections.
$ docker run -t --rm --privileged -v /etc/localtime -v $(pwd):/mnt -w /mnt --entrypoint=singularity quay.io/singularity/docker2singularity:v3.7.1 build ubuntu.sif singularity.def 
INFO:    Starting build...
INFO:    Downloading library image
52.1MiB / 52.1MiB [==============================================================================] 100 % 18.3 MiB/s 0s
INFO:    Verifying bootstrap image /root/.singularity/cache/library/sha256.35055a42993db1469734fd488feddc747bdaabc0956b3aabf72c9b1f9385b209
WARNING: integrity: signature not found for object group 1
WARNING: Bootstrap image could not be verified, but build will continue.
INFO:    Running setup scriptlet
+ touch /file1
+ touch /tmp/build-temp-471078291/rootfs/file2
INFO:    Copying test.py to /tmp/build-temp-471078291/rootfs/opt
INFO:    Running post scriptlet
+ apt-get update
Hit:1 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://us.archive.ubuntu.com/ubuntu bionic/main Translation-en [516 kB]
Fetched 516 kB in 1s (693 kB/s)        
Reading package lists... Done
+ apt-get install -y netcat
Reading package lists... Done
Building dependency tree... Done
Note, selecting 'netcat-openbsd' instead of 'netcat'
The following additional packages will be installed:
  libbsd0
The following NEW packages will be installed:
  libbsd0 netcat-openbsd
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 81.0 kB of archives.
After this operation, 270 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 libbsd0 amd64 0.8.7-1 [41.5 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 netcat-openbsd amd64 1.187-1 [39.6 kB]
Fetched 81.0 kB in 0s (322 kB/s)     
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libbsd0:amd64.
(Reading database ... 4006 files and directories currently installed.)
Preparing to unpack .../libbsd0_0.8.7-1_amd64.deb ...
Unpacking libbsd0:amd64 (0.8.7-1) ...
Selecting previously unselected package netcat-openbsd.
Preparing to unpack .../netcat-openbsd_1.187-1_amd64.deb ...
Unpacking netcat-openbsd (1.187-1) ...
Setting up libbsd0:amd64 (0.8.7-1) ...
Setting up netcat-openbsd (1.187-1) ...
update-alternatives: using /bin/nc.openbsd to provide /bin/nc (nc) in auto mode
Processing triggers for libc-bin (2.27-3ubuntu1) ...
+ date
+ NOW=Thu Feb 25 21:21:31 UTC 2021
+ echo export NOW="Thu Feb 25 21:21:31 UTC 2021"
INFO:    Adding help info
INFO:    Adding labels
INFO:    Adding environment to container
INFO:    Adding startscript
INFO:    Adding runscript
INFO:    Adding testscript
INFO:    Running testscript
Container base is Ubuntu as expected.
INFO:    Creating SIF file...
INFO:    Build complete: ubuntu.sif
$ docker run -it --rm --privileged -v /etc/localtime -v $(pwd):/mnt -w /mnt --entrypoint=singularity quay.io/singularity/docker2singularity:v3.7.1 shell ubuntu.sif
Singularity> cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
Singularity> ls /opt
test.py

If there aren't too many drawbacks to this approach (I realize this use of %setup wouldn't impact the "real" host), maybe it could be mentioned in the README.md (with an example)? Or would there be a more preferable base image to use?

Hi, I'm trying to create a Singularity image from a Docker image, which was successfully pulled and runs fine. But I get an invalid tar magic error when running:

$ sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v ~/Downloads:/output --privileged -t --rm quay.io/singularity/docker2singularity --name jukemir jukemir/representations_jukebox

Image Format: squashfs
Docker Image: jukemir/representations_jukebox
Container Name: jukemir

Inspected Size: 15941 MB

(1/10) Creating a build sandbox...
(2/10) Exporting filesystem...
tar: invalid tar magic

So, I opened an interactive shell (-it --entrypoint /bin/bash), installed BSDtar (apk add libarchive-tools), created a symbolic link to /usr/bin/bsdtar from /bin/tar, then it works now:

bash-5.0# ./docker2singularity.sh jukemir/representations_jukebox

Image Format: squashfs
Docker Image: jukemir/representations_jukebox

Inspected Size: 15941 MB

(1/10) Creating a build sandbox...
(2/10) Exporting filesystem...
(3/10) Creating labels...
(4/10) Adding run script...
(5/10) Setting ENV variables...
(6/10) Adding mount points...
(7/10) Fixing permissions...
(8/10) Stopping and removing the container...
(9/10) Building squashfs container...
INFO:    Starting build...

But could there be a workaround still using the BusyBox tar? Thanks!

When I try to convert the image uiuccs484parallelprog/cs484_student:latest from singularityhub, I get this cryptic message:

tar: can't open 'opt/rh/devtoolset-7/enable': Permission denied

I have no idea how to fix this. I remember there being something about the tar binary in this image having problems, but I think this is different, and that was long ago.

I don't even know how to investigate the cause of this.

Hi, I've previously been able to generate Singularity images using a much older version of the docker2singularity.sh script that I can then subsequently run as a non-root user.

Trying the latest master version, I can build the docker2singularity.sh docker container and run the tool to get a Singularity image file but I can only access the generated image as root due to the permissions of most of the files in the /.singularity.d/env directory. 10-docker.sh seems to have permissions set to 755 but all the other scripts have their permissions set to 700 resulting in the following error when trying to access the container as a non-root user:

/.singularity.d/actions/exec: .: line 7: can't open '/.singularity.d/env/01-base.sh'

I'm testing this using the alpine:3.7 docker image as follows (as root):

$ docker run -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/test:/output --privileged -t --rm docker2singularity -n testimage alpine:3.7

The testimage.simg image file is created in /tmp/test and I can successfully run it using:

$ singularity exec /tmp/test/testimage.simg /bin/sh

If I switch back to a non root user, it fails to run giving the error shown above. I am also able to successfully run the docker command to create the image as a non-root user that is a member of the docker group but the permissions of the files in /.singularity.d/env still prevent the image from being run while not root.

I'm running singularity version 3.3.0-1.

Apologies if I'm missing something here, or if this is intended behaviour, I couldn't see any information about this in the docs or previous issues. Thanks.

Hi Chris,

it would be great if one could have an option/flag to convert a Docker image into a Singularity directory in addition to the flat image file (which might make an automatized roll out to CVMFS easier)

Thanks,
Thomas

We are behind on releases here! After #30 goes through, I'll do a pull request for the most up to date version. Given security risks and quick change of the library, this is important for users.

Jays-MacBook-Pro-2:bin bj2348$ docker version
Client: Docker Engine - Community
Version: 19.03.1
API version: 1.40
Go version: go1.12.5
Git commit: 74b1e89
Built: Thu Jul 25 21:18:17 2019
OS/Arch: darwin/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.1
API version: 1.40 (minimum version 1.12)
Go version: go1.12.5
Git commit: 74b1e89
Built: Thu Jul 25 21:17:52 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.2.6
GitCommit: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc:
Version: 1.0.0-rc8
GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f
docker-init:
Version: 0.18.0
GitCommit: fec3683
Jays-MacBook-Pro-2:bin bj2348$ singularity version
3.2.0-rc2

Jays-MacBook-Pro-2:bin bj2348$ docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
basilisk latest 17b2070a17f4 13 days ago 1.08GB
centos6_cfd latest 64a9cf2d63ba 2 weeks ago 777MB
ubuntu latest 3556258649b2 3 weeks ago 64.2MB
centos 6 d0957ffdf8a2 5 months ago 194MB
centos latest 9f38484d220f 5 months ago 202MB
Jays-MacBook-Pro-2:bin bj2348$ sudo ./docker2singularity.sh basilisk
Password:

Image Format: squashfs
Docker Image: basilisk

Inspected Size: 1082 MB

(1/10) Creating a build sandbox...
(2/10) Exporting filesystem...
(3/10) Creating labels...
./docker2singularity.sh: line 224: /addLabel.py: No such file or directory

First of all nice tool, thanks!

The problem is that current version of docker2singularity.sh fails with flags -f|--folder and -w|--writable because two "shift" commands are invocated (one "shift" is enough because such flags have no arguments).

So please:

• remove lines 84 and 89
• correct the typo in line 86 (change the misspelled "-wrtiable" with "--writable")

Best.

This does not appear to make valid singularity images for all purposes in 2.6.
The resulting images kinda work until you do something like try to use the --nv flag and then get the following:

awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
awk: warning: escape sequence `\.' treated as plain `.'
WARNING: Library bind directory not present in container, update container

make smaller.

https://github.com/singularityware/docker2singularity/blob/master/docker2singularity.sh#L255

This is not a valid way to transform a CMD or ENTRYPOINT. From the Dockerfile documentation:

RUN has 2 forms:
RUN <command> (shell form, the command is run in a shell, which by default is /bin/sh -c on Linux or cmd /S /C on Windows).
RUN ["executable", "param1", "param2"] (exec form).

Using the following command:
docker run quay.io/singularity/docker2singularity -f <<container_name>>
I received the following output:
docker: error during connect: Post http://docker:2375/v1.39/containers/create: dial tcp: lookup docker on 192.168.65.1:53: no such host.

When I try to run the example on the readme, I run into the following error:

 docker run \
 -v /var/run/docker.sock:/var/run/docker.sock \
 -v ~/Desktop:/output \
 --privileged -t --rm \
 singularityware/docker2singularity \
 -m "/shared_fs /custom_mountpoint2" \
 ubuntu:14.04

unknown shorthand flag: 'f' in -f
See 'docker run --help'.

Could this be related to an issue with the handling of -m mount points in lines 23-41 of https://github.com/singularityware/docker2singularity/blob/master/docker2singularity.sh#L23 which then causes the image definition to be off and then cause line 49 to break?

runningid=docker run -d $image tail -f /dev/null

I'm running Server Version 1.13.1

I'm using Docker version 17.12.1-ce, build 7390fc6 on an Ubuntu 18.04 system. When I try to use -m with multiple directories, docker2singularity doesn't deal with the space in the -m string correctly. Note that it creates a directory "foo ".

It looks like the the "adding mount points" part of docker2singularity.sh changed at some point.

paciorek@shelob:~/staff/projects/docker-test/mpi> docker run -v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/output --privileged -t --rm    singularityware/docker2singularity -m "/foo /bar" ubuntu:18.04
Image Format: squashfs
Docker Image: ubuntu:18.04

Inspected Size: 85 MB

(1/10) Creating a build sandbox...
(2/10) Exporting filesystem...
(3/10) Creating labels...
(4/10) Adding run script...
(5/10) Setting ENV variables...
(6/10) Adding mount points...
(7/10) Fixing permissions...
(8/10) Stopping and removing the container...
(9/10) Building squashfs container...
Building image from sandbox: /tmp/ubuntu_18.04-2018-09-05-5ecea5d81788.build
Building Singularity image...
Singularity container built: /tmp/ubuntu_18.04-2018-09-05-5ecea5d81788.simg
Cleaning up...
(10/10) Moving the image to the output folder...
     27,947,039 100%  369.74MB/s    0:00:00 (xfr#1, to-chk=0/1)
Final Size: 27MB
paciorek@shelob:~/staff/projects/docker-test/mpi> singularity shell /tmp/ubuntu_18.04-2018-09-05-5ecea5d81788.simg
Singularity: Invoking an interactive shell within container...

Singularity ubuntu_18.04-2018-09-05-5ecea5d81788.simg:~/staff/projects/docker-test/mpi> ls /
 accounts   boot   environment	'foo '	 lib	 media	 opt    root   sbin	     singularity.json   sys   usr
 bin	    dev    etc		 home	 lib64	 mnt	 proc   run    singularity   srv	        tmp   var

If I simply start a Docker container using the docker2singularity image, and try to run the mkdir -p line from docker2singularity.sh, I essentially recreate the issue; 'foo bar' is a single directory here:

paciorek@shelob:~/staff/projects/docker-test/mpi> docker run -it --rm --entrypoint "/bin/bash" singularityware/docker2singularity
bash-4.3# mount_points="foo bar"
bash-4.3# new_container_name=/tmp/blah.id
bash-4.3# build_sandbox="${new_container_name}.build"
bash-4.3# echo ${build_sandbox}
/tmp/blah.id.build
bash-4.3# mkdir -p "${build_sandbox}/${mount_points}"
bash-4.3# ls /tmp/blah.id.build/
foo bar

Would love to see documentation for how to configure a github action to build a singularity image from a dockerfile and upload it to the singularity hub.

I execute the examples and receive an error from the shell script:

/docker2singularity.sh: line 110: $1: unbound variable

What is the problem?

Thanks

Hi friend, what is your question?

I installed the singularity using Conda, and use this tools to convert my docker image:

docker run \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /home/data_backup1/homelette/singularity_image:/output \
--privileged -t --rm \
quay.io/singularity/docker2singularity  \
homelette

But I got the error:

Image Format: squashfs
Docker Image: homelette

Template parsing error: template: :1:2: executing "" at <.Size>: map has no entry for key "Size"

How could I fix this problem?

I have a problem trying to run any Singularity image created with docker2singularity.
As an example, when converting the following image:

  docker run -v /var/run/docker.sock:/var/run/docker.sock \
            -v / tmp / test: / output \
            --privileged -t --rm \
            quay.io/singularity/docker2singularity \
            ubuntu: 14.04

The image is created correctly, but when executing:

singularity shell /tmp/test/ubuntu_14.04-2020-09-16-119fd5e2c6ae.sif

I get the following error:

ERROR  : Unknown image format/type: /tmp/test/ubuntu_14.04-2020-09-16-119fd5e2c6ae.sif
ABORT  : Retval = 255

The same happens when using the commands: exec, inspect and run.

When trying different alternatives, I found that converting the image with the -f option (as folder) does not produce an error. However, this is not practical, because a new conversion to the ".sif" format is required and for images of several GB it takes a long time.

https://github.com/sylabs/singularity/releases/tag/v3.2.1

This is a small issue, but we are moving to https://quay.io/repository/singularity/docker2singularity, so I want to update the recipe name to be instead of Deploy to Docker Hub, to "Deploy to Quay.io." Sorry Docker Hub, I can't do this lack of token generation with specific permissions anymore! :/

@chrisfilo I used this container to create a singularity image from an mriqc docker container, but I am running into some problems when running it on our cluster.

I changed the mriqc Dockerfile to create the mount points to our HPC cluster, as recommended by our HPC facility:
RUN mkdir /gpfs /spin1 /gs2 /gs3 /gs4 /gs5 /data /scratch /fdb /lscratch

I also removed the command in mriqc's Dockerfile to set the WORKDIR to /scratch since in our HPC we don't have read/write access to all of /scratch, just to a subdir of /scratch with our username.

When I try to run the image, mriqc executes properly, but after it is done executing, it attempts to rewrite the permissions from every subfolder in /scratch. Could this be a side effect of the way permissions are set within the singularity image when converting the Docker container? I tried creating an image last week and did not have this problem, but I might have been using an older version of docker2singularity I found referenced in a google discussion group.