Hello, I'm trying to create a writable image: sudo d

hey <a class="user-mention notranslate" data-hovercard-type="organization" data-hoverc

Thank you for the super quick attention here, <a class="user-mention notranslate" data

<a class="user-mention notranslate" data-hovercard-type="organization" data-hovercard-

Error When with -w or -f Options at line 110 $1: unbound variable,about singularityhub/docker2singularity

Comments (14)

vsoch commented on June 15, 2024 1

So sorry about that! I put in an exit 0 for testing the options argument, and just forgot to take it out!

Just removed it - you should be able to do the same above and have it test fully.

from docker2singularity.

vsoch commented on June 15, 2024

I think there is just an extra shift - let me put in a quick PR for you to test (you'll need to build the image first) - back in a bit!

from docker2singularity.

vsoch commented on June 15, 2024

hey @ppdac you should be able to pull this branch:

#75

and then build the container locally to test. E.g.,

git clone -b fix/extra-shifts https://www.github.com/singularityhub/docker2singularity
cd docker2singularity
docker build -t docker2singularity .

And then use docker2singularity to test. If/when that works let me know and we can merge the PR and update the containers. Also note that updated containers are being served from quay.io:

https://quay.io/repository/singularity/docker2singularity

It's late here and I'm done for the day, but I can finish up after you've tested tomorrow. Thanks for catching this!

from docker2singularity.

soyfrien commented on June 15, 2024

Thank you for the super quick attention here, @vsoch! 🥇

Hit the below issue with the -w flag, but -f was a huge success! Forward progress on my end, so you've helped me clear a hurdle here :)

louis@server:~/source/docker2singularity$ sudo docker image list
REPOSITORY                               TAG                 IMAGE ID            CREATED             SIZE
docker2singularity                       latest              3bfffb899de4        26 seconds ago      350MB
<none>                                   <none>              bacc1f0a5ea5        37 seconds ago      1.14GB
quay.io/singularity/docker2singularity   latest              606348dfb722        9 days ago          350MB
docker                                   18.09.8             e48dc77a28ea        6 weeks ago         171MB
singularityware/docker2singularity       latest              96a6f87eed76        6 weeks ago         339MB
golang                                   1.12.7-alpine3.10   6b21b4c6e7a3        2 months ago        350MB
nipy/mindboggle                          latest              9c29be1f7297        3 months ago        4.55GB
fcpindi/c-pac                            latest              20f00f383f26        4 months ago        3.81GB
fcpindi/c-pac                            <none>              af15c31b0b39        6 months ago        3.85GB
childmind/c-pac                          latest              9d2e274369b7        6 months ago        5.53MB
centos                                   latest              1e1148e4cc2c        10 months ago       202MB
childmind/c-pac                          <none>              482d3d5f1def        10 months ago       3.97GB
childmid/c-pac                           oldest              abd442b86718        11 months ago       4.64GB
childmind/c-pac                          <none>              abd442b86718        11 months ago       4.64GB
poldracklab/fmriprep                     1.1.8               ac3a493f03a7        12 months ago       11.5GB
nipy/mindboggle                          <none>              3f24f84333cf        12 months ago       5.19GB
hello-world                              latest              4ab4c602aa5e        13 months ago       1.84kB
filo/docker2singularity                  latest              54abb69c7312        19 months ago       102MB
louis@server:~/source/docker2singularity$ sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v /opt/SingularityImages/test:/output --privileged -t --rm docker2singularity -w fcpindi/c-pac:latest

Image Format: writable
Docker Image: fcpindi/c-pac:latest

Inspected Size: 3810 MB

(1/10) Creating a build sandbox...
(2/10) Exporting filesystem...
(3/10) Creating labels...
(4/10) Adding run script...
(5/10) Setting ENV variables...
(6/10) Adding mount points...
(7/10) Fixing permissions...
(8/10) Stopping and removing the container...
(9/10) Building writable container...
Error for command "build": unknown flag: --writable

Options for build command:

      --arch string       Architecture for remote build (default "amd64")
      --builder string    remote Build Service URL, setting this implies --remote (default
                          "https://build.sylabs.io")
  -d, --detached          submit build job and print build ID (no real-time logs and requires --remote)
      --disable-cache     do not use cache or create cache
      --docker-login      login to a Docker Repository interactively
  -e, --encrypt           build an image with an encrypted file system
  -f, --fakeroot          build using user namespace to fake root user (requires a privileged installation)
  -F, --force             delete and overwrite an image if it currently exists
  -h, --help              help for build
      --json              interpret build definition as JSON
      --library string    container Library URL (default "https://library.sylabs.io")
      --no-cleanup        do NOT clean up bundle after failed build, can be helpul for debugging
      --nohttps           do NOT use HTTPS, for communicating with local docker registry
  -T, --notest            build without running tests in %test section
      --passphrase        Enter a passphrase for an encrypted contaner
      --pem-path string   Enter an path to a PEM formated RSA key for an encrypted container
  -r, --remote            build image remotely (does not require root)
  -s, --sandbox           build image as sandbox format (chroot directory structure)
      --section strings   only run specific section(s) of deffile (setup, post, files, environment, test,
                          labels, none) (default [all])
      --tmpdir string     specify a temporary directory to use for build
  -u, --update            run definition over existing container (skips header)

Run 'singularity build --help' for more detailed usage information.

Using Singularity version: 2.4.2-dist. I'm not sure if this is due to something on my setup, but if not feel free to have me test more.

Thanks again!

from docker2singularity.

vsoch commented on June 15, 2024

Ah, so that's not an issue with the client here, but the version of singularity you are using (that doesn't have writable as an option to build). Your options are to use a newer version of singularity, or go back to use one of the older docker2singularity (the tags correspond with version). If this is the only issue, then the PR here should be good - let me know when you are comfortable with merging, and if/when you test older versions of the client, if you have any success (not that the branches also correspond to versions if you want to see the code).

I'll check up on this issue/thread tomorrow! It's like 7pm and I haven't even started making dinner yet :/

from docker2singularity.

soyfrien commented on June 15, 2024

I'll upgrade Singularity and let you know how it goes. Enjoy your dinner, and dessert. You deserve it!

from docker2singularity.

vsoch commented on June 15, 2024

Thanks!! 🍽️

One quick thought that just came to mind (I'm back, haha) is that if you figure out that your version needs a different flag (e.g., --sandbox instead of --writable) we could add an argument for custom options, that way any supported option (for any version of singularity) could be provided to the build. So if :

updating doesn't work
using an older version of docker2singularity doesn't work

Let's nail down the command you need to build, and then add the --options argument to handle it.
Okay - now I'm off!

from docker2singularity.

soyfrien commented on June 15, 2024

Firstly, hope you had a yummy meal! My dog ate my take out as I stepped out of the room for the second day in a row. :(

updating doesn't work

I'm not quite sure. I think --writable appeared in v2.5, but I updated directly to v3.2.1 to match what our HPC is running. Looking at the docs for versions older than v2.5 I don't see it mentioned (v2.4 docs are missing, which is the version I opened this ticket on, and those docs are also not on the Wayback Machine)

I'm guessing the --writable flag was dropped in v3: we can work with sandboxes and convert them to SIF images using singularity build new-sif. So perhaps the functionality was deemed redundant?

using an older version of docker2singularity doesn't work

I should have tried this first, well really I should have upgraded to the final v2 release and tried your patch first. But the tags/v2.3 failed for the same reasons, just on a different line. I think this is to be expected now that I am of the viewpoint that build --writable was short-lived on versions of Singularity that I didn't get to test.

I'm reluctant to mess around with the system any further, as I'm getting sleepy and don't want to start making errors; and also I think you solved the problem by fixing the -f sandbox flag.

I think support for -w can be dropped, or only kept around if a version check of Singularity for 2.5–6 is added.

Having -f's sandbox support complements the current documentation for Singularity.

I was just messing around, so don't look at this seriously, but interestingly, using build new-sif allows singularity shell --writable to work on .simgs. I'm new to Singularity, so I can't comment much on what I mean to say here other than, build new-sif seems to create mutable immutable images. Heheh...

#docker2singularity.sh 
elif [ "$image_format" == "writable" ]; then
    new_container_name=${new_container_name}.img
    singularity build new-sif $build_sandbox
    mv $build_sandbox $new_container_name

from docker2singularity.

vsoch commented on June 15, 2024

For newest Singularity the supported option is "Sandbox" instead of writable. I think what I should add to the current PR is support for a general option, that way you can add whatever you like. Note that this will just be for the current master (which builds 3.4.1).

from docker2singularity.

vsoch commented on June 15, 2024

@ppdac I've added an options argument, so you should be able to provide any options that you like to the build. Would you care to test it out? When you approve the PR, we can merge.

from docker2singularity.

soyfrien commented on June 15, 2024

Sorry for the late reply. I wasn't at work today, but work was sure coming at me.

I tried rebuilding docker2singularity again, but it doesn't seem to work anymore:

(base) louis@server:~/source$ git clone -b fix/extra-shifts https://www.github.com/singularityhub/docker2singularity
Cloning into 'docker2singularity'...
warning: redirecting to https://github.com/singularityhub/docker2singularity.git/
remote: Enumerating objects: 15, done.
remote: Counting objects: 100% (15/15), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 495 (delta 4), reused 11 (delta 3), pack-reused 480
Receiving objects: 100% (495/495), 210.43 KiB | 11.69 MiB/s, done.
Resolving deltas: 100% (250/250), done.
(base) louis@server:~/source$ cd docker2singularity
(base) louis@server:~/source/docker2singularity$ sudo docker build -t docker2singularity .
Sending build context to Docker daemon  424.4kB
Step 1/21 : FROM golang:1.12.7-alpine3.10 as base
 ---> 6b21b4c6e7a3
Step 2/21 : FROM docker:18.09.8 as builder
 ---> e48dc77a28ea
Step 3/21 : COPY --from=base /go /go
 ---> Using cache
 ---> cd78dc6d5580
Step 4/21 : COPY --from=base /usr/local/go /usr/local/go
 ---> Using cache
 ---> b35f4163f405
Step 5/21 : ENV GOPATH /go
 ---> Using cache
 ---> 1448046e9911
Step 6/21 : ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH
 ---> Using cache
 ---> 6040b6a8b3dc
Step 7/21 : ENV GOLANG_VERSION 1.12.7
 ---> Using cache
 ---> 14305b7d60db
Step 8/21 : RUN apk update &&     apk add --virtual automake build-base linux-headers libffi-dev
 ---> Using cache
 ---> 2755e0bb8723
Step 9/21 : RUN apk add --no-cache bash git openssh gcc squashfs-tools sudo libtool gawk ca-certificates libseccomp
 ---> Using cache
 ---> a34c3ee7993a
Step 10/21 : RUN apk add --no-cache linux-headers build-base openssl-dev util-linux util-linux-dev python rsync cryptsetup
 ---> Using cache
 ---> 818e63fdb112
Step 11/21 : RUN mkdir -p /usr/local/var/singularity/mnt &&     mkdir -p $GOPATH/src/github.com/sylabs &&     cd $GOPATH/src/github.com/sylabs &&     wget -qO- https://github.com/sylabs/singularity/releases/download/v3.4.1/singularity-3.4.1.tar.gz |     tar xzv &&     cd singularity &&     ./mconfig -p /usr/local/singularity &&     make -C builddir &&     make -C builddir install
 ---> Using cache
 ---> bacc1f0a5ea5
Step 12/21 : FROM docker:18.09.8
 ---> e48dc77a28ea
Step 13/21 : LABEL Maintainer vsochat@...
 ---> Using cache
 ---> 789a0b6300b8
Step 14/21 : COPY --from=builder /usr/local/singularity /usr/local/singularity
 ---> Using cache
 ---> 596473048912
Step 15/21 : RUN apk add --no-cache ca-certificates libseccomp squashfs-tools bash python rsync
 ---> Using cache
 ---> 186b075a9db2
Step 16/21 : ENV PATH="/usr/local/singularity/bin:$PATH"
 ---> Using cache
 ---> bd6e36062539
Step 17/21 : ADD docker2singularity.sh /docker2singularity.sh
 ---> f38d1671f5d8
Step 18/21 : ADD addLabel.py /addLabel.py
 ---> bb4951a08105
Step 19/21 : ADD scripts /scripts
 ---> 91ad1e5c09c3
Step 20/21 : RUN chmod a+x docker2singularity.sh
 ---> Running in 1f3b5d6a0341
Removing intermediate container 1f3b5d6a0341
 ---> f9c23b3e431f
Step 21/21 : ENTRYPOINT ["docker-entrypoint.sh", "/docker2singularity.sh"]
 ---> Running in 3e4adcf0b850
Removing intermediate container 3e4adcf0b850
 ---> 18fd6d59168b
Successfully built 18fd6d59168b
Successfully tagged docker2singularity:latest

(base) louis@server:~/source/docker2singularity$ sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v /opt/SingularityImages/test:/output --privileged -t --rm docker2singularity -w fcpindi/c-pac
(base) louis@server:~/source/docker2singularity$ sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v /opt/SingularityImages/test:/output --privileged -t --rm docker2singularity --writable fcpindi/c-pac
(base) louis@server:~/source/docker2singularity$ sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v /opt/SingularityImages/test:/output --privileged -t --rm docker2singularity -f fcpindi/c-pac
(base) louis@server:~/source/docker2singularity$ sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v /opt/SingularityImages/test:/output --privileged -t --rm docker2singularity -f fcpindi/c-pac:latest
(base) louis@server:~/source/docker2singularity$ sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v /opt/SingularityImages/test:/output --privileged -t --rm docker2singularity fcpindi/c-pac:latest

(base) louis@server:~/source/docker2singularity$ singularity --version
singularity version 3.4.1

Hope all is well,
Louis

from docker2singularity.

soyfrien commented on June 15, 2024

Okay, so -w fails as expected in Singularity 3.4.1, but -f works, which is great and an improvement.

$ sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v /opt/SingularityImages/test:/output --privileged -t --rm docker2singularity -w fcpindi/c-pac

Image Format: writable
Docker Image: fcpindi/c-pac

Inspected Size: 3810 MB

(1/10) Creating a build sandbox...
(2/10) Exporting filesystem...
(3/10) Creating labels...
(4/10) Adding run script...
(5/10) Setting ENV variables...
(6/10) Adding mount points...
(7/10) Fixing permissions...
(8/10) Stopping and removing the container...
(9/10) Building writable container...
Error for command "build": unknown flag: --writable

Options for build command:

      --arch string       Architecture for remote build (default "amd64")
      --builder string    remote Build Service URL, setting this implies --remote (default "https://build.sylabs.io")
  -d, --detached          submit build job and print build ID (no real-time logs and requires --remote)
      --disable-cache     do not use cache or create cache
      --docker-login      login to a Docker Repository interactively
  -e, --encrypt           build an image with an encrypted file system
  -f, --fakeroot          build using user namespace to fake root user (requires a privileged installation)
  -F, --force             delete and overwrite an image if it currently exists
  -h, --help              help for build
      --json              interpret build definition as JSON
      --library string    container Library URL (default "https://library.sylabs.io")
      --no-cleanup        do NOT clean up bundle after failed build, can be helpul for debugging
      --nohttps           do NOT use HTTPS, for communicating with local docker registry
  -T, --notest            build without running tests in %test section
      --passphrase        Enter a passphrase for an encrypted contaner
      --pem-path string   Enter an path to a PEM formated RSA key for an encrypted container
  -r, --remote            build image remotely (does not require root)
  -s, --sandbox           build image as sandbox format (chroot directory structure)
      --section strings   only run specific section(s) of deffile (setup, post, files, environment, test, labels, none) (default [all])
      --tmpdir string     specify a temporary directory to use for build
  -u, --update            run definition over existing container (skips header)


$ sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v /opt/SingularityImages/test:/output --privileged -t --rm docker2singularity -f fcpindi/c-pac

Image Format: sandbox
Docker Image: fcpindi/c-pac

Inspected Size: 3810 MB

(1/10) Creating a build sandbox...
(2/10) Exporting filesystem...
(3/10) Creating labels...
(4/10) Adding run script...
(5/10) Setting ENV variables...
(6/10) Adding mount points...
(7/10) Fixing permissions...
(8/10) Stopping and removing the container...
(9/10) Building sandbox container...
(10/10) Moving the image to the output folder...
  4,622,208,358  99%  203.20MB/s    0:00:21 (xfr#107299, to-chk=0/129506)   
Final Size: 3765MB

I created a VM to build Singularity from the 2.6.1 tag, and surprisingly the writable flag isn't there:

I then downgraded to 2.5.2-HEAD.b258b6513 because that was around when the old docs mention build --writable, and tried docker2singularity with -w again:

Given this, I would recommend dropping support for the flag altogether. Or maybe warning users to use -f instead and work with sandboxes.

Sandboxes are easy to later convert to SIF images (singularity build new-sif sandboxName).

Thanks for all of your help with this! Thanks to you, I had fun learning to use git and GitHub in new ways 👍

from docker2singularity.

vsoch commented on June 15, 2024

Awesome! So - ready for merge?

from docker2singularity.

soyfrien commented on June 15, 2024

Yes! We are go for launch!

from docker2singularity.

Error When with -w or -f Options at line 110 $1: unbound variable about docker2singularity HOT 14 CLOSED

Comments (14)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent