gdraheim / docker-copyedit Goto Github PK

edit docker image metadata (including remove docker volumes)

License: European Union Public License 1.2

Python 97.36% Makefile 2.50% Dockerfile 0.14%

docker-copyedit's Introduction

edit docker image metadata

The initial motiviation for the creation of the tool came from the fact that it is not possible to remove VOLUME entries in an image. You can basically change a USER or WORKDIR setting but you can only ever add VOLUME and PORT entries.

The wish to REMOVE ALL VOLUMES came from the fact that I did want to download a tested image for local tests where the data part should be committed to the history as well in order to turn back both program and data to a defined state so that another test run will start off the exact same checkpoint.

While docker does not allow to edit the metadata of an image directly, there is a workaround - one may "docker save" an image into an archive file that contains all the layers and metadata json files. After modifying the content one can "docker load" the result back with the history being preserved.

Correcting some images from other sources became such a regular task that I started to fill in a python script to help with the daily work. In order to allow coworkers to understand what was intended, the input syntax is somewhat descriptive (likeSQL).

 ./docker-copyedit.py \
 FROM image1 INTO image2 REMOVE ALL VOLUMES
     
 ./docker-copyedit.py FROM image1 INTO image2 -vv \
     add volume /var/tmp
 ./docker-copyedit.py FROM image1 INTO image2 -vv \
     REMOVE VOLUME /var/atlassian/jira-data
 ./docker-copyedit.py FROM image1 INTO image2 -vv \
     REMOVE VOLUMES '/var/*' AND RM PORTS 80%0
 
 ./docker-copyedit.py \
     into image2 from image1 set no user
 ./docker-copyedit.py \
     set null user and set null cmd from image1 into image2
 ./docker-copyedit.py FROM image1 INTO image2 \
     set null user + set null cmd + rm all volumes

 ./docker-copyedit.py FROM image1 INTO image2 -vv \
     set null entrypoint and set cmd /entrypoint.sh
 ./docker-copyedit.py FROM image1 INTO image2 -vv \
     set shell cmd "/entrypoint.sh foo"
 ./docker-copyedit.py FROM image1 INTO image2 -vv \
     set label author "real me" and rm labels old%
 ./docker-copyedit.py FROM image1 INTO image2 -vv \
     set env MAINDIR "/path" and rm env backupdir

 ./docker-copyedit.py FROM image1 INTO image2 -vv \
     REMOVE PORT 4444
 ./docker-copyedit.py FROM image1 INTO image2 -vv \
     remove port ldap and rm port ldaps
 ./docker-copyedit.py FROM image1 INTO image2 -vv \
     remove all ports
 ./docker-copyedit.py FROM image1 INTO image2 -vv \
     add port ldap and add port ldaps

Of course you may have image1 and image2 to be the same tag name but remember that the image hash value will change while copyediting the image archive on the disk. You will be left with a dangling old (untagged) image.

Other than 'entrypoint','cmd' and 'user' you can also set the string values for 'workdir'/'workingdir', 'domainname', 'hostname', 'arch'/'architecture' and 'author' in configs. The values in the env list and label list can be modified too. Healthcheck can be removed. If the edit command did not really change something then the edited image is not loaded back from disk. Instead the old image is possibly just tagged with the new name.

For podman it is not possible to check service user examples and healthcheck examples as it seems to be not supported.

By default the tool will use a local "load.tmp" temporary directory. You may set "-T $TMPDIR" explicitly to have it run in a normal temporary directory - but be aware that the archive files during save/load can be quite big and the tool will even unpack the archives temporarily. That's why the "-T tmpdir" should point to a space that is hopefully big enough (like the build server workspace you are already in).

... I take patches! ... (however please run the docker-copyedit-tests.py / make check before)

docker-copyedit's People

Stargazers

Watchers

docker-copyedit's Issues

dockerize you application

It would be nice if you dockerized your application.
We could use it in ci/cd processes.

can't set architecture variant

docker-copyedit doesn't allow us to change the architecture "variant" metadata.

For example,

$ docker inspect arm32v7/ubuntu | jq -r .[].Variant
v7

$ ./docker-copyedit.py FROM arm32v7/ubuntu into foo set variant foo
ERROR:edit:unknown edit command starting with set variant

STDERR: invalid tag

When trying to remove a certain port of an image, I only receive this error in the last step of the process:

DEBUG:edit:changed 1 layer metadata
ERROR:edit:CMD docker load -i load.tmp/ready.tar
ERROR:edit:EXIT 1
ERROR:edit:STDOUT
ERROR:edit:STDERR invalid tag "jannikz/gluster-prod"

Traceback (most recent call last):
  File "./docker-copyedit.py", line 706, in <module>
    edit_image(inp, out, commands)
  File "./docker-copyedit.py", line 252, in edit_image
    sh(cmd.format(**locals()))
  File "./docker-copyedit.py", line 43, in sh
    raise Exception("shell command failed")
Exception: shell command failed

No matter what tag I'm entering .. did you ever face a problem like this?

./docker-copyedit.py \
FROM jannikz/gluster-prod:latest INTO jannikz/gluster-prod -vvv REMOVE PORT 111
DEBUG:edit:image parsing = jannikz/gluster-prod:latest
DEBUG:edit:.registry = None
DEBUG:edit:.image = jannikz/gluster-prod
DEBUG:edit:.tag = :latest
DEBUG:edit:image parsing = jannikz/gluster-prod
DEBUG:edit:.registry = None
DEBUG:edit:.image = jannikz/gluster-prod
DEBUG:edit:.tag = None
DEBUG:edit:total 1484224

__version__ = "1.2.1402"
Docker version 18.06.1-ce, build e68fc7a

Python 3 support?

Alpine has recently got rid of Python2, which is needed for this (awesome) script.

While this is workaround-able for now I was wondering if there was ever any plans to support Python 3?

set-label causes TypeError on images with no existing labels

Running docker-copyedit with a 'set-label name value' leads to an error if the image has no labels. It works fine when an image already has some label(s). The Python stack trace appears thus:

$ docker-copyedit.py from postgres:13.5 into b:latest set-label somename somevalue
  File "docker-copyedit.py", line 938, in <module>
    edit_image(inp, out, commands)
  File "docker-copyedit.py", line 310, in edit_image
    changed = edit_datadir(datadir, out_tag, edits)
  File "docker-copyedit.py", line 539, in edit_datadir
    if target in config[CONFIG][key]:
TypeError: argument of type 'NoneType' is not iterable

An example of an image with no existing labels is postgres:13.5. Here is the 'docker inspect' of this image:

[
    {
        "Id": "sha256:e01c76bb1351f3fb966f97cf108f1ff586056a517d56c9ec2629ede6be756691",
        "RepoTags": [
            "postgres:13.5"
        ],
        "RepoDigests": [
            "postgres@sha256:2e2e213c0fe4d1229373bebf114f255f68d86fba9e9f30e9bcc08e92cdf7b5b4"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2022-01-27T01:11:05.913589796Z",
        "Container": "32bd06472b5cfa54b87f8772fe083d5bab2b1144868ac6ce408b757f54bb982a",
        "ContainerConfig": {
            "Hostname": "32bd06472b5c",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "5432/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/13/bin",
                "GOSU_VERSION=1.14",
                "LANG=en_US.utf8",
                "PG_MAJOR=13",
                "PG_VERSION=13.5-1.pgdg110+1",
                "PGDATA=/var/lib/postgresql/data"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "CMD [\"postgres\"]"
            ],
            "Image": "sha256:8004cdfe9dee6a2263d8cc24d9f67d6c41dd0fc090c1be01b98418d9d2497bce",
            "Volumes": {
                "/var/lib/postgresql/data": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {},
            "StopSignal": "SIGINT"
        },
        "DockerVersion": "20.10.7",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "5432/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/13/bin",
                "GOSU_VERSION=1.14",
                "LANG=en_US.utf8",
                "PG_MAJOR=13",
                "PG_VERSION=13.5-1.pgdg110+1",
                "PGDATA=/var/lib/postgresql/data"
            ],
            "Cmd": [
                "postgres"
            ],
            "Image": "sha256:8004cdfe9dee6a2263d8cc24d9f67d6c41dd0fc090c1be01b98418d9d2497bce",
            "Volumes": {
                "/var/lib/postgresql/data": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": null,
            "StopSignal": "SIGINT"
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 371198583,
        "VirtualSize": 371198583,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/4b5c59b2df623b68662dff9dd759e8b1e5e15ba1065d4cd72737e709d44e38c8/diff:/var/lib/docker/overlay2/79cdf68ee1096872e5a416d505793aa391f2a1ee0b0d25bc406f84954bb3ac94/diff:/var/lib/docker/overlay2/445a42c7637f1b00fd12a9249f55171bdd31ffcbb9de169467030f9cb6073001/diff:/var/lib/docker/overlay2/a1fcb9929fb699e647fc716627cc4bfd64413af74b9af3442c9c8d08784d1bd0/diff:/var/lib/docker/overlay2/d140bf2c2f284959d56fec2afb7b3ef2acfa0ed4ce35188208d775e2212b66c1/diff:/var/lib/docker/overlay2/754602e81673683379f5ca51ac26d3a110ade09ded64671d611e9f77b7468198/diff:/var/lib/docker/overlay2/bd0b0f158d1bae5922aadbf7ce49123e5ea0db06f4d09f36a9bed5a27514d7e1/diff:/var/lib/docker/overlay2/186b4af03de1f2231bfc00d2a5026d5e0ea2741f2f7f3b86726dcbfbe4aac4e8/diff:/var/lib/docker/overlay2/1a582292220a3e41a400d932828548029d48416c4d3b1673a2b9fe253978c515/diff:/var/lib/docker/overlay2/b37a11f200770cec046366d4a5edf2d921c3873234f3f540dd2e94d5a0886fb9/diff:/var/lib/docker/overlay2/dbf7b34785d5b0ac0936370bb6d2a00a886c083e745df51a6b185efe68b9bc75/diff:/var/lib/docker/overlay2/39cf89b7f028e5f5aab843dc56def8f5817dc182f416ad4d59df4a09865e134f/diff",
                "MergedDir": "/var/lib/docker/overlay2/a23616e078d0e485b807e0affe1b0e35dc315a1b80ad4439572a02b9dab8dd6c/merged",
                "UpperDir": "/var/lib/docker/overlay2/a23616e078d0e485b807e0affe1b0e35dc315a1b80ad4439572a02b9dab8dd6c/diff",
                "WorkDir": "/var/lib/docker/overlay2/a23616e078d0e485b807e0affe1b0e35dc315a1b80ad4439572a02b9dab8dd6c/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:7d0ebbe3f5d26c1b5ec4d5dbb6fe3205d7061f9735080b0162d550530328abd6",
                "sha256:504d3c78f1942134301a91d83b6d2102b2c398eab2c81a76f62b9eb33f8c000b",
                "sha256:8c1f84338867a554126fe7100341ff8174fe71e13518389803e4e1050aa8f200",
                "sha256:10e60ad30e5f437ff02dd7273545deeeabb0d07595dae73cc0f1ea2d38a2956f",
                "sha256:96d824739a641ee2e8f36bcd8ff576c7ba72029184aaa7c8ec5c418f5c60802f",
                "sha256:a4b9517dfa8c2c05e3cd3423f52535a7e551ac7e4c026541742c054b19b1c5e4",
                "sha256:f2e4fedb957df6c5380f46b9b08e2554b1936a553acbc0c77579faf53724feda",
                "sha256:c1cf0fe2cfa15adc5603ebd8d0047d12afe9b82a634784c8b3967be55ffc33f2",
                "sha256:b7035179c078b0703b5d5f322123535179489aa8b0eb820e317862a6448ff240",
                "sha256:6fc0879b8f3a07b74f6de0a76c22175ec89d0dcc46827b3f433b736e977a66b7",
                "sha256:954be07f25afe8092888629a0603fc6fc773c093bd5b8d9f20e6bfdeff1b8a06",
                "sha256:ae0a327d2532a9199bb5d3f7a198315f0ea092eb9c9a7719a0bec76b22da307e",
                "sha256:d69fa3581c263ec07ff28eb3edd558c4fb20e4bbad8a039eec0c005cc3152819"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

Perhaps the $.Config.Labels:null is the culprit.

remove all volumes doesn't work

Hi,

Probably, I did something wrong, but it was from README:
in readme:

./docker-copyedit.py \
 FROM image1 INTO image2 REMOVE ALL VOLUMES

but it doesn't work:

[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$ ./docker-copyedit.py FROM mysql:5.6 INTO mysql-novol:5.6 REMOVE ALL VOLUMES
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$ ./docker-copyedit.py FROM mysql:5.6 INTO mysql-novol REMOVE ALL VOLUMES
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$ ./docker-copyedit.py FROM mysql INTO mysql-novol REMOVE ALL VOLUMES
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
ERROR:edit:CMD docker tag mysql mysql-novol
ERROR:edit:EXIT 1
ERROR:edit:STDOUT
ERROR:edit:STDERR Error response from daemon: No such image: mysql:latest

Traceback (most recent call last):
  File "./docker-copyedit.py", line 538, in <module>
    edit_image(inp, out, commands)
  File "./docker-copyedit.py", line 91, in edit_image
    sh(cmd.format(**locals()))
  File "./docker-copyedit.py", line 41, in sh
    raise Exception("shell command failed")
Exception: shell command failed
Exit code: 1
[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$

But this works:

[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$ ./docker-copyedit.py FROM mysql:5.6 INTO mysql-novol:5.6 -vv REMOVE VOLUME /var/lib/mysql
INFO:edit:written new load.tmp/data/19706e2768c04c8301865815ea821c173e3715f40f470a17a057b667db610d54.json
INFO:edit:removed old load.tmp/data/1f47fade220d0ddedb916156cf4d7121a020afff03c19b4638ac494c5d1eca63.json

Manifest file already exists

Hi Guido,

I wanted to remove the /database volume from the ibmcom/db2 image.
But when I process it with your script using:
docker-copyedit.py FROM ibmcom/db2:latest INTO db2:1.0 REMOVE ALL VOLUMES
I get the error that the manifest file already exists:
Traceback (most recent call last): File "docker-copyedit.py", line 929, in <module> edit_image(inp, out, commands) File "docker-copyedit.py", line 301, in edit_image changed = edit_datadir(datadir, out_tag, edits) File "docker-copyedit.py", line 723, in edit_datadir os.rename(manifest_filename + ".tmp", manifest_filename) FileExistsError: [WinError 183] Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist: 'load.tmp\\data\\manifest.json.tmp' -> 'load.tmp\\data\\manifest.json'

I tried to fix it but I'am a novice to python.

Thanks Mario

Manifest not found

Hi Guido,

I wanted to remove the /xyz volume from the image1.
But when I process it with your script using:
python docker-copyedit.py from image1 into image2 -vv remove volume /xyz

and start the container i get the message

pulling image2 ...
ERROR: manifest for image2 not found: manifest unknown: manifest unknown

What can I do?

Thanks Uwe

Tabs/spaces broken

Hi, I've spotted that the main python file has a number of places where indent is broken. Can you please re-tab (re-space) it?

Possible to alter arm64 images on amd64 machine?

Using moby buildkit and buildx bake I can build arm64 images on my amd64 machine. Is there a way to use your script to alter an arm64 image but doing so on a amd64 machine? I assume it would, like moby buildkit, have to do so in a dockerized arm64 container using qemu.

speed up docker save process

the tar file generated from "docker save" is optionally keeped by the control of KEEPDIR , which I understand mostly for debugging purpose.
but even KEEPDIR =0, the tar file is always saved on disk, only deleted afterwards
I am a bit impatient and did following patch , pipe-lining with tee , to save disk IO, save time

23c23
< KEEPDIR = 1

KEEPDIR = 0
251,256c251,254
< if KEEPDIR >= 2:
< cmd = "docker save {inp} |tee {inputfile}|tar x -f - -C {datadir}"
< sh(cmd.format(**locals()))
< else:
< cmd = "docker save {inp} |tar x -f - -C {datadir}"
< sh(cmd.format(**locals()))

    cmd = "docker save {inp} -o {inputfile}"
    sh(cmd.format(**locals()))
    cmd = "tar xf {inputfile} -C {datadir}"
    sh(cmd.format(**locals()))

280,284c278,282
< # if KEEPDIR >= 2:
< # logg.warning("keeping %s", inputfile)
< # else:
< # if os.path.exists(inputfile):
< # os.remove(inputfile)

    if KEEPDIR >= 2:
        logg.warning("keeping %s", inputfile)
    else:
        if os.path.exists(inputfile):
            os.remove(inputfile)

Removing ports doesn't work

I've tried

/docker-copyedit.py FROM nginx:alpine INTO nginx:noexpose REMOVE PORT 80
/docker-copyedit.py FROM nginx:alpine INTO nginx:noexpose remove all ports
Both create a new container, but without the EXPOSE removed

docker inspect nginx:noexpose | jq '.[].ContainerConfig.ExposedPorts'
{
  "80/tcp": {}
}

Copyedit script no longer executable

Hi, thanks for this handy util!

The most recent commit changed the permission bits on the copyedit script. Its not difficult to get around - but it means the docs dont work out of the box

gdraheim / docker-copyedit Goto Github PK

docker-copyedit's Introduction

edit docker image metadata

docker-copyedit's People

Stargazers

Watchers

Forkers

docker-copyedit's Issues

23c23 < KEEPDIR = 1

280,284c278,282 < # if KEEPDIR >= 2: < # logg.warning("keeping %s", inputfile) < # else: < # if os.path.exists(inputfile): < # os.remove(inputfile)

Recommend Projects

Recommend Topics

Recommend Org

23c23
< KEEPDIR = 1

280,284c278,282
< # if KEEPDIR >= 2:
< # logg.warning("keeping %s", inputfile)
< # else:
< # if os.path.exists(inputfile):
< # os.remove(inputfile)