Weekly image cannot be used as base for new images about docker HOT 20 CLOSED

I'm wondering why you are adding files in you own image instead of providing your own volume container for those directories? I ask because you would loose the data in those directories the second your jenkins server container has been removed. Check out my project I'm setting up for our internal Jenkins setup. I'm not saying it's exactly what you need, but it shows the usage of the volume container.

https://github.com/mtscout6/jenkins-docker

from docker.

The purpose of my image is to provide a Jenkins instance similar to that of
our production Jenkins server. Having that makes it possible to develop
and/or debug the build scripts we execute in Jenkins without committing the
changes to our VCS. The scripts are provided through read/write volumes and
can thus be edited on the host, while the Jenkins setup as such is
unchanged from one run to the next.
On Oct 13, 2014 5:59 PM, "Matt Smith" [email protected] wrote:

I'm wondering why you are adding files in you own image instead of
providing your own volume container for those directories? I ask because
you would loose the data in those directories the second your jenkins
server container has been removed. Check out my project I'm setting up for
our internal Jenkins setup. I'm not saying it's exactly what you need, but
it shows the usage of the volume container.

https://github.com/mtscout6/jenkins-docker

—
Reply to this email directly or view it on GitHub
#26 (comment)
.

from docker.

We have thought about this. It was viewed that the risk of people not using -v was too high - storing files incontainer (data) is bad - and is a bad experience.

However - it makes things like this really hard to do - ie adding bundled plugins and such...
The closest I could think of was someting that updated init.groovy just before startup and have it to the work the first time - which is nothing to do with docker - but perhaps @ndeloof has some other ideas as I know he was talking about it.

from docker.

I created https://issues.jenkins-ci.org/browse/JENKINS-24986 for a comparable use-case. jenksin_home must be a volume (for persistence) but then we can't customize jenkins installation as this one only do support a single home directory.

Sole option I can image so far is to have the jenkins.sh script copy files from image to jenkins_home if they don't yet exists. Something like :

for (f in /var/jenkins_ref/*) do
   if [ -e /var/jenkins_home/$f ] cp -r /var/jenkins_ref/$f /var/jenkins_home/
done

wdyt ?

from docker.

The suggested approach is error prone since there may be files left in the
destinations inside the volume from the last container.

An alternative that would work for built-in jobs and workspaces would be to
change config.xml before starting Jenkins such that the pattern for
workspace and jobs point to directories outside the volume, e.g.
/var/jenkins_ref/{workspace,jobs}. This approach will however not help with
plugins and tools.

I suppose it could be investigated if softlinks could point from the volume
and into the container (replacing the plugins and tools folders in the
volume with a symlink), but I can imagine that would not work with volumes
residing on an NTFS partition.

My pragmatic workaround so far consists of downloading your Dockerfile and
removing the VOLUME instruction as well as the USER and related
instructions (I experienced permission problems). Works for me.

On 24 October 2014 08:24, Nicolas De loof [email protected] wrote:

I created https://issues.jenkins-ci.org/browse/JENKINS-24986 for a
comparable use-case. jenksin_home must be a volume (for persistence) but
then we can't customize jenkins installation as this one only do support a
single home directory.

Sole option I can image so far is to have the jenkins.sh script copy files
from image to jenkins_home if they don't yet exists. Something like :

for (f in /var/jenkins_ref/*) do
if [ -e /var/jenkins_home/$f ] cp -r /var/jenkins_ref/$f /var/jenkins_home/
done

wdyt ?

—
Reply to this email directly or view it on GitHub
#26 (comment)
.

from docker.

files to be left in the volume from last container is the expected behavior. Your jenkins_home should be persistent and evolve with jenkins use buy developers team. If you change the workspace setting then any change made to a jenkins instance won't survive a container fresh restart (i.e. from base image). Container data should ALWAYS be persisted on a volume, consider your container as immutable, only use container non-volume folders for temporary files.

I also don't want the container to enforce plugins or config files, just to provide reasonable defaults value, then let user customize them. So the proposed if [ -e ... ] do cp ... - this let initial installation have the adequate configuration file set based on the "best practices" you want to promote, but then customizing settings is under user responsibility.

from docker.

The purpose of my derived image is to allow toolsmiths to hack on our
custom build scripts without making permanent changes to our shared Jenkins
installation. I want each toolsmith to be able to start with a 'clean'
environment (i.e. without jobs for all our different projects) but with the
usual set of plugins that we use in our build scripts. This way, a
toolsmith can work our build scripts locally and verify that they will work
as intended in an environment that looks very similar to our shared one,
but without breaking things.

Sune :)

On 24 October 2014 10:57, Nicolas De loof [email protected] wrote:

files to be left in the volume from last container is the expected
behavior. Your jenkins_home should be persistent and evolve with jenkins
use buy developers team. If you change the workspace setting then any
change made to a jenkins instance won't survive a container fresh restart
(i.e. from base image). Container data should ALWAYS be persisted on a
volume, consider your container as immutable, only use container non-volume
folders for temporary files.

I also don't want the container to enforce plugins or config files, just
to provide reasonable defaults value, then let user customize them. So the
proposed if [ -e ... ] do cp ... - this let initial installation have the
adequate configuration file set based on the "best practices" you want to
promote, but then customizing settings is under user responsibility.

—
Reply to this email directly or view it on GitHub
#26 (comment)
.

from docker.

sure, but this doesn't prevent them to use a volume... I don't get your argument.
the "copy file if not yet exists in jenkins_home" should be fine to address this use-case

from docker.

I don't want any extra plugins the dev installed in their previous session
to be present in the next. If they are in a volume, I am forced to remove
the volume if I want a clean start.

Sune :)

On 24 October 2014 11:29, Nicolas De loof [email protected] wrote:

Closed #26 #26.

—
Reply to this email directly or view it on GitHub
#26 (comment)
.

from docker.

they just can choose a distinct volume... or just run without -v (VOLUME is only metadata, not actually used by docker) I use to run without a volume to test the image before it get published.

from docker.

We originally didn't have a volume and prople complained - as volumes are
really neded to store any sort of persistent data. The jenkins_home is
essentially a data store directory.

I would have been happy to leave out volume but it seems even the most
basic instructions are beyond a large number of users of docker (it
arguably makes down things too easy)

On Saturday, October 25, 2014, Nicolas De loof [email protected]
wrote:

they just can choose a distinct volume... or just run without -v (VOLUME
is only metadata, not actually used by docker) I use to run without a
volume to test the image before it get published.

—
Reply to this email directly or view it on GitHub
#26 (comment)
.

Michael D Neale
home: www.michaelneale.net
blog: michaelneale.blogspot.com

from docker.

When I run the following command:

docker run --name jenkins -d -it jenkins:weekly

I can inspect the container and get the metadata below, including an automatically created volume under /var/lib/docker/vfs. This indicates that the volume is created regardless of any -v arguments, and fits with my described experience.

[{
    "Args": [],
    "Config": {
        "AttachStderr": false,
        "AttachStdin": false,
        "AttachStdout": false,
        "Cmd": null,
        "CpuShares": 0,
        "Cpuset": "",
        "Domainname": "",
        "Entrypoint": [
            "/usr/local/bin/jenkins.sh"
        ],
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "JAVA_VERSION=7u65",
            "JENKINS_VERSION=1.585",
            "JENKINS_HOME=/var/jenkins_home"
        ],
        "ExposedPorts": {
            "50000/tcp": {},
            "8080/tcp": {}
        },
        "Hostname": "cd4a04aa883a",
        "Image": "jenkins:weekly",
        "Memory": 0,
        "MemorySwap": 0,
        "NetworkDisabled": false,
        "OnBuild": null,
        "OpenStdin": true,
        "PortSpecs": null,
        "SecurityOpt": null,
        "StdinOnce": false,
        "Tty": true,
        "User": "jenkins",
        "Volumes": {
            "/var/jenkins_home": {}
        },
        "WorkingDir": ""
    },
    "Created": "2014-10-25T08:10:58.57853994Z",
    "Driver": "aufs",
    "ExecDriver": "native-0.2",
    "HostConfig": {
        "Binds": null,
        "CapAdd": null,
        "CapDrop": null,
        "ContainerIDFile": "",
        "Devices": [],
        "Dns": null,
        "DnsSearch": null,
        "ExtraHosts": null,
        "Links": null,
        "LxcConf": [],
        "NetworkMode": "bridge",
        "PortBindings": {},
        "Privileged": false,
        "PublishAllPorts": false,
        "RestartPolicy": {
            "MaximumRetryCount": 0,
            "Name": ""
        },
        "VolumesFrom": null
    },
    "HostnamePath": "/var/lib/docker/containers/cd4a04aa883a91c83b43804a20e231009a0a3fe786ba035265ee900f81d767e3/hostname",
    "HostsPath": "/var/lib/docker/containers/cd4a04aa883a91c83b43804a20e231009a0a3fe786ba035265ee900f81d767e3/hosts",
    "Id": "cd4a04aa883a91c83b43804a20e231009a0a3fe786ba035265ee900f81d767e3",
    "Image": "f39b5fc33e56201b7dcef5ca17a8676cdbe16705e09e60ec361864e9e7f147a9",
    "MountLabel": "",
    "Name": "/jenkins",
    "NetworkSettings": {
        "Bridge": "docker0",
        "Gateway": "172.17.42.1",
        "IPAddress": "172.17.0.3",
        "IPPrefixLen": 16,
        "MacAddress": "02:42:ac:11:00:03",
        "PortMapping": null,
        "Ports": {
            "50000/tcp": null,
            "8080/tcp": null
        }
    },
    "Path": "/usr/local/bin/jenkins.sh",
    "ProcessLabel": "",
    "ResolvConfPath": "/var/lib/docker/containers/cd4a04aa883a91c83b43804a20e231009a0a3fe786ba035265ee900f81d767e3/resolv.conf",
    "State": {
        "ExitCode": 0,
        "FinishedAt": "0001-01-01T00:00:00Z",
        "Paused": false,
        "Pid": 15572,
        "Restarting": false,
        "Running": true,
        "StartedAt": "2014-10-25T08:10:59.324796982Z"
    },
    "Volumes": {
        "/var/jenkins_home": "/var/lib/docker/vfs/dir/cc1eaad5c966560017d57fe494fbd40aa9cd008b8d4407ce14b4e4ae37b81aae"
    },
    "VolumesRW": {
        "/var/jenkins_home": true
    }
}
]

from docker.

this demonstrate docker daemon dynamically created a volume for this container. This volume won't be reused by a subsequent docker run command.

from docker.

I realize now that my beef is really with how fig up 'recreates' containers by reusing volume bindings from previously created containers for the same service. Since we're using fig for orchestrating the local dev environment, we will have to either run fig rm jenkins each time a clean environment is needed. Or we can create our own image without a VOLUME instruction.

from docker.

interesting. Need to investigate on fig usage on my own.

from docker.

That makes sense. They are trying to prevent removal of data by default,
but you specifically want it to remove.

On Sunday, October 26, 2014, Nicolas De loof [email protected]
wrote:

interesting. Need to investigate on fig usage on my own.

—
Reply to this email directly or view it on GitHub
#26 (comment)
.

Michael D Neale
home: www.michaelneale.net
blog: michaelneale.blogspot.com

from docker.

I guess the conclusion is that we have to maintain our own weekly images to
suit our special purpose.

On Sat, Oct 25, 2014, 23:49 Michael Neale [email protected] wrote:

That makes sense. They are trying to prevent removal of data by default,
but you specifically want it to remove.

On Sunday, October 26, 2014, Nicolas De loof [email protected]
wrote:

interesting. Need to investigate on fig usage on my own.

—
Reply to this email directly or view it on GitHub
<
https://github.com/cloudbees/jenkins-ci.org-docker/issues/26#issuecomment-60486415>

.

Michael D Neale
home: www.michaelneale.net
blog: michaelneale.blogspot.com

—

Reply to this email directly or view it on GitHub
#26 (comment)
.

from docker.

Or have people aware of fig behaviour for this specific usage.
Le 26 oct. 2014 07:46, "Sune Keller" [email protected] a écrit :

I guess the conclusion is that we have to maintain our own weekly images
to
suit our special purpose.

On Sat, Oct 25, 2014, 23:49 Michael Neale [email protected]
wrote:

That makes sense. They are trying to prevent removal of data by default,
but you specifically want it to remove.

On Sunday, October 26, 2014, Nicolas De loof [email protected]
wrote:

interesting. Need to investigate on fig usage on my own.

—
Reply to this email directly or view it on GitHub
<

https://github.com/cloudbees/jenkins-ci.org-docker/issues/26#issuecomment-60486415>

.

Michael D Neale
home: www.michaelneale.net
blog: michaelneale.blogspot.com

—

Reply to this email directly or view it on GitHub
<
https://github.com/cloudbees/jenkins-ci.org-docker/issues/26#issuecomment-60498608>

.

—
Reply to this email directly or view it on GitHub
#26 (comment)
.

from docker.

We'd rather have the burden on us to create an image of our own rather than
have each user remember specific steps for each iteration.

On Sun, Oct 26, 2014, 07:48 Nicolas De loof [email protected]
wrote:

Or have people aware of fig behaviour for this specific usage.
Le 26 oct. 2014 07:46, "Sune Keller" [email protected] a écrit :

I guess the conclusion is that we have to maintain our own weekly images
to
suit our special purpose.

On Sat, Oct 25, 2014, 23:49 Michael Neale [email protected]
wrote:

That makes sense. They are trying to prevent removal of data by
default,
but you specifically want it to remove.

On Sunday, October 26, 2014, Nicolas De loof [email protected]

wrote:

interesting. Need to investigate on fig usage on my own.

—
Reply to this email directly or view it on GitHub
<

https://github.com/cloudbees/jenkins-ci.org-docker/issues/26#issuecomment-60486415>

.

Michael D Neale
home: www.michaelneale.net
blog: michaelneale.blogspot.com

—

Reply to this email directly or view it on GitHub
<

https://github.com/cloudbees/jenkins-ci.org-docker/issues/26#issuecomment-60498608>

.

—
Reply to this email directly or view it on GitHub
<
https://github.com/cloudbees/jenkins-ci.org-docker/issues/26#issuecomment-60508267>

.

—
Reply to this email directly or view it on GitHub
#26 (comment)
.

from docker.

Issue at docker github: moby/moby#3484

I was wondering if an ONBUILD VOLUME instruction would do the trick, but pprobably it doesn't apply when running the base image itself...

from docker.