Giter Club home page Giter Club logo

systemd-docker's Introduction

systemd-docker

This is a wrapper for docker run so that you can sanely run Docker containers under systemd. The key thing that this wrapper does is move the container process from the cgroups setup by Docker to the service unit's cgroup. This handles a bunch of other quirks so please read through documentation to get an understanding of all the implications of running Docker under systemd.

Using this wrapper you can manage containers through systemctl or the docker CLI and everything should just stay in sync. Additionally you can leverage all the cgroup functionality of systemd and systemd-notify.

Why I wrote this?

The full context is in Docker Issue #6791 and this mailing list thread. The short of it is that systemd does not actually supervise the Docker container but instead the Docker client. This makes systemd incapable of reliably managing Docker containers without hitting a bunch of really odd situations.

Installation

Copy systemd-docker to /opt/bin (really anywhere you want). You can download/compile through the normal go get github.com/ibuildthecloud/systemd-docker

Quick Usage

Basically, in your unit file use systemd-docker run instead of docker run. Here's an example unit file that runs nginx.

[Unit]
Description=Nginx
After=docker.service
Requires=docker.service

[Service]
ExecStart=/opt/bin/systemd-docker run --rm --name %n nginx
Restart=always
RestartSec=10s
Type=notify
NotifyAccess=all
TimeoutStartSec=120
TimeoutStopSec=15

[Install]
WantedBy=multi-user.target

If you are writing your own unit file, Type=notify and NotifyAccess=all is really important

Special Note About Named Containers

In short, it's best to always have --name %n --rm in your unit file's ExecStart.

The best way I've found to run containers under systemd is to always assign the container a name. Even better is to put --name %n in your unit file and then the name of the container will match the name of the service unit.

If you don't name your container, you will essentially be creating a new container on every start that will get orphaned. You're probably clever and thinking you can just add --rm and that will take care of the orphans. The problem with this is that --rm is not super reliable. By naming your container, systemd-docker will take extra care to keep the systemd unit and the container in sync. For example, if you do --name %n --rm, systemd-docker will ensure that the container is really deleted each time. The issue with --rm is that the remove is done from the client side. If the client dies, the container is not deleted.

If you do --name %n --rm, systemd-docker on start will look for the named container. If it exists and is stopped, it will be deleted. This is really important if you ever change your unit file. If you change your ExecStart command, and it is a named container, the old values will be saved in the stopped container. By ensuring the container is always deleted, you ensure the args in ExecStart are always in sync.

Options

Logging

By default the container's stdout/stderr will be piped to the journal. If you do not want to use the journal, add --logs=false to the beginning of the command. For example:

ExecStart=/opt/bin/systemd-docker --logs=false run --rm --name %n nginx

Environment Variables

Using Environment= and EnvironmentFile=, systemd can set up environment variables for you, but then unfortunately you have to do run -e ABC=${ABC} -e XYZ=${XYZ} in your unit file. You can have the systemd environment variables automatically transfered to your docker container by adding --env. This will essentially read all the current environment variables and add the appropriate -e ... flags to your docker run command. For example:

EnvironmentFile=/etc/environment
ExecStart=/opt/bin/systemd-docker --env run --rm --name %n nginx

The contents of /etc/environment will be added to your docker run command

Cgroups

The main magic of how this works is that the container processes are moved from the Docker cgroups to the system unit cgroups. By default all application cgroups will be moved. This means by default you can't use --cpuset or -m in Docker. If you don't want to use the systemd cgroups, but instead use the Docker cgroups, you can control which cgroups are transfered using the --cgroups option. Minimally you must set name=systemd; otherwise, systemd will lose track of the container. For example

ExecStart=/opt/bin/systemd-docker --cgroups name=systemd --cgroups=cpu run --rm --name %n nginx

The above command will use the name=systemd and cpu cgroups of systemd but then use Docker's cgroups for all the others, like the freezer cgroup.

Pid File

If for whatever reason you want to create a pid file for the container PID, you can. Just add --pid-file as below

ExecStart=/opt/bin/systemd-docker --pid-file=/var/run/%n.pid --env run --rm --name %n nginx

systemd-notify support

By default systemd-docker will send READY=1 to the systemd notification socket. You can instead delegate the READY=1 call to the container itself. This is done by adding --notify. For example

ExecStart=/opt/bin/systemd-docker --notify run --rm --name %n nginx

What this will do is set up a bind mount for the notification socket and then set the NOTIFY_SOCKET environment variable. If you are going to use this feature of systemd, take some time to understand the quirks of it. More info in this mailing list thread. In short, systemd-notify is not reliable because often the child dies before systemd has time to determine which cgroup it is a member of

Detaching the client

The -d argument to docker has no effect under systemd-docker. To cause the systemd-docker client to detach after the container is running, simply use --logs=false --rm=false. If either --logs or --rm is true, the systemd-docker client will stay alive until it is killed or the container exits.

Running on CoreOS

If you are running on CoreOS, it may be more problematic to install systemd-docker to /opt/bin. To make this easier add the following line to your unit file.

ExecStartPre=/usr/bin/docker run --rm -v /opt/bin:/opt/bin ibuildthecloud/systemd-docker

That command will install systemd-docker to /opt/bin. The full nginx example that is above would now be as below.

[Unit]
Description=Nginx
After=docker.service
Requires=docker.service

[Service]
ExecStartPre=/usr/bin/docker run --rm -v /opt/bin:/opt/bin ibuildthecloud/systemd-docker
ExecStart=/opt/bin/systemd-docker run --rm --name %n nginx
Restart=always
RestartSec=10s
Type=notify
NotifyAccess=all
TimeoutStartSec=120
TimeoutStopSec=15

[Install]
WantedBy=multi-user.target

Known issues

Inconsistent cgroup

CentOS 7 is inconsistent in the way it handles some cgroups. It has 3:cpuacct,cpu:/user.slice in /proc/[pid]/cgroups which is inconsistent with the cgroup path /sys/fs/cgroup/cpu,cpuacct/ that systemd-docker is trying to move pids to.

This will cause systemd-docker to fail unless run withsystemd-docker --cgroups name=systemd run

See #15 for details.

License

Apache License, Version 2.0

systemd-docker's People

Contributors

apatil avatar bcwaldon avatar clehene avatar emmetog avatar ibuildthecloud avatar jonboulle avatar marcelk avatar notti avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

systemd-docker's Issues

docker/pkg/mflag dependency no longer exists

When I try to build systemd-docker, it fails with the error message 'cannot find package "github.com/docker/docker/pkg/mflag"', and indeed, it looks like that package was removed from the docker sources in June 2016.

At this point I'm trying to decide whether it makes more sense to try to fix this, or to find another way to deploy Docker containers...

Tom

json: cannot unmarshal object into Go value of type string

I get this error trying to run this script

/usr/local/bin/systemd-docker run --rm --name mycontainer
docker info
lient:
 Debug Mode: false

Server:
 Containers: 7
  Running: 1
  Paused: 0
  Stopped: 6
 Images: 18
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.3.0-7625-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.22GiB
 Name: #####
 ID: ######
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

ExecStartPost not executed

Hi,

I'm using your project, and it's very handy. I can't wait to switch to rocket actually...

I'm hitting a problem, I have the following unit file, but the ExecStartPost is randomly executed, Sometime it is, sometimes, it isn't. I can't figure out why. I'm wondering if it could be systemd-docker.

Here is the output from the unit start:

Feb 18 12:41:00 t1 systemd[1]: Starting web-known.test...
Feb 18 12:41:00 t1 docker[3120]: known.test
Feb 18 12:41:00 t1 docker[3150]: known.test
Feb 18 12:41:00 t1 bash[3217]: + /opt/bin/systemd-docker --env run --rm --name known.test -v /data/runtime/domains/known.test/log/apache2:/var/log/apache2 --link mysql-known.test:db -v /data/domains/known.test/kno
Feb 18 12:41:00 t1 systemd[1]: [email protected]: Supervising process 3399 which is not our child. We'll most likely not notice when it exits.
Feb 18 12:41:00 t1 systemd[1]: Started web-known.test.
Feb 18 12:41:00 t1 bash[3217]: 2015/02/18 12:41:00 Moving pid 3399 to /sys/fs/cgroup/cpuset/cgroup.procs
Feb 18 12:41:00 t1 bash[3217]: 2015/02/18 12:41:00 Moving pid 3399 to /sys/fs/cgroup/freezer/cgroup.procs
Feb 18 12:41:00 t1 bash[3217]: 2015/02/18 12:41:00 Moving pid 3399 to /sys/fs/cgroup/devices/system.slice/system-web.slice/cgroup.procs
Feb 18 12:41:00 t1 bash[3217]: 2015/02/18 12:41:00 Moving pid 3399 to /sys/fs/cgroup/memory/system.slice/system-web.slice/cgroup.procs
Feb 18 12:41:00 t1 bash[3217]: 2015/02/18 12:41:00 Moving pid 3399 to /sys/fs/cgroup/blkio/system.slice/system-web.slice/cgroup.procs
Feb 18 12:41:00 t1 bash[3217]: 2015/02/18 12:41:00 Moving pid 3399 to /sys/fs/cgroup/cpu,cpuacct/system.slice/system-web.slice/cgroup.procs
Feb 18 12:41:00 t1 bash[3217]: 2015/02/18 12:41:00 Moving pid 3399 to /sys/fs/cgroup/systemd/system.slice/system-web.slice/[email protected]/cgroup.procs
Feb 18 12:41:01 t1 bash[3217]: => Trying to connect to MySQL/MariaDB using:
Feb 18 12:41:01 t1 bash[3217]: ========================================================================
Feb 18 12:41:01 t1 bash[3217]: Database Host Address:  172.17.0.19
Feb 18 12:41:01 t1 bash[3217]: Database Port number:   3306
Feb 18 12:41:01 t1 bash[3217]: Database Name:          known
Feb 18 12:41:01 t1 bash[3217]: Database Username:      admin
Feb 18 12:41:01 t1 bash[3217]: Database Password:      OThjOTBkN2
Feb 18 12:41:01 t1 bash[3217]: ========================================================================
Feb 18 12:41:06 t1 bash[3217]: => Skipped creation of database known <E2><80><93> it already exists.
Feb 18 12:41:06 t1 bash[3217]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.21. Set the 'ServerName' directive globally to suppress this message

But it never called the ExecStartStop script.

Do you have any idea? If not, I'll keep it in a separate file, and wait to switch to rocket.

error when running go get

Hi, if i try to go get systemd-docker i got the following errors, if i clone the git repo and run build, it works without errors

koch@linux-np5s:~/Workspace/go> go get github.com/ibuildthecloud/systemd-docker
# github.com/ibuildthecloud/systemd-docker
src/github.com/ibuildthecloud/systemd-docker/main.go:169: unknown docker.RemoveContainerOptions field 'Force' in struct literal
src/github.com/ibuildthecloud/systemd-docker/main.go:265: undefined: docker.NewVersionedClient
src/github.com/ibuildthecloud/systemd-docker/main.go:474: client.Logs undefined (type *docker.Client has no field or method Logs)
src/github.com/ibuildthecloud/systemd-docker/main.go:474: undefined: docker.LogsOptions
src/github.com/ibuildthecloud/systemd-docker/main.go:523: unknown docker.RemoveContainerOptions field 'Force' in struct literal

Vendor code committed to the repository

Instead of committing vendor code directly to the repository and adding a readme file instructing people not to touch the files, why not just gitignore the files?

json: cannot unmarshal object into Go value of type string

I get this error trying to run this script

/usr/local/bin/systemd-docker run --rm --name mycontainer
docker info
lient:
 Debug Mode: false

Server:
 Containers: 7
  Running: 1
  Paused: 0
  Stopped: 6
 Images: 18
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.3.0-7625-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.22GiB
 Name: #####
 ID: ######
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

panic when docker unavailable

I ran a systemd-docker run and my docker.socket was up and listening, but the underlying docker service was not up:

Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: panic: runtime error: invalid memory address or nil pointer dereference
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: [signal 0xb code=0x1 addr=0x58 pc=0x4029ce]
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: goroutine 16 [running]:
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: runtime.panic(0x6c7940, 0x89ef53)
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: /usr/local/go/src/pkg/runtime/panic.c:279 +0xf5
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: main.lookupNamedContainer(0xc208048460, 0x0, 0x0)
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: /home/darren/src/systemd-docker/main.go:159 +0x11e
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: main.runContainer(0xc208048460, 0x0, 0x0)
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: /home/darren/src/systemd-docker/main.go:231 +0x46
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: main.mainWithArgs(0xc20800e010, 0x4, 0x4, 0x42986b, 0x0, 0x0)
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: /home/darren/src/systemd-docker/main.go:530 +0x8e
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: main.main()
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: /home/darren/src/systemd-docker/main.go:566 +0x75
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: goroutine 17 [runnable]:
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: runtime.MHeap_Scavenger()
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: /usr/local/go/src/pkg/runtime/mheap.c:507
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: runtime.goexit()
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: /usr/local/go/src/pkg/runtime/proc.c:1445
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: goroutine 18 [runnable]:
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: bgsweep()
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: /usr/local/go/src/pkg/runtime/mgc0.c:1976
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: runtime.goexit()
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: /usr/local/go/src/pkg/runtime/proc.c:1445
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: goroutine 19 [runnable]:
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: runfinq()
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: /usr/local/go/src/pkg/runtime/mgc0.c:2606
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: runtime.goexit()
Aug 19 17:03:59 ip-10-236-146-73.ec2.internal systemd-docker[639]: /usr/local/go/src/pkg/runtime/proc.c:1445

Docker daemon is moved to service's slice

Hi,
On a debian freshly upgraded to jessie, using the example unit file from the Quick Usage section causes systemd-docker to move the docker daemon itself to the new cgroup.

For example (The outputs have been shortened):

$ sudo systemctl status
● nuc
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Sun 2015-04-26 18:50:16 CEST; 1 day 16h ago
   CGroup: /
           ├─1 /sbin/init
           └─system.slice
             [...]
             ├─docker.service
             │ └─26127 /usr/bin/docker -d -H fd://
             [...]
$ sudo systemctl start nginx
$ sudo systemctl status
● nuc
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Sun 2015-04-26 18:50:16 CEST; 1 day 16h ago
   CGroup: /
           ├─1 /sbin/init
           └─system.slice
             [...]
             ├─nginx.service
             │ ├─26127 /usr/bin/docker -d -H fd://
             │ ├─26164 /usr/local/bin/systemd-docker --cgroups name=systemd run --rm --name nginx.service nginx
             │ ├─26177 nginx: master process nginx -g daemon off
             │ └─26201 nginx: worker proces
             [...]
$ sudo journalctl -u nginx.service
[...]
Apr 28 12:08:34 nuc systemd[1]: nginx.service: Supervising process 26177 which is not our child. We'll most likely not notice when it exits.
Apr 28 12:08:34 nuc systemd-docker[26164]: 2015/04/28 12:08:34 Moving pid 26127 to /sys/fs/cgroup/systemd/system.slice/nginx.service/cgroup.procs
Apr 28 12:08:34 nuc systemd-docker[26164]: 2015/04/28 12:08:34 Moving pid 26177 to /sys/fs/cgroup/systemd/system.slice/nginx.service/cgroup.procs
Apr 28 12:08:34 nuc docker[26127]: time="2015-04-28T12:08:34+02:00" level=info msg="GET /v1.11/version"
Apr 28 12:08:34 nuc docker[26127]: time="2015-04-28T12:08:34+02:00" level=info msg="+job version()"
Apr 28 12:08:34 nuc docker[26127]: time="2015-04-28T12:08:34+02:00" level=info msg="-job version() = OK (0)"
Apr 28 12:08:34 nuc docker[26127]: time="2015-04-28T12:08:34+02:00" level=info msg="GET /v1.11/version"
Apr 28 12:08:34 nuc docker[26127]: time="2015-04-28T12:08:34+02:00" level=info msg="+job version()"
[...]
$ cat /sys/fs/cgroup/systemd/system.slice/docker.service/cgroup.procs
$ cat /sys/fs/cgroup/systemd/system.slice/nginx.service/cgroup.procs
26127
26164
26177
26201

This is using --cgroups name=systemd. It behaves the same way without it.
If other containers are running, they get moved as well.

systemd comes from the official debian repositories, docker from the get.docker.io one

$ systemctl --version
systemd 215
+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR
$ docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64

Wrong cgroup name on CentOS 7

Environment:

  • CentOS 7.0, systemd 208, docker 1.2.0/1.14

Test service unit file /etc/systemd/system/sd-test.service:

[Unit]
Description=test
After=docker.service
Requires=docker.service

[Service]
ExecStartPre=/usr/bin/docker run --rm -v /opt/bin:/opt/bin ibuildthecloud/systemd-docker
ExecStart=/opt/bin/systemd-docker run --rm --name %n busybox /bin/sh -c 'while true ; do date ; sleep 5 ; done'
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target

If you try to start it you'll have stopped (but not deleted) docker container. In systemctl status sd-test you can see something like this:

[root@test system]# systemctl status sd-test
sd-test.service - test
   Loaded: loaded (/etc/systemd/system/sd-test.service; disabled)
   Active: failed (Result: timeout) since Wed 2014-11-26 14:43:34 UTC; 5min ago
  Process: 11976 ExecStart=/opt/systemd-docker run --rm --name %n busybox /bin/sh -c while true ; do date ; sleep 5 ; done (code=exited, status=1/FAILURE)
 Main PID: 11976 (code=exited, status=1/FAILURE)

Nov 26 14:42:04 ac-test systemd-docker[11976]: 2014/11/26 14:42:04 Moving pid 12006 to /sys/fs/cgroup/systemd/system.slice/sd-test.service/cgroup.procs
Nov 26 14:42:04 ac-test systemd-docker[11976]: 2014/11/26 14:42:04 Moving pid 12006 to /sys/fs/cgroup/blkio/system.slice/sd-test.service/cgroup.procs
Nov 26 14:42:04 ac-test systemd-docker[11976]: 2014/11/26 14:42:04 Moving pid 12006 to /sys/fs/cgroup/freezer/cgroup.procs
Nov 26 14:42:04 ac-test systemd-docker[11976]: 2014/11/26 14:42:04 Moving pid 12006 to /sys/fs/cgroup/devices/system.slice/cgroup.procs
Nov 26 14:42:04 ac-test systemd-docker[11976]: 2014/11/26 14:42:04 Moving pid 12006 to /sys/fs/cgroup/memory/system.slice/cgroup.procs
Nov 26 14:42:04 ac-test systemd-docker[11976]: 2014/11/26 14:42:04 open /sys/fs/cgroup/cpuacct,cpu/system.slice/docker-688bc8168ef5fd406a91c7b2d5522eab3a1e969378857669ba8b50e6a2bb01d2.scope/cgroup.procs: no such file or directory
Nov 26 14:42:04 ac-test systemd[1]: sd-test.service: main process exited, code=exited, status=1/FAILURE
Nov 26 14:43:34 ac-test systemd[1]: sd-test.service stopping timed out. Killing.
Nov 26 14:43:34 ac-test systemd[1]: Failed to start test.
Nov 26 14:43:34 ac-test systemd[1]: Unit sd-test.service entered failed state.

This unit starts fine on:

  • ArchLinux with systemd 217, docker 1.3.1/1.15,
  • CoreOS 509.1.0, systemd 215, docker 1.3.2/1.15.

Also ls /sys/fs/cgroup shows that there's no cpuacct,cpu cgroup (on both centos, arch and coreos). But cpu,cpuacct is present and contains appropriate docker container scope.

Test environment that I used can be bootstrapped by

vagrant box add chef/centos-7.0
vagrant init chef/centos-7.0
vagrant up
# await it's started
vagrant ssh

# in vbox
sudo yum install -y docker

Is there a way to prevent systemd-docker from calling systemd-notify?

My use case is wanting to call systemd-notify when health checks against a container pass. This happens after the container starts, so systemd-docker's current notify call is too early.

I could use --notify=true and modify the code inside the container to do the health checks, but prefer to do the health checks from outside so that the container code is agnostic to whether it runs on systemd & therefore more portable.

Mention the docker version in the documentation

It would be nice if you would document the latest Docker version you have tested the assumptions described in the documentation.

As I'm reading, I'm wondering which of the information is still valid from your writings.

Is this maintained?

Sad to say this docker image is not actively maintained. Due to my current work situation I do not have the time to dedicate to properly ensure that this software continues to work. While I still very actively use Docker, I am no longer a user of systemd. Since I don't use this software myself any more, I'm not confident to say that it will continue to work.

systemd-docker from the beginning was a hack to work around incompatibilities between systemd and Docker. The intention was that overtime proper solutions could be found and systemd-docker could be phased out. Unfortunately that has not happened fast enough. There has not been much progress made on reconciling the difference between these two camps.

With the recent additions in Docker for --cgroup-parent and --exec-opt native.cgroupdriver=cgroupfs it is possible that there is a better approach than what systemd-docker does today. Unfortunately I don't have time to investigate this, it would still be a slight hack, and wouldn't work out of the box on CoreOS or CentOS/RHEL without first disabling the Docker systemd cgroup driver.

systemd-docker has been pulled over 500,000 times so I have good reason to believe it is used by a good amount of people. If there is anybody interested in taking over ownership of this project I'll gladly redirect traffic to a better maintained fork.

Getting systemd-docker

Hi, I'm trying to use systemd-docker and running into:

$ go get github.com/ibuildthecloud/systemd-docker
package github.com/ibuildthecloud/systemd-docker
    imports github.com/docker/docker/opts
    imports github.com/Sirupsen/logrus
    imports github.com/docker/libtrust
    imports github.com/docker/docker/autogen/dockerversion
    imports github.com/docker/docker/autogen/dockerversion
    imports github.com/docker/docker/autogen/dockerversion: cannot find package "github.com/docker/docker/autogen/dockerversion" in any of:
    /usr/src/go/src/github.com/docker/docker/autogen/dockerversion (from $GOROOT)
    /go/src/github.com/docker/docker/autogen/dockerversion (from $GOPATH)

I did see you'd opened a docker issue about this moby/moby#11699.

Unfortunately, I'm not a go programmer, so I don't understand the fix. Is there any chance you could provide a binary? For obvious reasons, most of us are going to be using 64bit linux...

Can't get stdout when pseudo-TTY is allocated

Hey,

I'm running an Erlang service in a container through systemd-docker. This service's foreground process is an Erlang console, which requires a TTY to be able to run.

If I run a container with the -t flag which makes Docker allocate a pseudo-TTY, the output of the container is seemingly lost. Is there a way to work around this?

Logging lost after docker restart

SystemD seems to lose the handles to the container's stdout and stderr after restarting it with docker restart. I see this:

Feb 04 19:12:23 iris systemd-docker[12192]: [04-Feb-2017 19:12:23] NOTICE: Terminating ...
Feb 04 19:12:23 iris systemd-docker[12192]: [04-Feb-2017 19:12:23] NOTICE: exiting, bye-bye!

And then nothing else, even though the container is (still) running, with the same container ID. Is there any way to reattach the journal after a restart?

Host config of container is reset when it starts

If systemd-docker is used without the "--rm" argument, the host configuration of the container is reset when the container is started for the second time. For example, if the "--privileged" parameter is used, the container will run in privileged mode the first time, but when it is started for the second time, it's not privileged any more. Also, the port bindings are gone, and all other HostConfig properties.

This is caused by this line in lookupNamedContainer:

err = client.StartContainer(container.ID, nil)

The second parameter, which is the host configuration, is left blank. I'll submit a pull request with a fix.

Manual Restart

I'm having some issues with manual service restarts via systemctl. For some reason doing systemctl restart on a service with systemd-docker doesn't restart it, instead it transitions to the inactive state. With restart policy always manual restart triggers auto-restart of the service. I don't see anything interesting in journal apart from occasional reports from the process of SIGTERM. Maybe you can help me understand or debug this situation. Thanks!

stopping service/container drops Docker socket file

On Debian when I systemctl stop container service it removes Docker socket /var/run/docker.sock which makes communication with Docker daemon impossible:

# docker ps
  dial unix /var/run/docker.sock: connect: no such file or directory.
  * Is your docker daemon up and running?

I can re-create socket file using systemctl restart docker.socket but it is quite inconvenient...

Docker client API version is too old

I'm getting the following error with the docker version shipped with CoreOS alpha images (1.7.1):

"client is too old, minimum supported API version is 1.12"

Not sure if the fix is trivial (just a version bump) or if more care is needed to update the API calls.

bind: cannot assign requested address

Hi,

I have the following mydockerserv.service:

[Unit]
Description=mydockerserv
After=docker.service
Requires=docker.service

[Service]
ExecStart=/opt/bin/systemd-docker --cgroups name=systemd --cgroups=cpu run  -p X.W.Y.Z:443:443    --name %n sebelk/mydockerserv
Restart=always
RestartSec=10s
Type=notify
NotifyAccess=all
TimeoutStartSec=120
TimeoutStopSec=15

[Install]
WantedBy=multi-user.target

If I change the old IP address of the host A.B.C.D by X.W.Y.Z, and then restart the service, it does not start the service, the error is:

Mar  9 07:23:43 newhope systemd: Unit mydockersev.service entered failed state.
Mar  9 07:23:43 newhope systemd: mydockerserv.service failed.
Mar  9 07:23:43 newhope systemd-docker: 2015/03/09 07:23:43 API error (500): Cannot start container d3e704d3135cfc99992f33d8da263bf79312fbb6f30aa8171dbb434363c4996d: Error starting userland proxy: listen tcp A.B.C.D:443: bind: cannot assign requested address

I've found that only removing the old container fixes the problem, any ideas?

Doesn't work with recent systemd and/or docker releases

Something went wrong after an update and I haven't dug into it too much since this project seems dead. Basically, systemd sees the systemd-docker executable as dying with a status code of 1, but the container is still running. I don't know if there's some mixup with Docker communication going on or if systemd changed somewhere to break it. I suspect Docker, but I have no evidence.

The best solution seems to be migrating to rkt for running the images, but that is blocked by rkt/rkt#2392 without the ability to push images to a registry.

all container scopes get removed after a while

This might be a systemd issue, a docker issue or a systemd-docker issue, but when using systemd-docker with --cgroups name=systemd only, the scope directory for the container eventually disappears from the standard cgroups in /sys/fs/cgroups/

This can be seen when using datadog for monitoring. For a while, cpu and mem metrics are seen in datadog, and eventually they stop showing up. The datadog collector logs show that at some point files like these disappear:

 /sys/fs/cgroup/cpu,cpuacct/system.slice/docker-dccaefeac08e6a5885fd479936ebaca1da770bdf30a940ed4827c2132c6721ba.scope/cpuacct.stat
 /sys/fs/cgroup/memory/system.slice/docker-a5a680186bedb98f4ec6838e6da5fa4d2bc2e79fc35a6c762ef88a23d3fe31da.scope/memory.stat

In the meantime, similar slices still exist in /sys/fs/cgroup/systemd/system.slice, but instead of being named docker-<container full-id>.scope, they are named docker-<container name>.scope.

I am at a loss as to why these scopes disappear all of a sudden. Any pointers on how to track this down would be very helpful. Looking at the code I can't see anything obvious.

Error while removing dangling images

I'm using custom script for updating containers, as a last step I'm cleaning dangling (docker images -f 'dangling=true'). When using systemd-docker in unit files, dangling images fail to remove for several seconds after unit restarts, after some time (about 5s) it works. Steps to reproduce:

  1. Pull new image;
  2. Restart unit to make it use new image;
  3. Try to clean dangling images (for example docker rmi $(docker images -q -f 'dangling=true')).

I think that:

  1. Ideally docker rmi should not fail on images returned by docker images -q -f 'dangling=true', or
  2. docker images -q -f 'dangling=true' should not return such images for container used by system-docker.

Would appreciate any suggestion about how I can solve/debug somehow this situation. Thanks.

Service dependencies cause startup hanging

I stumbled across this project last week and decided to try it out. I'm running Docker 1.6.2 on the latest CoreOS stable release.

Things seem to work properly for most of my containers, but I've noticed that certain services won't start from time-to-time.

It seems to be isolated to containers that have a "--link" to another container. Note that in these cases I use Requires= and After= to make sure systemd knows about the start order dependency.

There isn't much in the logs, the systemd start command simply hangs. Sometimes the docker container has actually started and systemd doesn't recognize it, sometimes it hasn't started the container.

Any ideas?

Exit code is not respected

$ docker run --rm -ti alpine sh -c "exit 1"                                                                                                                                                       
$ echo $?
1
$ sudo /opt/bin/systemd-docker run --rm -ti alpine sh -c "exit 1"                                                                                                                           
2016/06/20 09:24:38 Moving pid 3995 to /sys/fs/cgroup/memory/system.slice/system-sshd.slice/cgroup.procs
2016/06/20 09:24:38 Moving pid 3995 to /sys/fs/cgroup/cpuset/cgroup.procs
2016/06/20 09:24:38 Moving pid 3995 to /sys/fs/cgroup/freezer/cgroup.procs
2016/06/20 09:24:38 Moving pid 3995 to /sys/fs/cgroup/cpu,cpuacct/system.slice/system-sshd.slice/cgroup.procs
2016/06/20 09:24:38 Moving pid 3995 to /sys/fs/cgroup/blkio/system.slice/system-sshd.slice/cgroup.procs
2016/06/20 09:24:38 Moving pid 3995 to /sys/fs/cgroup/devices/system.slice/system-sshd.slice/cgroup.procs
2016/06/20 09:24:38 Moving pid 3995 to /sys/fs/cgroup/net_cls,net_prio/cgroup.procs
$ echo $?
0

Can this be used with `socket` systemd items?

The documentation shows how to make a systemd service pretty easily. This ask is that the documentation comment on the suitability of this solution for a socket systemd definition which uses such a service.

journal plagued with log lines

I've found that systemd is full of lines like that when I issue journalctl -f

jul 09 21:13:47 newhope.belkin.home docker[1430]:
time="2015-07-09T21:13:47-03:00" level=info msg="GET
/v1.11/containers/35f101d97881e543d40370ee4242216dfb88103f2947c0446ad0ad03a1bcd810/json"
jul 09 21:13:47 newhope.belkin.home docker[1430]:
time="2015-07-09T21:13:47-03:00" level=info msg="+job
container_inspect(35f101d97881e543d40370ee4242216dfb88103f2947c0446ad0ad03a1bcd810)"
jul 09 21:13:47 newhope.belkin.home docker[1430]:
time="2015-07-09T21:13:47-03:00" level=info msg="-job
container_inspect(35f101d97881e543d40370ee4242216dfb88103f2947c0446ad0ad03a1bcd810)
= OK (0)" jul 09 21:13:48 newhope.belkin.home docker[1430]: time="2015-07-09T21:13:48-03:00" level=info msg="GET
/v1.11/containers/35f101d97881e543d40370ee4242216dfb88103f2947c0446ad0ad03a1bcd810/json"
jul 09 21:13:48 newhope.belkin.home docker[1430]:
time="2015-07-09T21:13:48-03:00" level=info msg="+job
container_inspect(35f101d97881e543d40370ee4242216dfb88103f2947c0446ad0ad03a1bcd810)"
jul 09 21:13:48 newhope.belkin.home docker[1430]:
time="2015-07-09T21:13:48-03:00" level=info msg="-job
container_inspect(35f101d97881e543d40370ee4242216dfb88103f2947c0446ad0ad03a1bcd810)
= OK (0)" jul 09 21:13:49 newhope.belkin.home docker[1430]: time="2015-07-09T21:13:49-03:00" level=info msg="GET
/v1.11/containers/35f101d97881e543d40370ee4242216dfb88103f2947c0446ad0ad03a1bcd810/json"
jul 09 21:13:49 newhope.belkin.home docker[1430]:
time="2015-07-09T21:13:49-03:00" level=info msg="+job
container_inspect(35f101d97881e543d40370ee4242216dfb88103f2947c0446ad0ad03a1bcd810)"
jul 09 21:13:49 newhope.belkin.home docker[1430]:
time="2015-07-09T21:13:49-03:00" level=info msg="-job
container_inspect(35f101d97881e543d40370ee4242216dfb88103f2947c0446ad0ad03a1bcd810)
= OK (0)

I have a configuration as follows:

# cat /etc/systemd/system/kanboard.service 
[Unit]
Description=kanboard
After=docker.service
Requires=docker.service
`
[Service]
ExecStart=/opt/bin/systemd-docker --cgroups name=systemd --cgroups=cpu run --rm -p 192.168.0.250:443:443    --volumes-from  data  --name %n sebelk/https 
Restart=always
RestartSec=10s
Type=notify
NotifyAccess=all
TimeoutStartSec=120
TimeoutStopSec=15

[Install]
WantedBy=multi-user.target

I've noted no error in running docker, any idea?

Thanks in advance

Bad Argument Parsing Destroys "Unrelated" Containers

I've been having trouble with a new setup and I believe I've found a major issue with systemd-docker. I've the following on my unit file:

ExecStart=/opt/bin/systemd-docker --cgroups name=systemd run
--name postgres-ambassador
-v /var/run/docker.sock:/docker.sock
-p 127.0.0.1:5432:5432
--expose 5432
--rm
cpuguy83/docker-grand-ambassador
-name postgres
-sock /docker.sock

As you can see, the container I want to run is named postgres-ambassador and one of the arguments the container receives is "-name postgres". The issue here is that systemd-docker's argument parsing seems to be using "postgres" as the controlled container, instead of the correct "postgres-ambassador".

This is a major issue because it will stop and remove the wrong container. As it is right now, the postgres unit (also launched using systemd-docker) doesn't fail but the container is stopped and destroyed. Trying to restart postgres-ambassador will fail because it's trying to "--rm postgres" and launching "postgres-ambassor" which docker refuses due to a conflicting name. Meanwhile, all links and dependencies to "postgres" will cascade fail.

Restarting containerd does not restart containers

If you have any systemd-docker-managed containers running and restart (or stop and start) dockerd/containerd, the containers are stopped and never restarted. Adding docker.service to the WantedBy= statement in the [Install] section of the container service unit file seems to solve the problem so that may be worth adding to the README.

Apt-get Install in debian

Hi,

I installed the package on Ubuntu 18.04 LTS
(I believe) withh apt-get Install (or only apt Install, dont know)

But in Debian the package is not found (Stretch), even not when googling it.

Is there a way to Install it that way?
Is there a reason that it is not in the repo of debian?

I am trying to use it with bitwarden_rs
dani-garcia/vaultwarden#452

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.