linuxkit / linuxkit Goto Github PK

A toolkit for building secure, portable and lean operating systems for containers

License: Apache License 2.0

Makefile 4.29% Go 60.21% Shell 17.20% C 3.18% OCaml 6.47% Cap'n Proto 0.22% Python 0.59% HTML 0.02% Standard ML 0.01% Dockerfile 7.82%

linux devops-tools containers cloud linuxkit immutable-infrastructure

linuxkit's Introduction

LinuxKit

LinuxKit, a toolkit for building custom minimal, immutable Linux distributions.

Secure defaults without compromising usability
Everything is replaceable and customisable
Immutable infrastructure applied to building Linux distributions
Completely stateless, but persistent storage can be attached
Easy tooling, with easy iteration
Built with containers, for running containers
Designed to create reproducible builds [WIP]
Designed for building and running clustered applications, including but not limited to container orchestration such as Docker or Kubernetes
Designed from the experience of building Docker Editions, but redesigned as a general-purpose toolkit
Designed to be managed by external tooling, such as Infrakit (renamed to deploykit which has been archived in 2019) or similar tools
Includes a set of longer-term collaborative projects in various stages of development to innovate on kernel and userspace changes, particularly around security

LinuxKit currently supports the x86_64, arm64, and s390x architectures on a variety of platforms, both as virtual machines and baremetal (see below for details).

Subprojects

LinuxKit kubernetes aims to build minimal and immutable Kubernetes images. (previously projects/kubernetes in this repository).
LinuxKit LCOW LinuxKit images and utilities for Microsoft's Linux Containers on Windows.
linux A copy of the Linux stable tree with branches LinuxKit kernels.
virtsock A go library and test utilities for virtio and Hyper-V sockets.
rtf A regression test framework used for the LinuxKit CI tests (and other projects).
homebrew Homebrew packages for the linuxkit tool.

Getting Started

Build the `linuxkit` tool

LinuxKit uses the linuxkit tool for building, pushing and running VM images.

Simple build instructions: use make to build. This will build the tool in bin/. Add this to your PATH or copy it to somewhere in your PATH eg sudo cp bin/* /usr/local/bin/. Or you can use sudo make install.

If you already have go installed you can use go install github.com/linuxkit/linuxkit/src/cmd/linuxkit@latest to install the linuxkit tool.

On MacOS there is a brew tap available. Detailed instructions are at linuxkit/homebrew-linuxkit, the short summary is

brew tap linuxkit/linuxkit
brew install --HEAD linuxkit

Build requirements from source using a container

GNU make
Docker
optionally qemu

For a local build using make local

go
make
go get -u golang.org/x/lint/golint
go get -u github.com/gordonklaus/ineffassign

Building images

Once you have built the tool, use

linuxkit build linuxkit.yml

to build the example configuration. You can also specify different output formats, eg linuxkit build --format raw-bios linuxkit.yml to output a raw BIOS bootable disk image, or linuxkit build --format iso-efi linuxkit.yml to output an EFI bootable ISO image. See linuxkit build -help for more information.

Booting and Testing

You can use linuxkit run <name> or linuxkit run <name>.<format> to execute the image you created with linuxkit build <name>.yml. This will use a suitable backend for your platform or you can choose one, for example VMWare. See linuxkit run --help.

Currently supported platforms are:

Local hypervisors
- Virtualization.Framework (macOS) [x86_64, arm64]
- HyperKit (macOS) [x86_64]
- Hyper-V (Windows) [x86_64]
- qemu (macOS, Linux, Windows) [x86_64, arm64, s390x]
- VMware (macOS, Windows) [x86_64]
Cloud based platforms:
- Amazon Web Services [x86_64]
- Google Cloud [x86_64]
- Microsoft Azure [x86_64]
- OpenStack [x86_64]
- Scaleway [x86_64]
Baremetal:
- packet.net [x86_64, arm64]
- Raspberry Pi Model 3b [arm64]

Running the Tests

The test suite uses rtf To install this you should use make bin/rtf && make install. You will also need to install expect on your system as some tests use it.

To run the test suite:

cd test
rtf -v run -x

This will run the tests and put the results in a the _results directory!

Run control is handled using labels and with pattern matching. To run add a label you may use:

rtf -v -l slow run -x

To run tests that match the pattern linuxkit.examples you would use the following command:

rtf -v run -x linuxkit.examples

Building your own customised image

To customise, copy or modify the linuxkit.yml to your own file.yml or use one of the examples and then run linuxkit build file.yml to generate its specified output. You can run the output with linuxkit run file.

The yaml file specifies a kernel and base init system, a set of containers that are built into the generated image and started at boot time. You can specify the type of artifact to build eg linuxkit build -format vhd linuxkit.yml.

If you want to build your own packages, see this document.

Yaml Specification

The yaml format specifies the image to be built:

kernel specifies a kernel Docker image, containing a kernel and a filesystem tarball, eg containing modules. The example kernels are built from kernel/
init is the base init process Docker image, which is unpacked as the base system, containing init, containerd, runc and a few tools. Built from pkg/init/
onboot are the system containers, executed sequentially in order. They should terminate quickly when done.
services is the system services, which normally run for the whole time the system is up
files are additional files to add to the image

For a more detailed overview of the options see yaml documentation

Architecture and security

There is an overview of the architecture covering how the system works.

There is an overview of the security considerations and direction covering the security design of the system.

Roadmap

This project was extensively reworked from the code we are shipping in Docker Editions, and the result is not yet production quality. The plan is to return to production quality during Q3 2017, and rebase the Docker Editions on this open source project during this quarter. We plan to start making stable releases on this timescale.

This is an open project without fixed judgements, open to the community to set the direction. The guiding principles are:

Security informs design
Infrastructure as code: immutable, manageable with code
Sensible, secure, and well-tested defaults
An open, pluggable platform for diverse use cases
Easy to use and participate in the project
Built with containers, for portability and reproducibility
Run with system containers, for isolation and extensibility
A base for robust products

Development reports

There are monthly development reports summarising the work carried out each month.

Adopters

We maintain an incomplete list of adopters. Please open a PR if you are using LinuxKit in production or in your project, or both.

FAQ

See FAQ.

Released under the Apache 2.0 license.

linuxkit's People

Contributors

Stargazers

Watchers

Forkers

svendowideit subfuzion solidnerd jefflaplante noncreature0714 samoht mihirv0ra hairyhenderson salekseev vdemeester rudymccomb mikebrow miqui quinnlin dave-tucker mor1 kryptblue dhiltgen djs55 fxfactorial richardpen inthecloud247 hendrakwong dmreiland sunilpentapati bobveznat signed8bit imcsk8 seanb4t cestith libby neuroradiology thebsdbox yupengzte laijs menglingwei mylearning2017 papajo zhangxiaoyu-zidif tharanga-abeyseela test3lyn bravechou2009 junjie0510 cypggs rv4devops hj3938 quinndiggity maniacs-ops cofyc isulauncle prateekpandey14 papanikge ellerbrock mato sfc-gh-eraigosa ijc viafreekab mtanlee jbarnette xrodneylee curtiszimmerman nishanthvasudevan olivierh59500 yomimono robbkistler rn davidmr001 rayrc sqleejan yangkf1985 josuesdiaz etsangsplk empia t-koulouris ubaidsayyed54 kosmoit dockernuts samverneck wlmgithub riyazdf shubham0d alexxnica kryndex patel66 bsmr gaocegege barkinet timothywei akinswin magnuss rasouli anonymuse friism wenhuizhang tomwillfixit guoyu07 jai2033shankar firdouskhan pwfoo heehee365

linuxkit's Issues

vendor Go libraries

We have 3 packages that are written in Go, they should all vendor libraries.

9pudc. OK, has no external libraries.
hupper. OK, has no external libraries.
mdnstool. includes hashicorp mdns, should be vendored

Should also include mdns in licenses (MIT).

dns fixup script + dhcp lease renewal?

It occurred to me while fiddling with DHCP and DNS that the dnsfix script which writes 8.8.8.8 into /etc/resolv.conf might have its effects undone if the DHCP lease is renewed and udhcpc reruns the script hook and re-erases the file?

Similarly my careful ordering of eth0 and eth1 in order to make eth1's DNS win might not work if eth0's lease is renewed and it overwrites /etc/resolv.conf.

standalone aufs build

Would be nicer to have standalone aufs rather than patched.

Support a linux security module

Either grsecurity or apparmor I guess, not selinux.

vsock code not cross building for arm

neither proxy nor vsudd build on arm as they currently need to cross build native code. I think at least some of this can be done in pure go to fix this. Also diagnostics, as uses vsock now.

mdns - reply for subdomains

Can we have an option to wildcard access subdomains and reply on those, eg reply on *.docker.local as well as just docker.local?

containerd nofile warning

containerd: low RLIMIT_NOFILE changing to max current=1024 max=4096

Log collection boot mode

In order to get critical logs out of the VM/qcow2 volume from a previously unsuccessful or fateful execution, a special boot mode for log exfiltration would be nice. We could either use a transfused mount or, if that is the failing component, some kind of encoding output sent to the console and then decoded or, if that is too hacky, a raw qcow2 block device without a file system.

AWS support

Boot and bring up docker on AWS.

Most of the Pinata specific functionality will just not run, but this may need tweaking, and we may need to modify the config tool.

cc @dmp42

SSH?

Consider having an sshd service and if so we need to follow the instructions at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/building-shared-amis.html#public-amis-install-credentials to populate it on Amazon.

i386 build

Some interest in 32 bit boot2docker, should be fairly easy now have cross compile support, although this needs some cleanup still

StefanScherer/boot2docker#1

core dump handling

Can we add something like sysctl -w kernel.core_pattern='/var/core_%e.%p' to the early start-up sequence? Or maybe a pattern with a timestamp/sequence number?

This would let us capture core dumps to persistent storage in those rare cases where processes die violently. As a bonus, it would be astounding to have /var (or a subdir) transferred to /var/log on start-up so dumps can be packaged and uploaded with logs.

bad docker config leaves a stray pidfile

Errors in docker config daemon.json eg {"debug":"true"} vs boolean can apparently leave a stray pidfile stopping docker starting up.

Occasional modprobe failure during boot

I've seen this issue in the past and I'm not sure what could cause it. This time, it's reported by Daniel Dunbar in the Docker for Mac forum.

* Configuring host block device ...modprobe: FATAL: Module ext4 not found in directory /lib/modules/4.1.19
modprobe: FATAL: Module ext4 not found in directory /lib/modules/4.1.19
modprobe: FATAL: Module ext4 not found in directory /lib/modules/4.1.19
7  0% [                                           ]8[0KInitializing partitions on /dev/vda...
Creating file systems...
mke2fs 1.42.13 (17-May-2015)
 [ ok ]

kernel nfs support

User request for kernel nfs support. We probably want this for aws (nfsv4) soon, so worth adding.

Testing DNS resolution takes a long time without networking

When using moby without networking, Testing DNS resolution in dnsfix takes a long time while dig times out. It would be nice to skip this step if there is no networking configured.

Standardize on shell script format

Not a huge issue, but the init scripts and system scripts are a bit inconsistent right now so it'd be good to have a format we're at least aiming for. Some use tabs, some use 4 spaces, others use 2, some have then on the next line, and so on. 4 or 8 spaces indent and same-line then would be my preference, but I'm happy to conform to whatever we want.

Investigate splice for transfused

We should investigate whether we can replace the read/write loop in transfused with a call to splice, saving a copy through a user-space buffer. An accompanying benchmark will help to make the argument that the change is worthwhile.

Licensing and Remote host errors

When building the license, there are quite a few errors like the following:

looking for source for: http://dev.alpinelinux.org/archive/alpine-conf/alpine-conf-3.3.2.tar.xz
Connecting to dev.alpinelinux.org (91.220.88.36:80)
wget: can't connect to remote host (91.220.88.36): Connection refused
Connecting to distfiles.alpinelinux.org (91.220.88.29:80)
wget: can't connect to remote host (91.220.88.29): Connection refused
http://dev.alpinelinux.org/archive/alpine-conf/alpine-conf-3.3.2.tar.xz

Same for:

apk-tools-2.6.5.tar.xz
alpine-conf-3.3.2.tar.xz
fuse-2.9.4.tar.gz
LVM2.2.02.138.tgz
mkinitfs-3.0.4.tar.xz

I'm not sure if anything is broken; the output dir has the following in it:

acct-6.6.2               apk-tools-2.6.5          busybox-1.24.1           cryptsetup-1.7.1         fuse-2.9.4               kernel                   libcap-2.24              mtools-4.0.18            openssl-1.0.2g           syslinux-6.03
alpine-baselayout-2.3.2  attr-2.4.47              busybox-initscripts-2.3  curl-7.47.0              gcc-5.3.0                kmod-22                  libssh2-1.6.0            musl-1.1.12              pax-utils-0.9.1          util-linux-2.27.1
alpine-conf-3.3.2        aufs-util                ca-certificates-20160104 e2fsprogs-1.42.13        git-2.6.6                lddtree-1.25             lvm2-2.02.138            openrc-0.19              pcre-8.38                xz-5.2.2
alpine-keys-1.1          bind-9.10.3_p4           chrony-2.2.1             expat-2.1.0              iptables-1.4.21          libc-dev-0.7             mkinitfs-3.0.4           openssh-7.2_p2           strace-4.10              zlib-1.2.8

minimal git install

Docker requires git for ADD functionality, but the default alpine git install is larger than necessary, including lua and ncurses. Can we build a smaller one?

docker 1.11

Initial testing has issues with running off ramdisk: ./usr/local/bin/docker: Error response from daemon: rpc error: code = 2 desc = "oci runtime error: could not synchronise with container process: pivot_root invalid argument".

switch_root errors before OpenRC

switch_root: forcing unmount of /proc
switch_root: failed to mount moving /sys to /mnt/sys: Invalid argument
switch_root: forcing unmount of /sys
switch_root: failed to mount moving /run to /mnt/run: Invalid argument
switch_root: forcing unmount of /run
switch_root: failed to unlink bin: Directory not empty
switch_root: failed to unlink apk: Directory not empty
switch_root: failed to unlink cache: Directory not empty
switch_root: failed to unlink cron: Directory not empty
switch_root: failed to unlink spool: Directory not empty
switch_root: failed to unlink chrony: Directory not empty
switch_root: failed to unlink lib: Directory not empty
switch_root: failed to unlink var: Directory not empty
switch_root: failed to unlink bin: Directory not empty
switch_root: failed to unlink udhcpc: Directory not empty
switch_root: failed to unlink mkinitfs: Directory not empty
switch_root: failed to unlink completions: Directory not empty
switch_root: failed to unlink bash-completion: Directory not empty
switch_root: failed to unlink site-functions: Directory not empty
switch_root: failed to unlink zsh: Directory not empty
switch_root: failed to unlink w: Directory not empty
switch_root: failed to unlink q: Directory not empty
switch_root: failed to unlink s: Directory not empty
switch_root: failed to unlink M: Directory not empty
switch_root: failed to unlink Q: Directory not empty
switch_root: failed to unlink m: Directory not empty
switch_root: failed to unlink 7: Directory not empty
switch_root: failed to unlink b: Directory not empty
switch_root: failed to unlink X: Directory not empty
switch_root: failed to unlink f: Directory not empty
switch_root: failed to unlink o: Directory not empty
switch_root: failed to unlink 2: Directory not empty
switch_root: failed to unlink A: Directory not empty
switch_root: failed to unlink l: Directory not empty
switch_root: failed to unlink i: Directory not empty
switch_root: failed to unlink E: Directory not empty
switch_root: failed to unlink 4: Directory not empty
switch_root: failed to unlink 6: Directory not empty
switch_root: failed to unlink r: Directory not empty
switch_root: failed to unlink n: Directory not empty
switch_root: failed to unlink N: Directory not empty
switch_root: failed to unlink p: Directory not empty
switch_root: failed to unlink k: Directory not empty
switch_root: failed to unlink 9: Directory not empty
switch_root: failed to unlink d: Directory not empty
switch_root: failed to unlink g: Directory not empty
switch_root: failed to unlink P: Directory not empty
switch_root: failed to unlink z: Directory not empty
switch_root: failed to unlink e: Directory not empty
switch_root: failed to unlink a: Directory not empty
switch_root: failed to unlink u: Directory not empty
switch_root: failed to unlink 1: Directory not empty
switch_root: failed to unlink c: Directory not empty
switch_root: failed to unlink j: Directory not empty
switch_root: failed to unlink x: Directory not empty
switch_root: failed to unlink L: Directory not empty
switch_root: failed to unlink 8: Directory not empty
switch_root: failed to unlink h: Directory not empty
switch_root: failed to unlink v: Directory not empty
switch_root: failed to unlink 3: Directory not empty
switch_root: failed to unlink 5: Directory not empty
switch_root: failed to unlink t: Directory not empty
switch_root: failed to unlink terminfo: Directory not empty
switch_root: failed to unlink efi64: Directory not empty
switch_root: failed to unlink com32: Directory not empty
switch_root: failed to unlink dosutil: Directory not empty
switch_root: failed to unlink diag: Directory not empty
switch_root: failed to unlink syslinux: Directory not empty
switch_root: failed to unlink info: Directory not empty
switch_root: failed to unlink hooks: Directory not empty
switch_root: failed to unlink templates: Directory not empty
switch_root: failed to unlink git-core: Directory not empty
switch_root: failed to unlink mozilla: Directory not empty
switch_root: failed to unlink ca-certificates: Directory not empty
switch_root: failed to unlink share: Directory not empty
switch_root: failed to unlink xtables: Directory not empty
switch_root: failed to unlink engines: Directory not empty
switch_root: failed to unlink lib: Directory not empty
switch_root: failed to unlink sbin: Directory not empty
switch_root: failed to unlink mergetools: Directory not empty
switch_root: failed to unlink git-core: Directory not empty
switch_root: failed to unlink libexec: Directory not empty
switch_root: failed to unlink usr: Directory not empty
switch_root: failed to unlink chrony: Directory not empty
switch_root: failed to unlink protected_paths.d: Directory not empty
switch_root: failed to unlink keys: Directory not empty
switch_root: failed to unlink apk: Directory not empty
switch_root: failed to unlink ssh: Directory not empty
switch_root: failed to unlink init.d: Directory not empty
switch_root: failed to unlink crontabs: Directory not empty
switch_root: failed to unlink features.d: Directory not empty
switch_root: failed to unlink mkinitfs: Directory not empty
switch_root: failed to unlink modprobe.d: Directory not empty
switch_root: failed to unlink profile: Directory not empty
switch_root: failed to unlink lvm: Directory not empty
switch_root: failed to unlink network: Directory not empty
switch_root: failed to unlink s: Directory not empty
switch_root: failed to unlink l: Directory not empty
switch_root: failed to unlink r: Directory not empty
switch_root: failed to unlink d: Directory not empty
switch_root: failed to unlink a: Directory not empty
switch_root: failed to unlink x: Directory not empty
switch_root: failed to unlink v: Directory not empty
switch_root: failed to unlink terminfo: Directory not empty
switch_root: failed to unlink PWRF: Directory not empty
switch_root: failed to unlink acpi: Directory not empty
switch_root: failed to unlink local.d: Directory not empty
switch_root: failed to unlink lbu: Directory not empty
switch_root: failed to unlink rules.d: Directory not empty
switch_root: failed to unlink udev: Directory not empty
switch_root: failed to unlink misc: Directory not empty
switch_root: failed to unlink certs: Directory not empty
switch_root: failed to unlink ssl: Directory not empty
switch_root: failed to unlink logrotate.d: Directory not empty
switch_root: failed to unlink kernel-patches: Directory not empty
switch_root: failed to unlink update.d: Directory not empty
switch_root: failed to unlink ca-certificates: Directory not empty
switch_root: failed to unlink profile.d: Directory not empty
switch_root: failed to unlink conf.d: Directory not empty
switch_root: failed to unlink default: Directory not empty
switch_root: failed to unlink shutdown: Directory not empty
switch_root: failed to unlink boot: Directory not empty
switch_root: failed to unlink sysinit: Directory not empty
switch_root: failed to unlink default: Directory not empty
switch_root: failed to unlink runlevels: Directory not empty
switch_root: failed to unlink 15min: Directory not empty
switch_root: failed to unlink periodic: Directory not empty
switch_root: failed to unlink sysctl.d: Directory not empty
switch_root: failed to unlink etc: Directory not empty
switch_root: failed to unlink db: Directory not empty
switch_root: failed to unlink apk: Directory not empty
switch_root: failed to unlink bin: Directory not empty
switch_root: failed to unlink sh: Directory not empty
switch_root: failed to unlink sbin: Directory not empty
switch_root: failed to unlink rc: Directory not empty
switch_root: failed to unlink device-mapper: Directory not empty
switch_root: failed to unlink mdev: Directory not empty
switch_root: failed to unlink lib: Directory not empty
switch_root: failed to unlink mount: Directory not empty
switch_root: failed to unlink run: Directory not empty
switch_root: failed to unlink sbin: Directory not empty
switch_root: failed to unlink dev: Directory not empty

   OpenRC 0.19.368c2a1 is starting up Linux 4.4.6 (x86_64)

 * Mounting /proc ... [ ok ]

add audit logging

For debugging seccomp and related issues.

remove swap?

Not sure having swap is necesary/useful?

btrfs support

Request for btrfs for testing engine. Would also allow DM. Used to be enabled but was slowing boot time, but maybe compiling as modules will work?

Log shuffling + failures cause ever-increasing logfiles and large start-up transfers

There are failure conditions where logs are written to the qcow block device and then concatenated to the host log files when the shared file system comes up. With enough build up of these failures, excessive logs will accrete in the block device, sapping its capacity and introducing lots of duplicate logs on the host. The solution is to move to syslog (#74). Reported via #pinata on slack by @mbentley.

fix cross compile

Some parts still not being cross compiled for arm, eg aufs utils

is writing the 'ip' file a bit racy?

In the CI I'm seeing intermittent failures to talk to the VM caused by a lack of an ip file, even though the moby console looks healthy. I wonder if there's something racy about creation of the ip file?

The CI looks like: (from 3662)

Feb 29 14:05:22 => /Users/distiller/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/ip does not exist yet, sleeping for 1s
Feb 29 14:05:23 => /Users/distiller/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/ip does not exist yet, sleeping for 1s
Feb 29 14:05:24 => /Users/distiller/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/ip does not exist yet, sleeping for 1s
Feb 29 14:06:17 => /Users/distiller/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/ip does not exist yet, sleeping for 1s
--------------------------------------------------------------------------------
ASSERT Feb 29 14:06:18 /Users/distiller/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/ip does not exist after 60 seconds
--------------------------------------------------------------------------------
Error: Alcotest.Check_error("Error Feb 29 14:06:18 /Users/distiller/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/ip does not exist after 60 seconds.")
...
Feb 29 14:06:19 => stdout: ✓ Network connected:           inet addr:192.168.64.2  Bcast:0.0.0.0  Mask:255.255.255.0

Feb 29 14:06:19 => stdout: ✓ Process 9pudc running: /sbin/9pudc -path /Socket -sock /var/run/docker.sock

Feb 29 14:06:19 => stdout: ✓ Process transfused running

Feb 29 14:06:19 => stdout: ✓ Process mdnstool running: /sbin/mdnstool -if eth0 -hostname docker.local.

Feb 29 14:06:19 => stdout: ✓ Process hupper running: /bin/hupper

Feb 29 14:06:19 => stdout: ✓ Process docker running: /usr/bin/docker daemon -p /run/docker.pid -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --config-file /Database/branch/master/ro/com.docker.driver.amd64-linux/etc/docker/daemon.json

Feb 29 14:06:19 => stdout: ✓ Docker daemon working

Feb 29 14:06:19 => stdout:  * Redirecting logs to host ... [ ok ]

Currently the docker init.d script has:

if cat /proc/cmdline | grep -q 'com.docker.driverDir'
then
        DRIVER="$(cat /proc/cmdline | sed -e 's/.*com.docker.driverDir="//' -e 's/".*//')"
        INET=$(ifconfig eth0 2> /dev/null | grep 'inet addr' | cut -f 2 -d ":" | cut -f 1 -d " ")
        [ -d "/Mac/$DRIVER" ] && echo $INET > "/Mac/$DRIVER/ip"
fi

Since docker is running, the init.d script must have run. I can think of the following possibilities:

the 9P filesystem wasn't mounted: however I think since docker depends on 9pudc which depends on 9pinit, I think it can't be this
the IP address isn't available: in this case $INET would be "", which would cause an empty file to be written, which wouldn't cause the ip does not exist yet message.
the -d "/Mac/$DRIVER" test failed: some strange transient issue in com.docker.osxfs
startup is slower than 60s (!)

None of these seem particularly likely :(

userns support broken

the fragile useradd stuff strikes back

time="2016-05-25T22:30:34.666551293Z" level=debug msg="received past containerd event: &types.Event{Type:\"live\", Id:\"\", Status:0x0, Pid:\"\", Timestamp:0x5746278a}" 
time="2016-05-25T22:30:34.670147114Z" level=fatal msg="Error starting daemon: Error during \"dockremap\" user creation: Error adding user \"dockremap\": Failed to add user with error: exit status 1; output: \"-r -s /bin/false dockremap\\n/usr/sbin/adduser: unrecognized option: r\\nBusyBox v1.24.1 (2015-12-16 08:00:02 GMT) multi-call binary.\\n\\nUsage: adduser [OPTIONS] USER [GROUP]\\n\\nCreate new user, or add USER to GROUP\\n\\n\\t-h DIR\\t\\tHome directory\\n\\t-g GECOS\\tGECOS field\\n\\t-s SHELL\\tLogin shell\\n\\t-G GRP\\t\\tAdd user to existing group\\n\\t-S\\t\\tCreate a system user\\n\\t-D\\t\\tDon't assign a password\\n\\t-H\\t\\tDon't create home directory\\n\\t-u UID\\t\\tUser id\\n\\t-k SKEL\\t\\tSkeleton directory (/etc/skel)\\n\""

make docker unable to be killed by oom killer?

Also containerd.

make mobyconfig speak 9p

We shouldnt need to mount a file system to use 9p config.

Current issues:
docker config eg /etc/docker/daemon.json, needs copying to filesystem
hupper getas a file path for a watch, but will probably just merge into mobyconfig.

maybe increase ulimit defaults

See https://github.com/docker/pinata/issues/2253

open source moby

Checklist of what needs to be done before open sourcing (incomplete)

remove old binary builds #6
merge Windows build
clean up arm build #33 #34
testing, CI linked to github
review included pinata code
release schedule
remove daemons used by pinata and move to pinata repo
clean up config so there is a single config API #71

ARM build broken

Probably this is a first step to fix it:

diff --git a/alpine/kernel/Makefile b/alpine/kernel/Makefile
index b287ede..42d48b4 100644
--- a/alpine/kernel/Makefile
+++ b/alpine/kernel/Makefile
@@ -13,8 +13,8 @@ zImage: kernel_config.arm Dockerfile
        docker build --build-arg ARCH=arm -t mobyarmkernel:build .
        docker run --rm mobyarmkernel:build cat /linux/arch/arm/boot/zImage > $@
        docker run --rm mobyarmkernel:build cat /aufs-utils.tar > aufs-utils.tar
-       docker run --rm mobykernel:build cat /kernel-source-info > kernel-source-info
-       docker run --rm mobykernel:build cat /kernel-patches.tar > kernel-patches.tar
+       docker run --rm mobyarmkernel:build cat /kernel-source-info > kernel-source-info
+       docker run --rm mobyarmkernel:build cat /kernel-patches.tar > kernel-patches.tar

With that passing, it fails at:

Step 8 : RUN go install --ldflags '-extldflags "-fno-PIC"'
 ---> Running in 1adc4fe68471
can't load package: package vsudd: no buildable Go source files in /go/src/vsudd

remove binary builds from tree

I will remove these and force push new history once they are not being used by any pinata builds, eg early January.

Nicer transfused startup logging

prepend transfused startup log to runtime log
print critical startup messages to the system console

licensing container fails to download source for some components

See https://gist.github.com/dave-tucker/0e2b62aafd1473c6971ba2adbe4d74f0 for the full list

Incorrect docker errors during moby startup

I see

✗ No docker process
✗ Docker ps failed: Cannot connect to the Docker daemon. Is the docker daemon running on this host?

during moby startup but upon login, docker is running and docker ps succeeds. I think this may be related to offshoring the daemon config and hupping it to reload but it's only a hunch based on previous erroneous error messages I've seen about config reload during startup.

faster shutdown time

See https://github.com/docker/pinata/issues/945

reboot ourself if our config has changed

Currently the host reboots us if config changes, but putting the logic there is odd, especially as only we know which parts of the config require a reboot.

cant login to moby arm any more

it boots but no longer accepts login. It used to work, has been a kernel update and some misc changes. reported by @dave-tucker

Clean up config

Changing to a single tool that provides all config info mobyconfig rather than knowledge of how config is stored scattered everywhere. This is basically a set of keys, athough it also provides files to watch as well which is a bit unusual.

fully transparent binfmt_misc

There is now a linux kernel patch for this (not merged yet)

http://thread.gmane.org/gmane.linux.file-systems/105270

http://lwn.net/SubscriberLink/679308/a7f80e5c6833ad6a/

See https://github.com/docker/pinata/issues/1150

cc @yomimono

Provide licenses and source code as required

Turns out alpine helpfully records all the info needed; commit ids are from https://github.com/alpinelinux/aports

#!/usr/bin/lua

p = {}
f = io.open("/lib/apk/db/installed")

for line in f:lines() do
    if line == "" then
        print(p.P, p.L, p.c)
        p = {}
    else
        k, v = line:match("(%a):(.*)")
        if k then
            p[k] = v
        end
    end
end

more inodes

https://forums.docker.com/t/running-out-of-inodes/8805

shutdown not completing

On docker/pinata@60d94025003e3a3a0c0644fa68df91fc57a53a35 I sent Moby a SIGTERM and it started shutting down, but didn't complete:

docker:~#  * Saving random seed ... [ ok ]
 * Stopping mDNS server ... [ ok ]
 * Stopping busybox cron ... [ ok ]
 * Stopping busybox syslog ... [ ok ]
 * Stopping chronyd ... [ ok ]
Written /Mac//Users/djs/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/ip = 169.254.0.2
 * Stopping docker


docker:~# 
docker:~# w
-ash: w: not found
docker:~# ps uax | grep docker
 1026 root       0:00 /sbin/9pudc -path /Socket -sock /var/run/docker.sock
 1731 root       0:00 {docker} /sbin/openrc-run /etc/init.d/docker --lockfd 15 stop
 1733 root       0:00 {openrc-run.sh} /bin/sh /lib/rc/sh/openrc-run.sh /etc/init.d/docker stop
 1777 root       0:00 grep docker
docker:~#

9pudfuse does not cross compile

Should be able to cross build for arm, etc.

resize partition correctly

currently we are just resizing the filesystem, but the underlying partition is not resized.

syslog

Use syslog (over vsock) rather than remoting over fuse.

clean up adduser hack

useradd, groupadd

Debian, Ubuntu

"useradd is a low level utility for adding users. On Debian, administrators should usually use adduser(8) instead."

http://manpages.ubuntu.com/manpages/trusty/man8/useradd.8.html
http://manpages.ubuntu.com/manpages/trusty/man8/groupadd.8.html

useradd is not interactive, and supports both long and short command options, eg -g and --gid. SUpports getting and setting defaults options with -D. The short options are very similar to many other systems, and the exit codes correspond to HPUX wg 4=uid alread in use and -o not specified.

Redhat, Centos

useradd and groupadd are symlinks to adduser, addgroup. These are basically the same as the Debian, Ubuntu useradd commands, with long and short form options and the multiple return values for scripting.

Busybox

no useradd or groupadd applets.

HPUX

useradd very like Linux one with only short options eg -g, and supports -D to set defaults.
groupadd only supports -g and -o. Provides 10 exit codes for different types of failure.

http://ods.com.ua/win/eng/unix/usail/man/hpux/useradd.1.html
http://ods.com.ua/win/eng/unix/usail/man/hpux/groupadd.1.html

claims: STANDARDS COMPLIANCE: useradd: SVID3 groupadd: SVID3

FreeBSD, NetBSD

short form options only, supports -D to change defaults. Man page does not mention return values.

SVID

System V Interface Definition, Fourth Edition Volume 2
http://www.sco.com/developers/devspecs/vol2.pdf

useradd is specified in here, with short form options, without -D to set defaults.
useradd [-u uid [-o] [-i]] [-g group] [-G group[[,group] . . . ]] [-d dir]
[-s shell] [-c comment] [-m [-k skel_dir] -f inactive] [-e expire]
[-h level [-h level [ . . . ]]] [-v def_level] [-w hd_level] [-a event[, . . . ]]
login

adduser