We need to find some way to NAT jails to the outside.

Options:

• pf anchor we add rules into

We should have some way to push our own DevFS rulesets into the kernel, as we don't really want to touch system config files. Looking at usr.sbin/devfs/... that should be pretty easily possible by calling the appropriate ioctl's.

Same as #6, but instead of having runtime in <base>/runtime, have it in a md(4) device.

Open questions

• what to do on commit?

Probably the first backing file system we want to support.

Spec

State

All calldown datasets shall be under a single base dataset, here denoted by <base>

• <base>/storage - extract fetched layers here
• <base>/storage/empty@extracted - the empty base dataset
• <base>/storage/<layer_hash>@extracted - extracted dataset including all sub-datasets
• <base>/runtime - clone images here from <base>/storage and start jails on them
• <base>/runtime/<container_id> - root filesystem of a Jail

Operations

Extract

Extracting an image would basically be something along the lines of the following pseudocode:

let lower_snapshot = "<base>/storage/empty@extracted";
for layer in image {
    if !exists("<base>/storage/<layer_hash>@extracted") {
        let layer_dataset = lower_snapshot.clone_into("<base>/storage/<layer_hash>");
        layer.extract_over("<base>/storage/<layer_hash>");
        layer_dataset.snapshot("extracted");
    }
    let current_dataset = "<base>/storage/<layer_hash>@extracted";
}

Start a Jail

// 1. Determine the storage "image" to use as the root filesystem.
let layer_hash = runtime_config.topmost_layer;
let basefs = "<base>/storage/<layer_hash>@extracted";
// 2. Clone it into a new runtime root filesystem
basefs.clone_into("<base>/runtime/<container_id>");
// 3. Set up other mounts
// 4. Start Jail

Commit a Jail

After the Jail is stopped, committing runtime state creates a new storage layer with the current changes.

// 1. Get the runtime path and calculate the hash of the new layer
let runtime_rootfs = "<base>/runtime/<container_id>";
let hash = calculate_layer_hash(runtime_rootfs);
// 2. Snapshot the current runtime state 
let snap = runtime_rootfs.snapshot("extracted");
let storage_layer = snap.clone_into("<base>/storage/<hash>");
// 3a) Promote the cloned storage layer.
//     At the moment the dependency chain is
//         [base layer] -> [runtime]@extracted -> [new layer].
//     After promotion, this is reversed:
//         [base layer] -> [new layer]@extracted -> [runtime].
storage_layer.promote();
storage_layer.snapshot("extracted");
// 3b) If we aren't going to reuse the runtime (e.g. to build another layer), delete it:
runtime_rootfs.destroy();
// 4. Update container config to add layer <hash>

Package

This would be going through all changes between a layer and the layer immediately below it with zfs diff <base>/storage/<base_layer>@extracted <base>/storage/<layer>@extracted or similar, and collect the following in a tarball:

• whiteout list containing deleted files
• changed / added files

This would probably end up being a UFS over ZVOLs

Open questions:

• Should we keep zvols in <base>/storage, or not. We probably have the following options:
  1. Extraction scheme would be similar to #3, but in zvols. Each layer would be its own zvol. We could quickly clone this into <base>/runtime when starting a jail.
  2. Copy files from dataset in <base>/storage into a new zvol in <base>/runtime on start. Copy files back on commit.
  3. Copy files from dataset in <base>/storage into a new zvol in <base>/runtime on first start. Clone that into <base>/runtime. Copy files back into dataset and/or clone zvol back into <base>/runtime on commit.