Try to rollback an invalid partial mount/unmount

Docker-On-Top

Docker volume driver that implements bind-like mounts that use copy-on-write: the containers can write to the mounted volume and the changes are seen by all containers using a volume, but the base host directory remains unchanged.

Usage

After starting the plugin (using one of the methods described below), one can create volumes and mount them to containers like this:

docker volume create --driver docker-on-top VolumeName -o base=/path/to/host/directory/
docker run -v VolumeName:/where/to/mount image:tag

There's also a video demonstration of how plugin works. It is somewhat outdated in terms of the feature set but demonstrates the concept:

Limitations

Docker-on-top requires a UNIX-like operating system with a kernel that provides the overlay filesystem (for example, Linux).
Docker-on-top is based on the overlay filesystem, so it inherits all the limitations of that filesystems, which include peculiarities with hard links behavior and some other stuff but, most importantly, changes to the base directory are not allowed while a volume is mounted: if it happens, quoting the documentation, "the behavior of the overlay is undefined, though it will not result in a crash or deadlock".

While a volume is unmounted (all containers using a volume are stopped or terminated), changes in the volume's base directory are allowed.
Docker-on-top uses the flock system call to synchronize access inside its internal directory (aka "dot root directory"): /var/lib/docker-on-top/. Thus, the filesystem where your /var/lib/ is located must support flock. If you don't know what that is, then your filesystem most likely support it 🙂

Build

To build the plugin, go to the project directory and simply run go build.

Run

The simplest way to run the plugin after it's built is to do

sudo ./docker-on-top

Simple as that! The plugin will create a socket and on the next volume-related command the docker daemon will automatically discover the new plugin.

Run as a systemd service

It might be more convenient to manage the plugin as a systemd service (it also allows to automatically start the plugin together with the docker daemon). To enable this, run the following commands after building the plugin:

sudo cp ./docker-on-top /usr/local/bin/  # Put the executable to the system-wide accessible directory
sudo cp ./docker-on-top.service /etc/systemd/system/  # Create the service file for systemd
sudo systemctl enable docker-on-top.service  # If you want the plugin to be automatically started with docker

That's it. After these actions you can manage the plugin as a systemd service with commands like systemctl start, systemctl stop, etc.

Volatile volumes

(note: volatile volumes have nothing to do with overlayfs's "volatile mount")

Volatile volume, just like the usual one, keeps the changes visible to all the containers using it at the same time but, unlike the usual ones, discards the changes made to it after the last container using the volume is unmounted. Here's an example:

$ # First, use a usual (non-volatile) volume
$ docker volume create --driver docker-on-top usual-volume -o base=/
usual-volume
$ docker run -it --rm -v usual-volume:/dot ubuntu:22.04 bash
root@16762a8ddc71:/# echo 123 > /dot/f
root@16762a8ddc71:/# cat /dot/f
123
root@16762a8ddc71:/# exit
exit
$ docker run -it --rm -v usual-volume:/dot ubuntu:22.04 bash
root@ca7380aa393c:/# cat /dot/f  # Non-volatile volume - changes are preserved on container restart
123
root@ca7380aa393c:/# exit
exit
$ 
$ 
$ # Now, try a volatile volume
$ docker volume create --driver docker-on-top volatile-volume -o base=/ -o volatile=true
volatile-volume
$ docker run -it --rm -v volatile-volume:/dot ubuntu:22.04 bash
root@8cc600cd877f:/# echo 123 > /dot/f
root@8cc600cd877f:/# cat /dot/f
123
root@8cc600cd877f:/# exit
exit
$ docker run -it --rm -v volatile-volume:/dot ubuntu:22.04 bash
root@d4a6d78f5664:/# cat /dot/f  # Volatile volume - changes were discarded when the previous container exited
cat: /dot/f: No such file or directory
root@d4a6d78f5664:/# exit
exit
$

Note that not only exiting a container, but also stopping it currently results in the changes discard (unless another container is using the volume). This behavior may be subject to change.

To avoid accidental data losses, it is recommended to indicate the volume's volatility in its name: when creating a volatile volume, name it accordingly, for instance, add the -volatile suffix for all volatile volumes.

Technical note

Logically, the changes to a volatile volume only exist while there is at least one container using it. However, the changes are physically discarded not when the last container using the volume exits, but when after that a new container is created and wants to use the volume.

Thus, if you messed up with your volatile volumes, you can still recover changes made to a volume before it is mounted to a new container. The modified files can be found in /var/lib/docker-on-top/<volume name>/upper/.

	// A really bad situation!
	// We successfully mounted (`syscall.Mount`) the volume but failed to put information about the container
	// using the volume. In the worst case (if we just created the volume) the following happens:
	// Using the plugin, it is now impossible to unmount the volume (this container is not created, so there's
	// no one to trigger `.Unmount()`) and impossible to remove (the directory mountpoint/ is a mountpoint, so
	// attempting to remove it will fail with `syscall.EBUSY`).
	// It is possible to mount the volume again: a new overlay will be mounted, shadowing the previous one.
	// The new overlay will be possible to unmount but, as the old overlay remains, the Unmount method won't
	// succeed because the attempt to remove mountpoint/ will result in `syscall.EBUSY`.
	//
	// Thus, a human interaction is required.
	//
	// (if it's not us who actually mounted the overlay, then the situation isn't too bad: no new container is
	// started, the error is reported to the end user).
	log.Criticalf("Failed to create active mount file: %v. If no other container was currently "+
	"using the volume, this volume's state is now invalid. A human interaction or a reboot is required",
	err)
	return nil, fmt.Errorf("docker-on-top internal error: failed to create an active mount file: %w. "+
	"The volume is now locked. Make sure that no other container is using the volume, then run "+
	"`unmount %s` to unlock it. Human interaction is required. Please, report this bug",
	err, mountpoint)

	if os.IsExist(err1) {
	log.Warningf("Mountpoint of %s already exists. It might mean that the overlay is already mounted "+
	"but the plugin failed to detect it...", volumeName)
	// A possible thing to do here is try to `os.Remove` mountpoint/ and create it again. In case there's no funny
	// business going on, there's not much difference: either way it will work.
	//
	// Otherwise, this additional action would be beneficial in that it wouldn't let mount an overlay for a volume
	// in an invalid state, on the other hand, if the problem that caused the previous mount's failure is somehow
	// resolved (but the stale overlay from the previous failed Mount is still there), the additional actions will
	// prevent the volume from being used when it could be possible.
	//
	// Because I failed to come up even with a single scenario when such a stale overlay would be left, it's hard
	// for me to argue which way is better. Feel free to share your opinion.
	}

kolayne / docker-on-top Goto Github PK

docker-on-top's Introduction

Docker-On-Top

Usage

Limitations

Build

Run

Run as a systemd service

Volatile volumes

Technical note

docker-on-top's People

Contributors

Stargazers

Watchers

Forkers

docker-on-top's Issues

Recommend Projects

Recommend Topics

Recommend Org