Building skopeo on FreeBSD doesn't work because of errors in storage library.:

vendor/github.com/containers/storage/drivers/driver_freebsd.go:19:38: cannot use &buf (type *unix.Statfs_t) as type *syscall.Statfs_t in argument to syscall.Statfs

and

vendor/github.com/containers/storage/lockfile_unix.go:41:3: cannot use &lockfile literal (type *lockfile) as type Locker in return argument:
	*lockfile does not implement Locker (missing TouchedSince method)

I'll follow-up with pull request that fixes these errors.

I try to separate my "OS partition" and my /var/home from my /srv (really /var/srv) which has things that I can rebuild or redownload from the Internet. Things like my git repositories as well as VM/container images.

Today I have:

ls -alZ /var/lib/docker
lrwxrwxrwx. 1 root root system_u:object_r:container_var_lib_t:s0 13 Nov 23 16:18 /var/lib/docker -> ../srv/docker

Which works fine.

I tried the same with:

ls -alZ /var/lib/containers/storage
lrwxrwxrwx. 1 root root unconfined_u:object_r:container_var_lib_t:s0 28 Mar 28 11:08 /var/lib/containers/storage -> ../../srv/containers-storage

But now I get:

/build-base.sh registry.fedoraproject.org/fedora:27 cgwalters/fdev
+ base=registry.fedoraproject.org/fedora:27
+ prefix=cgwalters/fdev
++ buildah from registry.fedoraproject.org/fedora:27
+ ctr=fedora-working-container-2
+ bldr sed -i /tsflags=nodocs/d /etc/dnf/dnf.conf
+ buildah run fedora-working-container-2 -- sed -i /tsflags=nodocs/d /etc/dnf/dnf.conf
/var/lib/containers/storage/overlay/2f1767021121156895fe050bb92dda17ffba1ea1e8997a396fecde7296cee4d4/merged is not an absolute path or is a symlink

However, a bind mount does work.

I was building CRI-O (which vendors containers/storage) on a Gentoo box without libdevmapper, and the build crashed with:

No package 'devmapper' found

I'd rather not install a new library just to build a package I'll never use (I'd be using the btrfs driver). Folks using the devmapper and not the btrfs driver may feel the same way in reverse, etc. It would be nice if these various dependencies were covered by Go build tags, so we could turn off the drivers we didn't need. I'm fine matching the build tags up with the dependencies (e.g. libdevmapper or nolibdevmapper, depending on which way you want the defaults to run) or matching them with the drivers (e.g. devmapper or nodevmapper). You may also want to namespace them (e.g. containerStorageDevMapper or noContainerStorageDevMapper) depending on whether or not you expect consumers to have other dependencies with similar build tags that the consumers want to toggle independently. I'm fine with any of those choices, as long as it ends up with a way to build CRI-O with out installing libdevmapper ;).

Providing the option ostree_repo in a section where it's not supported ([storage.options.thinpool]) is being silently ignored without reporting neither a warning nor an error.

See issue #196 for an example.

Sorry if I misunderstand intention of this project, looks like it is aiming to implement docker features in a docker-independent way, yet it reuses a lot of docker code.

There were few fixes done do docker graphdrivers year ago, which I can find in a current codebase, which is worrying, how this project is meant to be supported and go forward? If there is no development resources allocated to evolve storage drivers, maybe worth reexporting everything from docker graphdrivers?

Example of missing PRs are :

• moby/moby#23497
• moby/moby#26212

Not sure the various json files must be user friendly so take this lightly. But, as since is super slow (and requires hacks like #121) I would love to switch to something binary-ish.

@rhatdan @nalind is this something we can consider? This will speed things up for sure.

containers/storage identifies images by names (strings). This allows fast lookups but it breaks compatibility with Docker.

For instance, when pulling a newer image via podman-pull or when over-tagging an image via podman-tag, the repository and the tag of the image will be lost and reported as <none> <none>. Docker will only remove the tag but keep the repository data (e..g, alpine <none>).

@nalind and I discussed to keep the storage API as is but to find a way to make it a bit smarter under the hood, for instance, by decomposing the image-string as libpod does.

Cc: @baude @mheon @rhatdan @TomSweeneyRedHat

Refs:

• moby/moby#34670
• moby/moby#35537

when integrating containers/storage in projectatomic/docker it fails with:

.gopath/src/github.com/docker/docker/vendor/github.com/containers/storage/drivers/devmapper/deviceset.go:31:2: no buildable Go source files in /home/amurdaca/go/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/vendor/github.com/containers/storage/storageversion

That should be because the package is imported but doesn't have an autogen build gtag in any file https://github.com/containers/storage/blob/master/storageversion/

I started working on improving the CRI-O docs (see cri-o/cri-o#1747) and am about to reference containers-storage.conf(5) to avoid redundancy among manpages and to have a reference for details about all storage drivers and options.

I think that the containers-storage.conf(5) should be improved to reflect the current state of containers/storage. I am also missing a list of supported drivers with some explanation. Maybe other aspects need to be completed as well? I only have limited knowledge about containers/storage and don't consider myself to be a good candidate working on this issue.

See cri-o/cri-o#1883 for more information, the range of commits which probably broke devmapper in CRI-O is:

# this is the range changed with https://github.com/kubernetes-sigs/cri-o/pull/1814 wihch broke devmapper
git log --oneline --ancestry-path 88d80428f9b146f8f9fe7e2e8cc8688a5aae1a4e..68332c059156eae970a03245cfcd4d717fb66ecd

68332c0 Merge pull request #216 from giuseppe/userns-restore-setuid-setgid
2df72f3 chown: restore SUID and SGID bits
1d49427 Merge pull request #215 from nalind/create-error
2805a43 layerStore.Put(): always check for Create() errors
02db7cb Merge pull request #214 from nalind/hold-locks
3df3c9f Hold the layer store lock while diffing
243c4cd Merge pull request #213 from nalind/rolayer-lock
c8670ef Fix a lock inversion
d0cb010 Merge pull request #212 from zmedico/locker-locked-stub-to-lockfile_windows
587b6cc Implement Locker.Locked() for windows
9fcbb57 Merge pull request #210 from zmedico/lock-sanity-checks-for-save-methods
c7ba574 Add lock sanity checks to Save() methods
17c7d1f Merge pull request #205 from rhatdan/subuid
ad12a70 Add more information in errors about missing uidimappings or gidmappings
956a197 Merge pull request #195 from giuseppe/fuse-overlayfs-shifting
3f55e5a shifting: raise an error if the container needs shifting
06cea37 tests: add test for shifting support
1897396 drivers: inform Mount of the mappings used by the container
140e0b6 Makefile: install ffjson
883ee15 store: use the original image if the driver supports shifting
1e60d8a store: include layerID in the error message
9bb1031 drivers: add new method SupportsShifting() to LayerIDMapUpdater
6480245 overlay: enable support for shifting when FUSE is used
8c814e0 overlay: use full paths to the mount program
9f0144d storage.conf: move ostree_repo and skip_mount_home to the correct section
afdedba Merge pull request #201 from rhatdan/mountopt
2569af9 Fix overlay to handle mountopt properly
7098fdc Merge pull request #176 from rhatdan/mount-options
8b1a0f8 Add default mount options to pass to drivers
cb8d712 Merge pull request #200 from rhatdan/mounted
1538971 Change Mounted to return the number of times mounted
46a8a1c Merge pull request #198 from rhatdan/umount
1075a73 Modify storage to allow callers to determine if a mount point is mounted
124de68 Merge pull request #199 from giuseppe/allow-to-override-conf-file
e933db5 storage: rename fuse_program to mount_program
759aab1 storage: allow to override .fuse_program from the conf file
c4aa7ad store: move configuration file parsing to new function
0ab7541 Merge pull request #197 from TomasTomecek/master
404315a log errors from ApplyUncompressedLayer
90d0a58 Merge pull request #194 from rhatdan/contributing
3a5d1ec Potential contributors need to find the CONTRIBUTOR information.
9cbb6cb Merge pull request #137 from giuseppe/ostree-storage
13af8ce tests: add test for ostree deduplication
d33931a tests: set mtime to the epoch
c7fdad4 ci: install ostree
85bf5a4 Vagrantfile: update to Fedora 28
6c122a2 vagrant: install ostree
f3e7ee3 configuration: new option skip_mount_home
52a3781 vfs: add support for ostree deduplication
0c7cb60 overlay: new option to support ostree deduplication
ffd808a ostree: new package
eca8a17 storage: add ostree files
58f557c Merge pull request #191 from giuseppe/overlay-fuse-program
fc4b862 containers-storage: add new option .fuse_program
51f1f85 Merge pull request #190 from marcov/better-failmsg
d990e2b Improve duplicate name error message on container create

Current configuration example provides the field ostree_repo under section [storage.options.thinpool], but it has to be in section [storage.options] to work.

To make life easier to maintain packages in other distros, it was requested to tag the repo properly.

If archive.Changes() is called and the two layers provided to it are not directly related (e.g. a layer and it's parent or grandparent), Changes() will provide the diff of the layer from its parent as opposed to the diff of the two layers.

This could be problematic in cases such as comparing two images with the same parent in order to determine their differences.

fuse-overlayfs seems to work fine on a ZFS dataset when tested manually yet podman refuses to use it when home directory is set up as a ZFS dataset and fuse-overlayfs is configured to be used in rootless mode. podman insists on using VFS storage backend in that case:

ERRO[0000] 'overlay' is not supported over zfs

What needs to be done to enable that use case?

Unfortunately this library cannot be used by unprivileged users since mounting is not permitted.

As a workaround for use cases where there is no root access a simple driver should be added that works without actually mounting anything (accepting poor performance).

Currently I cannot build a static binary because of the devmapper requirement. My application will never use devicemapper in practice and probably not several of the other options. I would like the ability to filter them.

I can provide the patch for this, just need the go-ahead.

My SSD space is over, so I want to move images to separate disk mapped as /data to /data/podman-images. Only images for now. And if containers won't need SSD performance, then move them to /data/podman-containers too.

Is that possible to implement/configure?

Since we started out as a branch of docker, the graph drivers and test logic frequently use the string "docker" when constructing file names and process titles. If a package that consumes this package is deployed on a system that's also running dockerd, I expect it'll be at least a bit confusing when monitoring processes and examining files.

Background: I'm working on packaging buildah, podman and skopeo, and all necessary depednencies, for Debian.

commit a6ca4fc#diff-4061fcef378a6d912e14e2ce162a1995 vendors in containers/image. This creates a circular dependency because containers/image itself vendors containers/storage, which is unfortunate at best. I wonder whether is this really necessary and could imagine that this indicates issues down the road.

Also, I notice that the vendor.conf specification in both packages specify "master". I wonder why that is. Are both packages so severely intertwined that they cannot reasonably be separated in a dependency hierarchy? If that is the case, I wonder whether it wouldn't make more sense to either wait with packaging until both packages have matured more, or package them in the same source package.

The CI in libpod is suffering some issues and one of them seems to be a growing number of open file descriptors. I am not familiar with the code of containers/storage, so consider this rather a question.

Having looked at the implementation of lockfile I am under the impression that lockfile.Unlock() should also close the fd here: https://github.com/containers/storage/blob/master/lockfile_unix.go#L68

@nalind
Cc: @mheon, @baude

Feature request:

Resulting from this issue I posted on CRI-O: cri-o/cri-o#1727

It would be really nice if checkDevAvailable in https://github.com/containers/storage/blob/932e49f1debc6fbc0e3e8d96f83bdc7d496a0909/drivers/devmapper/device_setup.go could evaluate symlinks. Since nvme device ordering is not guaranteed and I'm already doing the work to match up nvme -> xvd_, it would be really nice if I could use those symlinks for CRI-O's directlvm_device.

vendoring c/storage in docker now gives:

2017/01/08 13:38:01 duplicate symbol _cgoexp_57a08a852b47_DevmapperLogCallback (types 1 and 1) in github.com/docker/docker/pkg/devicemapper and $WORK/github.com/docker/docker/vendor/github.com/containers/storage/pkg/devicemapper.a(_go_.o)
cp: cannot stat 'bundles/latest/dynbinary-daemon/dockerd': No such file or directory

Is this a BUG REPORT or FEATURE REQUEST?:

kind feature

Description

In order to speed up rootless container creation using the vfs storage driver, it could be possible to use reflinks when available, for example over Btrfs or XFS (with reflink enabled format), when creating the container tree. The speed of container creation is one the drawbacks of the vfs driver, since it has to copy the entire tree, with reflinks this could be reduced significantly.

Defaults are nice and all, but we probably need a configuration file that we can consult by default to fill in missing information when a calling application doesn't want to care about locations or which driver to use.

A first guess is a TOML file with each section containing a tuple of the startup parameters (root directory, runtime root, driver name, driver options) that it will match against if some of the parameters aren't specified, so that it could fill in the blanks if it was given only the first of those, and we could hard-code a default for just that one value.

This would make it much easier for external code to dictate the settings for a given storage location, since we're not all that interested in setting that policy here, either.

There is an issue with the store lock that came up while refactoring libkpod to libpod.
https://github.com/containers/storage/blob/master/store.go#L1261 uses stores instead of ristores, thus it is trying to lock a store that has already been locked.

I'm looking for the best way to get in contact with this community and I cannot find that information anywhere in the project. I think it would be meaningful having this information available in the README.md.

Of course, I could just create an issue, but I'm not sure if it's the expected behavior also for generic questions. Even it that's the expected behavior, having that information in the README.md would be helpful for newcomers, like me 😄

Existing openSUSE deployments will run into the issue that /etc/sub{g,u}id are not present as no (base) package added those; useradd and usermod don't create those files. I've fixed this via the shadow package now adding those files so that new installs and new users have proper entries but that doesn't solve the issue for existing users.

In other words, existing users don't have this file and will always run into Podman failing with "open /etc/subuid: no such file or directory". I think Podman should catch those errors and guide the user a bit more, maybe even printing the corresponding section from podman (1).

Tracking-bug for containers/podman#1755

@nalind @mtrmac here's the situation:
when pulling an image by tag we need to always store the digested reference along with the tag (same as docker does)
we don't have a RepoDigests for ImageStatus and we just fill RepoTags with status.Names which just contains tagged references for that image
(guess what) we need RepoDigests slice
digests must not be stored and shown as the ID of an image (see reproducer 1). The image ID in docker is the digest of its config blob which should be unique due to content addressability!
if you pull an image by digest corresponding to a tag different than latest image list reports that it has a :latest tag whether that's not true at all (see reproducer 2)
image ID should be the config blob digest!

--

reproducer 1 (digest shown as image ID, the image ID should be the config digest blob!):

$ crioctl image pull nginx@sha256:4aacdcf186934dcb02f642579314075910f1855590fd3039d8fa4c9f96e48315
$ crioctl image list
ID: 4aacdcf186934dcb02f642579314075910f1855590fd3039d8fa4c9f96e48315
Tag: docker.io/library/nginx:latest

--

reproducer 2 (wrong tag applied to a digested image, there should not be any tag for this image unless someone has previously pulled the same image by tag!):

this is the digest for redis:alpine but redis:latest is shown below, incorrectly

notice how the same issue as reproducer 1 is shown below (ID != digest)

$ crioctl image pull redis@sha256:03789f402b2ecfb98184bf128d180f398f81c63364948ff1454583b02442f73b
$ crioctl image list
ID: 03789f402b2ecfb98184bf128d180f398f81c63364948ff1454583b02442f73b
Tag: docker.io/library/redis:latest

cri-o/cri-o#531

Hey, I'm not sure if 32bit support is a thing for containers/storage so feel free to close if out of scope.

The function ParseIDMap() parses the G/UID triple and then cast the uint32 results to int:

I tested the parser with a 32bit build and got a wrong result because int is not large enough:

func main() {
	m, _ := ParseIDMap([]string{"1:4294967285:0"}, "GID")
	fmt.Printf("%v\n", m)
}

> export GOARCH=386
> go run parser.go
[{1 -11 0}]

I fix would break APIs so I would like to ask you which way we want to go here. :)

I installed a fresh Fedora 26 vm and then did the setup listed in the CRI-O README.md. I then also installed golang 1.9 as the v1.7.6 that comes with Fedroa26 is not suitable. After building and installing CRIO I ran:

# kpod pull  george
"/var/lib/containers/storage" contains several valid graphdrivers: devicemapper, overlay; Please cleanup or explicitly choose storage driver (-s <DRIVER>)
could not create runtime
main.pullCmd
	/root/tsweeney/workspaces/cri-o/src/github.com/kubernetes-incubator/cri-o/_output/src/github.com/kubernetes-incubator/cri-o/cmd/kpod/pull.go:50
github.com/kubernetes-incubator/cri-o/vendor/github.com/urfave/cli.HandleAction
	/root/tsweeney/workspaces/cri-o/src/github.com/kubernetes-incubator/cri-o/_output/src/github.com/kubernetes-incubator/cri-o/vendor/github.com/urfave/cli/app.go:485
github.com/kubernetes-incubator/cri-o/vendor/github.com/urfave/cli.Command.Run
	/root/tsweeney/workspaces/cri-o/src/github.com/kubernetes-incubator/cri-o/_output/src/github.com/kubernetes-incubator/cri-o/vendor/github.com/urfave/cli/command.go:193
github.com/kubernetes-incubator/cri-o/vendor/github.com/urfave/cli.(*App).Run
	/root/tsweeney/workspaces/cri-o/src/github.com/kubernetes-incubator/cri-o/_output/src/github.com/kubernetes-incubator/cri-o/vendor/github.com/urfave/cli/app.go:250
main.main
	/root/tsweeney/workspaces/cri-o/src/github.com/kubernetes-incubator/cri-o/_output/src/github.com/kubernetes-incubator/cri-o/cmd/kpod/main.go:103
runtime.main
	/usr/local/go/src/runtime/proc.go:185
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:2337

However -s is not an option for kpod. Does it need to be added there or do we need to update the error message? Beyond that it would be nice to not see the entire stack dump, only the error.

Based on recent conversations, let's move away from this codebase that is carried from the Docker graphdriver. Ideally the model like stdlib database/sql would be ideal, where the package imported by other libraries is just a "headers only" style of interface/interaction, and the drivers are imported only by the final main.main application.

In doing this, the first/primary use-case will be for overlayfs. The existing implementation of devicemapper will be next, though with a fresh approach using lvm directly rather than re-inventing the wheel as the existing devicemapper graphdriver has done.

Approach

Preliminary review is thinking this model ought to be two interfaces, or exposed functionalities.
One for the notion of an "image storage" (including composed layers, and ephemeral layers (or "containers" which is overused)), and "tree storage" for the file systems themselves.

containers/storage.Store red ink

This is my notes from walking through the places that this behemoth interface is being used or referenced.

n/u - not used; c/i - containers/image; cri-o; c/s - internal use to containers/storage

https://godoc.org/github.com/containers/storage/storage?importers at least the footprint of places this is even used are minimal. Practically no risk of breaking others

n/u:

• GetRunRoot(): n/u (c/s)
• GetGraphOptions(): n/u
• GetGraphDriver(): n/u
• GetLayerStore(): n/u (c/s)
• GetImageStore(): n/u (c/s)
• CreateLayer(): n/u (c/s)
• GetMetadata(): n/u (c/s), (overused name), (this could about driver details) (nothing seems to even read from the Metadata fields on these structs)
• Exists(): n/u (c/s), (overused name)
• Delete(): ... n/u (overused name)
• Wipe(): n/u (c/s)
• Changes(): n/u (c/s)
• DiffSize(): n/u (c/s)
• ApplyDiff(): n/u
• Layers(): n/u (c/s)
• ListContainerBigData(): n/u (c/s)
• GetContainerBigData(): n/u (c/s)
• GetContainerBigDataSize(): n/u
• SetContainerBigData(): n/u (c/s)
• GetImagesByTopLayer(): n/u
• GetContainerByLayer(): n/u
• SetContainerDirectoryFile(): n/u
• SetContainerRunDirectoryFile(): n/u
• GetFromContainerRunDirectory(): n/u
• Lookup(): n/u (c/s)
• Version(): n/u (c/s)

crio only:

• CreateContainer(): cri-o, c/s, (overused name)
• DeleteContainer(): crio, c/s
• Mount(): crio, (c/s) (overused name)
• Unmount(): crio, c/s (overused name)
• Images(): crio, (c/s)
• Containers(): crio, (c/s)
• GetNames(): crio
• GetContainer(): crio
• GetContainerDirectory(): crio
• GetFromContainerDirectory(): crio
• GetContainerRunDirectory(): crio
• Shutdown(): crio

c/i only:

• Diff(): c/i
• GetGraphRoot(): c/i, c/s
• GetGraphDriverName(): c/i
• PutLayer(): c/i, c/s
• CreateImage(): c/i, c/s
• DeleteLayer(): c/i, c/s
• DeleteImage(): c/i (overused name)
• ListImageBigData(): c/i
• GetImageBigData(): c/i
• GetImageBigDataSize(): c/i
• SetImageBigData(): c/i
• GetLayer(): c/i

c/i and crio:

• SetMetadata(): c/i, cri-o, c/s, (overused name), (where as this is just a Metadata string on the Layer, Container, and Image structs inside c/s... this is not so straight forward)
• SetNames(): crio, c/i
• GetImage(): c/i, crio

Thanks to this comment in a test I've discovered that the OSTree deduplication only applies to read-only layers.

The manpages of container-storage don't report any command or option to make use of it, but the option exists for commands create-layer and import-layer.

On my experiments I couldn't find out how to get OSTree deduplication using create-layer --readonly.
With import-layer --readonly works as expected and tested by import-layer-ostree.bats.

Out of the previously mentioned issues I'd say that following tasks would be meaningful:

• Document in the manpage of containers-storage that a read-only layers are needed to get OSTree deduplication.
• Update manpage of create-layer. At least the option --readonly, -r is missing.
• Add a manpage for import-layer.
• Document how to create a read-only layer by using create-layer, or otherwise what's the consequence of create-layer --readonly.

We need to support quota support to limit containers in overlay2 with xfs similar to docker.

https://github.com/moby/moby/blob/8af4db6f002ac907b6ef8610b237879dfcaa5b7a/daemon/graphdriver/overlay2/overlay.go#L202

@nalind @rhatdan

I'm guessing that, at some point, it will become desirable to support newer versions of go, so I'm preemptively opening this issue 😄 The test-failure should be reproducible by executing hack/run_ci_tests.sh locally. In travis, it's done by un-commenting this in .travis.yml:

        - GO_VERSION="stable"
          DISTRO="ubuntu"

I can't quite tell from the output exactly what the problem is, the build appears to complete successfully.
It could easily be caused by something not setup properly in the environment. Here's what travis shows:

travis_log.txt

Any ideas what could be in need of fixing to make this work?

We should be able to support spawning containers with either overlay or devicemapper. This will support use cases where containers ask for a posix compliant filesystem through a k8s annotation via cri-o.

We need efficient way of returning number of inodes and size used by entire storage without having to walk the tree if possible. This is needed for the CRI ImageInfo call.

@sboeuf

Is there a plan to support ZFS as a storage layer? It would be great to use the built in COW features for kubernetes based storage.

Manpage for storage.conf is being built as container-storage.conf.5 (notice the missing s after container), but it should be containers-storage.conf.5 for consistency with the whole project naming.

This issue is based on a discussion from containers/podman#956. The underlying problem is that Podman doesn't print a user-friendly error message when, for instance, trying to run a container with the name of another one.

You may reproduce with the following commands:

$ podman run --name foo alpine echo "foo" > /dev/null
$ podman run --name foo alpine echo "foo" > /dev/null
error creating container storage: that name is already in use

Docker prints a more user-friendly message guiding the user to a potential conflict resolution:

docker: Error response from daemon: Conflict. The container name "/foo" is already in use by container "c6c908895cb656718e15d5313a8e94fc3743e5ed3aa7ceb16ff0c865e8bc6f5c". You have to remove (or rename) that container to be able to reuse that name.

I had a look at https://github.com/containers/storage/blob/master/errors.go and believe that more conflict-related error messages can be improved to make tools using containers/stoarge a bit more user friendly.

The upper exemplary issue, for instance, could be improved by giving a Docker-like error and printing the name of the other container in

I saw this while testing with busybox:1.24 image in crio:

  Normal   Scheduled              44s               default-scheduler   Successfully assigned myshell-85579c8fc4-fc7n9 to 127.0.0.1                                                                                                                           
  Normal   SuccessfulMountVolume  44s               kubelet, 127.0.0.1  MountVolume.SetUp succeeded for volume "default-token-b9gp9"                                                                                                                          
  Normal   Pulled                 7s (x6 over 43s)  kubelet, 127.0.0.1  Container image "busybox:1.24" already present on machine                                                                                                                             
  Warning  Failed                 7s (x6 over 43s)  kubelet, 127.0.0.1  Error: Manifest does not match provided manifest digest sha256:3181b2fff8cbdefa2302a04b1eb2273251749e9320d0083b807d32e9f0e18961   

cc: @runcom @rhatdan

see containers/podman#3000

The storage driver was trying to mount overlay on WSL's lxfs but it was unable to do so:

DEBU[0000] overlay test mount with multiple lowers failed mountfrom re-exec error: exit status 1: output: invalid argument
DEBU[0000] overlay test mount with a single lower failed mountfrom re-exec error: exit status 1: output: invalid argument

Notice that overlay fs does work under WSL:

root@HOST:~# uname -a
Linux HOST 4.4.0-17134-Microsoft #706-Microsoft Mon Apr 01 18:13:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux
root@HOST:~# mount -t overlay -o lowerdir=/tmp/lower,upperdir=/tmp/upper,workdir=/tmp/workdir none /tmp/overlay
root@HOST:~# mount | grep /tmp/overlay
none on /tmp/overlay type overlay (rw,relatime,lowerdir=/tmp/lower,upperdir=/tmp/upper,workdir=/tmp/workdir)

On one hand containerd snapshotters allow for building storage backends for the containerd project, while on the other hand CRI-O uses the containers/storage package to manage container images.

That causes implementation fragmentation (we're seeing divergent filesystem specific code pieces in the 2 projects, maintained by different folks) and I am wondering if we could at least leverage the containers/storage Driver implementation to implement containerd snapshotters. This is an open question that may be irrelevant, and we're open for discussions on how to reduce that fragmentation.

For the record, this is what the snapshotter single Go interface looks like:

type Snapshotter interface {
	// Stat returns the info for an active or committed snapshot by name or
	// key.
	//
	// Should be used for parent resolution, existence checks and to discern
	// the kind of snapshot.
	Stat(ctx context.Context, key string) (Info, error)

	// Update updates the info for a snapshot.
	//
	// Only mutable properties of a snapshot may be updated.
	Update(ctx context.Context, info Info, fieldpaths ...string) (Info, error)

	// Usage returns the resource usage of an active or committed snapshot
	// excluding the usage of parent snapshots.
	//
	// The running time of this call for active snapshots is dependent on
	// implementation, but may be proportional to the size of the resource.
	// Callers should take this into consideration. Implementations should
	// attempt to honer context cancellation and avoid taking locks when making
	// the calculation.
	Usage(ctx context.Context, key string) (Usage, error)

	// Mounts returns the mounts for the active snapshot transaction identified
	// by key. Can be called on an read-write or readonly transaction. This is
	// available only for active snapshots.
	//
	// This can be used to recover mounts after calling View or Prepare.
	Mounts(ctx context.Context, key string) ([]mount.Mount, error)

	// Prepare creates an active snapshot identified by key descending from the
	// provided parent.  The returned mounts can be used to mount the snapshot
	// to capture changes.
	//
	// If a parent is provided, after performing the mounts, the destination
	// will start with the content of the parent. The parent must be a
	// committed snapshot. Changes to the mounted destination will be captured
	// in relation to the parent. The default parent, "", is an empty
	// directory.
	//
	// The changes may be saved to a committed snapshot by calling Commit. When
	// one is done with the transaction, Remove should be called on the key.
	//
	// Multiple calls to Prepare or View with the same key should fail.
	Prepare(ctx context.Context, key, parent string, opts ...Opt) ([]mount.Mount, error)

	// View behaves identically to Prepare except the result may not be
	// committed back to the snapshot snapshotter. View returns a readonly view on
	// the parent, with the active snapshot being tracked by the given key.
	//
	// This method operates identically to Prepare, except that Mounts returned
	// may have the readonly flag set. Any modifications to the underlying
	// filesystem will be ignored. Implementations may perform this in a more
	// efficient manner that differs from what would be attempted with
	// `Prepare`.
	//
	// Commit may not be called on the provided key and will return an error.
	// To collect the resources associated with key, Remove must be called with
	// key as the argument.
	View(ctx context.Context, key, parent string, opts ...Opt) ([]mount.Mount, error)

	// Commit captures the changes between key and its parent into a snapshot
	// identified by name.  The name can then be used with the snapshotter's other
	// methods to create subsequent snapshots.
	//
	// A committed snapshot will be created under name with the parent of the
	// active snapshot.
	//
	// After commit, the snapshot identified by key is removed.
	Commit(ctx context.Context, name, key string, opts ...Opt) error

	// Remove the committed or active snapshot by the provided key.
	//
	// All resources associated with the key will be removed.
	//
	// If the snapshot is a parent of another snapshot, its children must be
	// removed before proceeding.
	Remove(ctx context.Context, key string) error

	// Walk all snapshots in the snapshotter. For each snapshot in the
	// snapshotter, the function will be called.
	Walk(ctx context.Context, fn func(context.Context, Info) error) error

	// Close releases the internal resources.
	//
	// Close is expected to be called on the end of the lifecycle of the snapshotter,
	// but not mandatory.
	//
	// Close returns nil when it is already closed.
	Close() error
}

Notable issues will be managing vendor/ and the context package.

The implementation of Mounted() in pkg/mount doesn't take into account the possibility of the location that it's checking having been mounted, and then another filesystem having been mounted over one of its parent directories. When that happens, the checked location is still listed in /proc/self/mountinfo, but subsequent attempts to manipulate mount flags or unmount it will fail.

The solution that bubblewrap uses looks like a good one: it walks the list of mounts, groups them by parent ID, and prunes out any that are "hidden" by virtue of being a subdirectory (say, /foo/bar) of another mountpoint (/foo) that has the same parent. This follows from knowing that if the subdirectory (/foo/bar) had been mounted later (and therefore, was not hidden), its parent ID would be the ID of the mountpoint of which it is a subdirectory (/foo).

That done, we can start at the mount ID for / (or the last one, if there's more than one), and either gather the full list of mounts by walking its list of mounts and using their IDs to find additional lists of mounts, or search as though we were walking a filesystem, finally either finding an entry for the location that we're checking, or not.

(while migrating nspawn machines into podman containers)

steps

cd /var/lib/machines/mymachine/
tar -cv --acls --selinux --xattrs . | podman import - mymachine

expected

successfully imported image

encountered

panic: value method github.com/containers/libpod/vendor/github.com/containers/image/storage.storageReference.NewImageDestination called using nil *storageReference pointer

goroutine 1 [running]:
github.com/containers/libpod/vendor/github.com/containers/image/storage.(*storageReference).NewImageDestination(0x0, 0x1854fa0, 0xc00003e110, 0xc00002f340, 0xc000431760, 0x14552a0, 0x8, 0x203000)
        <autogenerated>:1 +0xe1
github.com/containers/libpod/vendor/github.com/containers/image/copy.Image(0x1854fa0, 0xc00003e110, 0xc00028cbc0, 0x1861820, 0x0, 0x1861880, 0xc00002e6e0, 0xc0003aefc0, 0x0, 0x0, ...)
        /builddir/build/BUILD/libpod-0ad9b6bf2aea4ba37e8c9738feabec61343ba175/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:141 +0xfd
github.com/containers/libpod/libpod/image.(*Runtime).Import(0xc0001d89e0, 0x1854fa0, 0xc00003e110, 0x7fff92f8d617, 0x10, 0x7fff92f8d628, 0xc, 0x1841f00, 0xc00000e020, 0x0, ...)
        /builddir/build/BUILD/libpod-0ad9b6bf2aea4ba37e8c9738feabec61343ba175/_build/src/github.com/containers/libpod/libpod/image/image.go:926 +0x336
github.com/containers/libpod/libpod.(*Runtime).Import(0xc000330540, 0x1854fa0, 0xc00003e110, 0x7fff92f8d617, 0x10, 0x7fff92f8d628, 0xc, 0x27eeff0, 0x0, 0x0, ...)
        /builddir/build/BUILD/libpod-0ad9b6bf2aea4ba37e8c9738feabec61343ba175/_build/src/github.com/containers/libpod/libpod/runtime_img.go:187 +0x229
github.com/containers/libpod/pkg/adapter.(*LocalRuntime).Import(0xc000431340, 0x1854fa0, 0xc00003e110, 0x7fff92f8d617, 0x10, 0x7fff92f8d628, 0xc, 0x27eeff0, 0x0, 0x0, ...)
        /builddir/build/BUILD/libpod-0ad9b6bf2aea4ba37e8c9738feabec61343ba175/_build/src/github.com/containers/libpod/pkg/adapter/runtime.go:159 +0xde
main.importCmd(0x27ceb80, 0x0, 0x0)
        /builddir/build/BUILD/libpod-0ad9b6bf2aea4ba37e8c9738feabec61343ba175/_build/src/github.com/containers/libpod/cmd/podman/import.go:77 +0x26d
main.glob..func26(0x2628320, 0xc000514260, 0x2, 0x2, 0x0, 0x0)
        /builddir/build/BUILD/libpod-0ad9b6bf2aea4ba37e8c9738feabec61343ba175/_build/src/github.com/containers/libpod/cmd/podman/import.go:26 +0x7f
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x2628320, 0xc00003a0e0, 0x2, 0x2, 0x2628320, 0xc00003a0e0)
        /builddir/build/BUILD/libpod-0ad9b6bf2aea4ba37e8c9738feabec61343ba175/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:762 +0x473
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x262bc20, 0x6, 0x27f0b80, 0xc00018e300)
        /builddir/build/BUILD/libpod-0ad9b6bf2aea4ba37e8c9738feabec61343ba175/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:852 +0x2fd
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(0x262bc20, 0xc00053c370, 0xc0000aa058)
        /builddir/build/BUILD/libpod-0ad9b6bf2aea4ba37e8c9738feabec61343ba175/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
        /builddir/build/BUILD/libpod-0ad9b6bf2aea4ba37e8c9738feabec61343ba175/_build/src/github.com/containers/libpod/cmd/podman/main.go:225 +0x3c

version

[root@knob] /var/lib/machines# podman version
Version:            1.1.2
RemoteAPI Version:  1
Go Version:         go1.11.5
Git Commit:         a95a49d3038462d033f84ac314ec8a3064a99cff
Built:              Tue Mar  5 13:10:31 2019
OS/Arch:            linux/amd64
[root@knob] /var/lib/machines# rpm -q podman
podman-1.1.2-1.git0ad9b6b.fc29.x86_64

This is weird.

on macOS:

vendor/github.com/containers/storage/pkg/archive/changes.go:113:27: undefined: isENOTDIR

#141 adds a caller into the generic code, and a definition to changes_linux.go.

The thing is, I can’t see any directive making changes_linux.go conditional, so AFAICT it should be building on macOS as well, and available to be called.

Here is what I get using #4:

$ make lint
checking lint
cmd/oci-storage/create.go:171::warning: duplicate of cmd/oci-storage/create.go:186-200 (dupl)
cmd/oci-storage/create.go:186::warning: duplicate of cmd/oci-storage/create.go:171-185 (dupl)
cmd/oci-storage/container.go:90::warning: duplicate of cmd/oci-storage/image.go:76-127 (dupl)
cmd/oci-storage/image.go:76::warning: duplicate of cmd/oci-storage/container.go:90-141 (dupl)
cmd/oci-storage/containers.go:1::warning: duplicate of cmd/oci-storage/images.go:1-45 (dupl)
cmd/oci-storage/images.go:1::warning: duplicate of cmd/oci-storage/containers.go:1-45 (dupl)
cmd/oci-storage/delete.go:14::warning: duplicate of cmd/oci-storage/delete.go:44-72 (dupl)
cmd/oci-storage/delete.go:44::warning: duplicate of cmd/oci-storage/delete.go:116-144 (dupl)
cmd/oci-storage/delete.go:116::warning: duplicate of cmd/oci-storage/delete.go:14-42 (dupl)
storage/images.go:268::warning: duplicate of storage/layers.go:501-519 (dupl)
storage/layers.go:501::warning: duplicate of storage/images.go:268-286 (dupl)
pkg/devicemapper/devmapper.go:547::warning: duplicate of pkg/devicemapper/devmapper.go:574-597 (dupl)
pkg/devicemapper/devmapper.go:574::warning: duplicate of pkg/devicemapper/devmapper.go:547-570 (dupl)
storage/store.go:939::warning: duplicate of storage/store.go:1886-1917 (dupl)
storage/store.go:1886::warning: duplicate of storage/store.go:939-970 (dupl)
storage/store.go:1966::warning: duplicate of storage/store.go:2009-2050 (dupl)
storage/store.go:2009::warning: duplicate of storage/store.go:1966-2007 (dupl)
cmd/oci-storage/status.go:1::warning: duplicate of cmd/oci-storage/version.go:1-38 (dupl)
cmd/oci-storage/version.go:1::warning: duplicate of cmd/oci-storage/status.go:1-38 (dupl)
pkg/archive/archive_test.go:820::warning: duplicate of pkg/archive/example_changes.go:82-97 (dupl)
drivers/devmapper/deviceset.go:1737::warning: duplicate of drivers/devmapper/deviceset.go:1770-1800 (dupl)
drivers/devmapper/deviceset.go:1770::warning: duplicate of drivers/devmapper/deviceset.go:1737-1767 (dupl)
make: *** [Makefile:56: lint] Error 1

When running cri-o integration tests on F25 I sometimes get an error that overlay is not enabled in the kernel:

"'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded."

I haven't reproduced this when overlay is manually enabled before running. More details here.

Speaking to @rhatdan he suspects the 'supportsoverlay' function is racing through too quickly: https://github.com/containers/storage/blob/master/drivers/overlay/overlay.go#L219

The overlay driver doesn't have any notion of limiting disk usage, and the devicemapper driver lets us specify the size of the thin pool device, but not much more. We should explore implementing a more uniform (and uniformly enforceable) way of managing disk usage, and of putting upper bounds on how much space a container can use.

CCing @mrunalp, who I'm hoping has better insight as to what consumers of a runtime like cri-o actually expect usage limits to mean.