[RFE] Source and Destination for shared storage about skopeo HOT 12 CLOSED

@runcom Sure, in principle, why not.

We do need to make sure that we don’t duplicate the development work in projectatomic/atomic ; and to the extent extraction already exists there, is there an actual, current or expected, benefit to reimplementing it here?

from skopeo.

We do need to make sure that we don’t duplicate the development work in projectatomic/atomic ; and to the extent extraction already exists there, is there an actual, current or expected, benefit to reimplementing it here?

the benefit would be to move this functionality to containers/image so it will be used by other ppl in other project

from skopeo.

related containers/image issue containers/image#38

from skopeo.

By following discussion in containers/image#38, I am Implementing gluster transport as a start.

from skopeo.

@rhatdan @humblec You are both working on something broadly similar, or at least similar-sounding; are your efforts orthogonal or should they be coordinated?

from skopeo.

@mtrmac My plan is to enable gluster trasnsport for skopeo copy as a start. Atm, I would like to enable default access protocol of gluster which is fuse. This will look exact replica of dir transport. Later I am planning to add more access protocols ( gluster-nfs, libgfapi..etc) of gluster. I am not sure what @rhatdan is working on. If it sound similar, yeah, we need to co-ordinate and move ahead :). I am new to this repo, so may be I am missing something seriously. Thanks for your thoughts.

from skopeo.

I am working on the idea of multiple share storages based on remote file systems. The basic idea is to have a local writable file system for creating and modifying images, but allow source images to be stored on multiple different file systems.

For example you might setup storage on /var/lib/nfs/storage where the storage is shared as a read only mount from a remote NFS Store. I would figure the same would work for gluster. Then these stores could be used by buildah and CRI-O for running containers.

I think we should stick to OverlayFS format for all of these file system storages to keep it simple. Now we could theoretically use a gluster share for the local storage, but this could be risky.

from skopeo.

Hi, would this feature allow for something like below -- efficiently downloading a set of related images so duplicate layers are not re-downloaded? Looking for a better alternative to a series of docker saves.

# Copies a large image built FROM mybaseimage:latest
$ skopeo copy docker://myregistry:5000/myimage:latest dir:/var/lib/latest
# This should not have to re-download the layers for mybaseimage, since downloaded above
$ skopeo copy docker://myregistry:5000/mybaseimage:latest dir:/var/lib/latest

from skopeo.

Looking for a better alternative to a series of docker saves.

# Copies a large image built FROM mybaseimage:latest
$ skopeo copy docker://myregistry:5000/myimage:latest dir:/var/lib/latest
# This should not have to re-download the layers for mybaseimage, since downloaded above
$ skopeo copy docker://myregistry:5000/mybaseimage:latest dir:/var/lib/latest

Independently of the ”shared storage” plans, which may deal with this in a systematic way, the dir: backend does detect the presence of pre-existing layers, so something like this could work:

# Copies a large image built FROM mybaseimage:latest
# Downloads everything
$ skopeo copy docker://myregistry:5000/myimage:latest dir:/var/lib/derived-latest
# Hard-link all existing files into the destination for the other image. Trivially cheap.
$ cp -la /var/lib/derived-latest /var/lib/base-latest
# This should not have to re-download the layers for mybaseimage, since downloaded above
# … and it doesn't
$ skopeo copy docker://myregistry:5000/mybaseimage:latest dir:/var/lib/base-latest

The downside is that /var/lib/base-latest will also contain links to layers which are not used in mybaseimage at all; and of course, it is still the case that dir:$somedirectory can only contain one image. dir: is really a debugging tool, and if you want something with indexes and reference counts, right now it would be most straightforward to set up a full docker/distribution server, and use a direct skopeo copy docker://$yoursource dokcer://$thenewserver.

from skopeo.

@runcom @mtrmac @vrothberg Is this still something we want?

from skopeo.

@rhatdan Wasn’t there some kind of “additional stores” functionality added to c/storage?

Note that the original request for “shared storage” came from you, so do we want this? :)

from skopeo.

Ok we will rely on additional stores for this.

from skopeo.