containers / skopeo Goto Github PK
View Code? Open in Web Editor NEWWork with remote images registries - retrieving information, images, signing content
License: Apache License 2.0
Work with remote images registries - retrieving information, images, signing content
License: Apache License 2.0
I've been experimenting with CentOS Atomic Host and Fedora Atomic Host recently, and when looking at system containers and installation using the atomic command, I've hit a snag. These boxes don't get web access except via a proxy, and even though I've set HTTP_PROXY and HTTPS_PROXY environment variables, I'm still not able to pull images. Looking through the python code, I noticed that atomic calls skopeo, and on testing skopeo independently, it seems skopeo is not using the http(s) proxy environment variables.
The command I'm running is:
HTTPS_PROXY=http://wwwcache.keele.ac.uk:8080 HTTP_PROXY=http://wwwcache.keele.ac.uk:8080 skopeo --debug inspect docker://docker.io/fedora
The output I receive is:
DEBU[0254] Ping https://registry-1.docker.io/v2/ err &url.Error{Op:"Get", URL:"https://registry-1.docker.io/v2/", Err:(_net.OpError)(0xc820278b90)}
DEBU[0572] Ping http://registry-1.docker.io/v2/ err &url.Error{Op:"Get", URL:"http://registry-1.docker.io/v2/", Err:(_net.OpError)(0xc820278d70)}
FATA[0572] Get http://registry-1.docker.io/v2/: dial tcp 54.175.214.89:80: getsockopt: connection timed out
Additionally, running tcpdump shows no traffic heading to the web proxy, instead direct connection attempts are made to servers hosted in AWS, which fail.
Should this be working? If not, are there any plans to support operating over a web proxy?
Building skopeo in RHEL7 with go1.4.2 gives:
$ /usr/local/go1.4.2/go/bin/go build -o skopeo ./cmd/skopeo
# github.com/projectatomic/skopeo/signature
src/github.com/projectatomic/skopeo/signature/json.go:68: dec.Token undefined (type *json.Decoder has no field or method Token)
src/github.com/projectatomic/skopeo/signature/json.go:72: undefined: json.Delim
src/github.com/projectatomic/skopeo/signature/json.go:76: dec.Token undefined (type *json.Decoder has no field or method Token)
src/github.com/projectatomic/skopeo/signature/json.go:80: undefined: json.Delim
src/github.com/projectatomic/skopeo/signature/json.go:103: dec.Token undefined (type *json.Decoder has no field or method Token)
The fastest thing we can do to deal with this is to create a file signature/json_go1.4.2.go
(or the like) and put there a build tag to be compiled only with go1.4.2 and lower - and another file to build with go > 1.4.2 (which has the undefined methods above in the errors). We can just move there paranoidUnmarshalJSONObject
so we have two version of this function for the different go versions
.
@mtrmac sounds good to you? is paranoidUnmarshalJSONObject
fine to be reimplemented w/o using Token
and Delim
which aren't available in go1.4.2?
I would like to get the tarsum of docker save'd images so that I can use the same sha256 used in the v2 registry. Is Skopeo the right place to add this functionality? I would like something like `skopeo tarsum /path/to/foo.tar' and that internally uses https://github.com/docker/docker/tree/master/pkg/tarsum
What do you think?
it's been a while since the last release of this was put out. Can we cut a tag and so that a distro (like Fedora) can ship an updated version?
@mtrmac just a tracking question
Wouldn't it be better if ImageDestination return Writers? I don't have anything in mind about how to unify all the image destinations to do so but as we return Readers from ImageSource it came to my mind why don't do the same and return Writers there.
I'm saying, why don't we try and further abstract methods in ImageDestination to support this? is it doable? wdyt?
$ skopeo registry.access.redhat.com/rhel7 | jq '.Config.Labels.Name'
time="2016-03-08T16:31:20+01:00" level=error msg="Error trying v2 registry: Error parsing HTTP response: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//W3C//DTD HTML 3.2 Final//EN\\\">\\n<title>404 Not Found</title>\\n<h1>Not Found</h1>\\n<p>The requested URL was not found on the server.</p><p>If you entered the URL manually please check your spelling and try again.</p>\\n\""
"rhel7/rhel"
command is successful but that debug output is weird and docker.io registry doesn't suffer from this
Repository description isn't accurate anymore since skopeo is about to be able to download image's layers in addition to inspect images on a repository
In many cases, signatures are served from a different server than the registry. We need a way to map this signature server (sigstore?) to a registry.
The capability to use .kubeconfig files is great but we frequently don't have the oc client installed on the system to manage the .kubeconfig file.
This is a request for skopeo to support this so we can reduce the host dependencies.
The atomic type is awkward when using with "copy" workflow. The primary object the user knows about is a docker image, locally or in the registry. What we do with the atomic type is infer the docker image reference. This means we don't have the registry URL or port (see #189).
What seems to make more sense is to drop atomic transport type and use a flag to enable writing to openshift API.
Copy:
skopeo copy --sign-by [email protected] --atomic docker-daemon:registry.example.com/my/image docker://registry.example.com/my/image
Standalone:
skopeo standalone-sign --atomic MANIFEST registry.example.com/my/image KEY-FINGERPRINT
As a registry admin I am able to configure the registry to serve on any port. When using a port other than 5000 I get this error:
"Invalid format of docker reference: expecting port 5000"
Could we discover the port using ~/.docker/config.json
?
As of now, the Atomic CLI must use a subprocess to interact with skopeo. This is not entirely ideal. Among other things, you can see type differences between python2 and 3 when working with the returned information.
DBUS would allow us to "bind" with subprocess, define return types, etc.
DEBU[0019] Uploading mitr/busybox/blobs/uploads/?digest=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
DEBU[0019] POST https://atomic-registry.usersys.redhat.com:5000/v2/mitr/busybox/blobs/uploads/?digest=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
DEBU[0019] Error uploading, status 202
or after s/POST/PUT/
:
DEBU[0019] Uploading mitr/busybox/blobs/uploads/?digest=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
DEBU[0019] PUT https://atomic-registry.usersys.redhat.com:5000/v2/mitr/busybox/blobs/uploads/?digest=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
DEBU[0019] Error uploading, status 404
This is explained by distribution/distribution@2a4deee , the API never existed.
I have no idea how this got through all my testing 😞
Would make handling the json easier.
From kubernetes/kubernetes#26788 (comment)
Would be nice to support such a thing in skopeo itself as part of our types /cc @rhatdan
@mtrmac do you think it makes sense?
I'm doing some work on the above. Comments & questions?
Hi, I'm wondering if anyone here has looked at the signing discussion here:
When trying to download an old style docker v1 image format I get an error:
$ ./skopeo copy docker://dustymabe/fedora20-pdftk:latest oci:fedorapdftk
FATA[0010] Error writing manifest: Unrecognized manifest media type
talking with @runcom I was able to find out that the image format was old and I needed to point to newer v2 images in order to get it to work. would be great if we could add a helpful message for our users on this point.
The OpenShift API ImageSignature object provides storage for useful metadata. Skopeo should write this data if possible.
To clarify to the end user that this information is not verified, populate the SignatureCondition object with the following fields:
skopeo-0.1.11-1.fc23.x86_64
No matter what I do I can only get an error to be output:
# skopeo --debug inspect https://docker.io/fedora
FATA[0000] Please use a fully qualified repository name
Currently RH registry doesn't support V2 APIs and skopeo
does not support V1 APIs either:
$ ./skopeo inspect docker://registry.access.redhat.com/rhel7
FATA[0001] error pinging repository, response code 404
In order for skopeo
to work correctly the following APIs should be supported in the RH registry:
GET /v2/
GET /v2/<name>/tags/list
GET /v2/<name>/manifests/<reference>
GET /v2/<name>/blobs/<digest>
More on each of the above https://docs.docker.com/registry/spec/api/
The word "trust" seems to resonate with docker users. See Docker Content Trust. The skopeo policy file describes trust quite well but its use must be translated for the uninitiated.
I'm not suggesting we do a global search and replace, but in cases where users interact with files or output, "trust" is a better term to use IMO. Specifically, these files:
cmd/skopeo/copy.go
cmd/skopeo/main.go
docs/skopeo.1.md
Makefile
cc @rhatdan
e.g. cmd/skopeo/utils.py
uses c.GlobalBool("tls-verify")
which defaults to false
. Should audit everything, really (not that there is that much to audit, but the above may not be the only place).
it's ugly I know - let's move to something else such as glide
and glide-vc
https://github.com/Masterminds/glide
Various registry configuration items are stored in tool-specific configuration locations:
--insecure-registry
(which can be in /etc/sysconfig/docker
’s INSECURE_REGISTRY
, or OPTIONS
, or overridden in docker.service
); altogether quite difficult to parse; skopeo
does not read any of this, adds its own command-line option.skopeo
(but not docker
?!) also can be only specified on the command line.sigstore
* options considered in #170 , configured in atomic.conf
(which makes it awkward to include the parsing and semantics in containers/image
).Would it make sense to introduce a system-wide configuration file, perhaps /etc/containers/docker-registries.yaml
, which allows per-registry and per-repository configuration, to be used by all of the tools if possible?
--insecure-registry
automatically, no need to patch Docker)OPTIONS
would make it easier to manipulate with tools.x-reverse.domain.name
or whatever)After parseDockerImageName
, we treat digests and tags the same. This seems rather suspicious, when the general syntax is name:tag
but name@digest
; e.g. the CanonicalDockerReference
and IntendedDockerReference
implementations are probably incorrect.
Per #149 (comment) :
But I would prefer if we rewrote these all in markdown, so they could easily be displayed on web sites and it would make it easier for people to contribute.
Hello, I know I can do this:
$ skopeo copy docker://busybox:latest oci:existingemptydirectory
However, can I specify the source as a docker image which is only present locally? Trying it gives me:
$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
amitsaha/rootfs-1 latest bbb478419aa9 7 minutes ago 115.9 MB
$ sudo skopeo copy docker://amitsaha/rootfs-1:latest oci:existingemptydirectory1
FATA[0005] Error reading manifest: error fetching manifest: status code: 401, body: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Name":"amitsaha/rootfs-1","Action":"pull"}]}]}
If this is not yet available, is there any alternative to directly converting a docker image which is not in any supported registry to an OCI bundle? Thanks for any hints!
@mtrmac we need something like https://godoc.org/github.com/docker/distribution/manifest and https://godoc.org/github.com/docker/distribution#RegisterManifestSchema.
https://github.com/docker/distribution/blob/master/manifests.go#L86
This should play well with the manifest
interface also. We really need to abstract manifests code in skopeo.
I think their structure is perfect in our case when dealing with different kind of manifests media types (and marshaling/unmarshaling)
We want to be able to build an image then use skopeo copy to sign and push to a registry. For example:
docker build foo/bar
skopeo copy --sign-by [email protected] localdocker://foo/bar docker://reg.example.com/foo/bar
cc @baude
The outputs of skopeo --help
and skopeo $command --help
are not helpful enough; we need to review them and add the missing documentation.
it's empty right now
With all the new stuff we merged in master the README.md got a bit outdated now. We would need to updated it in order to reflect the current master code with examples. Also, with the upcoming move to libraries it will be far far better to have something easy shown in the main github page.
@mtrmac :)
Currently skopeo is under a freeze
in CAHC:
Trying to unfreeze, I get:
iinstall -m 755 skopeo /builddir/build/BUILDROOT/skopeo-0.1.12.10-0d95328125be9e411b8b4b5f301f2dced5a04d31.5d9911952c34defcf625a69b2ca451345e52e459.el7.centos.x86_64/usr/bin
install: cannot stat 'skopeo': No such file or directory
I looked at this briefly but it seems like the actual binary is ending up in some strange place (WORK=/tmp/go-build43465149
).
#39 fixed build with current RHEL 7, but the updated code is rejected with Go 1.6 (see proglottis/gpgme#5 ).
A hopefully final and fully correct fix was sent upstream as proglottis/gpgme#7 . After we are sure this is the best fix, we should update mtrmac/gpgme (if necessary) and then update the vendored copy in skopeo.
(No action needed right now; filing this now so that it happens eventually.)
skopeo inspect
skopeo pull
...
… to detect build breakage before we merge to master
.
I got as far as #46 , but finishing it would require more time than I can spare right now.
A recent demo of the signing functionality is fairly easy to reuse / explain. Within skopeo
, we will replace it with Go code in #118, so there is no reason to keep it in the git repo right now, but it seems a pity to just throw it away.
The shell script still might be useful as is, perhaps as a part of an introduction/walkthrough documentation in the repository, or as a blog post.
So, I will attach the shell script to this issue. Note that it requires the full --sign-by
and --policy
functionality and the fixtures from #118 (~per my current integrate-all-the-things branch), and it is designed to run inside the skopeo-dev
container built by Makefile
.
In the worst case, this issue can be closed (perhaps even immediately) and the attached shell script will be preserved in the issue.
start-openshift.sh.txt
it's ugly that a library is retrieving credentials from something on the host w/o users noticing. I think we should find a way to abstract credentials and put them directly on ImageSource initialization
This is more of a question: can I use skopeo
to copy an OCI image (directory containing refs/
and blobs/
) to a docker registry?
On a fresh machine install I just needed skopeo to pull some oci images needed for testing and I realized the tool is failing with:
$ skopeo copy docker://busybox oci:busybox
FATA[0000] Error loading verification policy: open /etc/containers/policy.json: no such file or directory
@mtrmac do you think we can relax this somehow? I know I can use default-policy.json
tbh but.. wdyt?
I'm on a Mac trying to fool around with skopeo. Building the Docker image locally takes "for ever". A simple docker run atomicregistry/skopeo
would be great.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.