Giter Club home page Giter Club logo

hocker's Introduction

Welcome!

The hocker package provides a small set of utilities to fetch docker image artifacts from docker registries and produce Nix derivations marrying docker and Nix elegantly:

  • hocker-image for fetching a docker image
  • hocker-layer for fetching a docker image's layers
  • hocker-config for fetching a docker image's configuration JSON
  • hocker-manifest for fetching a docker registry image manifest
  • docker2nix for generating Nix expressions calling the fetchdocker derivations, given a docker registry image manifest

These tools only work with version 2 of the docker registry and docker (>=) v1.10.

The motivation for this tool came from a need to fetch docker image artifacts from a docker registry without the stock docker tooling designed to only work with the docker daemon.

Our use case (and the reason why this package exposes a docker2nix tool) is pulling docker images into a NixOS system's store and loading those images from the store into the docker daemon running on that same system.

We desired this for two critical reasons:

  1. The docker daemon no longer required an internet connection in order to load the docker images
  2. By virtue of fetching the docker images at build-time as opposed to run-time, failures from non-existent images or image tags are caught earlier

We strived to make this tool useful outside of the context of Nix and NixOS, therefore all of these tools are usable without Nix in the workflow.

For high-level documentation of each utility, please refer to the README's in their respective directories (links are in the above list).

Quickstart

Let's first retrieve a docker registry image manifest for the debian:jessie docker image (note that we need the library/ repository prefix because we are pulling from the official debian repository!):

$ hocker-manifest library/debian jessie
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1528,
      "digest": "sha256:054abe38b1e6f863befa4258cbfaf127b1cc9440d2e2e349b15d22e676b591e7"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 52550276,
         "digest": "sha256:cd0a524342efac6edff500c17e625735bbe479c926439b263bbe3c8518a0849c"
      }
   ]
}

Next, we can easily generate a fetchdocker derivation using docker2nix:

$ hocker-manifest library/debian jessie | docker2nix library/debian jessie
{ fetchDockerConfig, fetchDockerLayer, fetchdocker }:
fetchdocker rec {
    name = "debian";
    registry = "https://registry-1.docker.io/v2/";
    repository = "library";
    imageName = "debian";
    tag = "jessie";
    imageConfig = fetchDockerConfig {
      inherit tag registry repository imageName;
      sha256 = "1rwinmvfc8jxn54y7qnj82acrc97y7xcnn22zaz67y76n4wbwjh5";
    };
    imageLayers = let
      layer0 = fetchDockerLayer {
        inherit registry repository imageName;
        layerDigest = "cd0a524342efac6edff500c17e625735bbe479c926439b263bbe3c8518a0849c";
        sha256 = "1744l0c8ag5y7ck9nhr6r5wy9frmaxi7xh80ypgnxb7g891m42nd";
      };
      in [ layer0 ];
  }

Private registries

We developed these tools with private registries in-mind and they currently support three modes of authentication:

  1. Nothing at all (simply do not supply --token or --username and --password)
  2. Bearer token-based authentication, you should retrieve a token and then give it via the --token flag
  3. Basic authentication with --username and --password (most common with nginx proxied registries providing basic auth protection; you should be careful to ensure you're only sending requests to registries exposed via TLS or SSL!)

A caveat to #1 if you do not supply any authentication credential flags and you also do not supply a --registry flag then the tools assume you wish to make a request to the public docker hub registry, in which case they ask for a short-lived authentication token from the registry auth server and then make the request to the public docker hub registry.

How to build

Building (and developing a patch for) this project using cabal is straight-forward if we have Nix installed:

$ nix-shell
[nix-shell:]$ cabal --version
cabal-install version 1.24.0.2
compiled using version 1.24.2.0 of the Cabal library

... cabal and all of the package dependencies will be in the shell environment so that we can then:

[nix-shell:]$ cabal build

Alternatively we can nix-build the project, this is not recommended for development because Nix will not build the project incrementally:

$ nix-build --attr hocker release.nix
these derivations will be built:
  /nix/store/3dwvcm66360fpfqrrc4swp9y4q0jzvh9-hocker-0.1.0.0.drv
building path(s) ‘/nix/store/g16mrfhlmb1z3qkdzr0diaqn2dhl8bv6-hocker-0.1.0.0’
...

hocker's People

Contributors

gabriella439 avatar intractable avatar ixmatus avatar j6carey avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hocker's Issues

Bump hackage release to 1.0.8

Hey, first of all thanks for the project. Really glad it's exists, since the only other way around would be using skopeo right now.

Anyway. I'm currently trying to use hocker for a project and as far as I understand nixpkgs is kept on-par with the latest hackage release via a cronjob. Currently hocker (and therefor fetchWithDocker) is marked as broken in nixpkgs.

NIXPKGS_ALLOW_BROKEN=1 nix build --impure nixpkgs#haskellPackages.hocker

error: builder for '/nix/store/5x0k8ab8yrcs1sw20f0hkngx4ddknwb5-hocker-1.0.6.drv' failed with exit code 1;
       last 10 log lines:
       >    |        ^^^^^^^^^^^^^^^^^^^^^^^^^^...
       > [ 7 of 16] Compiling Network.Wreq.ErrorHandling ( src/Network/Wreq/ErrorHandling.hs, dist/build/Network/Wreq/ErrorHandling.o, dist/build/Network/Wreq/ErrorHandling.dyn_o )
       >
       > src/Network/Wreq/ErrorHandling.hs:24:1: warning: [-Wunused-imports]
       >     The import of ‘Data.Semigroup’ is redundant
       >       except perhaps to import instances from ‘Data.Semigroup’
       >     To import instances alone, use: import Data.Semigroup()
       >    |
       > 24 | import           Data.Semigroup            ((<>))
       >    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       For full logs, run 'nix log /nix/store/5x0k8ab8yrcs1sw20f0hkngx4ddknwb5-hocker-1.0.6.drv'.

As far as I can see it's using a 1.0.6 is a release from 2020, which is the latest on hackage as well. Somehow the 1.0.7 version that's mentioned here #53 doesn't show up

Any occurrence Data.Semigroup was just recently removed and maybe fixed by 14fa8ec which was very recently and might also resolve NixOS/nixpkgs#122263 already?

In addition fetchWithDocker seem to be broken on latest nixos-unstable for me because of NixOS/nixpkgs@cdb39a8#diff-111a0825748a5c3eecc6253c03508fceffb01b67bf279031f5f6cfdd0026126dR38 maybe that can be resolved with an update in the docker2nix output? At least for me in my flake I'm running into this issue right now.

error: undefined variable 'lib'

       at /nix/store/h2mn9hkw60rhlv4fyhz287127dim9r01-source/pkgs/build-support/fetchdocker/credentials.nix:38:3:

           37| in
           38|   lib.optionalString (pathParts != []) ((builtins.head pathParts).path)
             |   ^
           39|
(use '--show-trace' to show detailed location information)

I really hope I'll get everything running, happy to contribute more documentation to the project - I think hocker is still underutilized in the world of nixos. Would allow very easily packaging/running any docker image in nix with systemd-nspawn with no extra complications.

Since we're reaching close to the nixpkgs-23.05 release, maybe there is still time to resolve outstanding issues/bump a release soon?

(cc @ixmatus)

Advice on hnix dependency

I'm trying to use hocker via Nix's fetchFromGitHub and cabal2Nix but seeing error: Package 'hnix-0.5.2' in /nix/store/<hash>-source/pkgs/development/haskell-modules/hackage-packages.nix:<line> is marked as broken, refusing to evaluate.

If I mark it as unbroken, I run into:

[12 of 35] Compiling Nix.Expr.Types.Annotated ( src/Nix/Expr/Types/Annotated.hs, dist/build/Nix/Expr/Types/Annotated.o )
src/Nix/Render.hs:26:17: error:
    • Couldn't match expected type ‘Int’
                  with actual type ‘NonEmpty SourcePos’
    • In the first argument of ‘FancyError’, namely ‘(beg :| [])’
      In the expression:
        FancyError
          (beg :| [])
          (Set.fromList [ErrorFail (show msg) :: ErrorFancy Void])
      In an equation for ‘posAndMsg’:
          posAndMsg beg msg
            = FancyError
                (beg :| [])
                (Set.fromList [ErrorFail (show msg) :: ErrorFancy Void])
   |
26 |     FancyError (beg :| [])
   |                 ^^^^^^^^^

src/Nix/Render.hs:35:28: error:
    • Variable not in scope:
        parseErrorPretty' :: ByteString -> ParseError t0 Void -> [Char]
    • Perhaps you meant ‘parseErrorPretty’ (imported from Text.Megaparsec.Error)
   |
35 |     return $ text $ init $ parseErrorPretty' contents (posAndMsg beg msg)
   |                            ^^^^^^^^^^^^^^^^^
builder for '/nix/store/4si6chlwafprv3fb70pb6sz9f8rk70m3-hnix-0.5.2.drv' failed with exit code 1
cannot build derivation '/nix/store/xvk74c2xb7xrbq8mckcyjsv3qc5j7nvc-hocker-1.0.5.drv': 1 dependencies couldn't be built
error: build of '/nix/store/xvk74c2xb7xrbq8mckcyjsv3qc5j7nvc-hocker-1.0.5.drv' failed

Any suggestions?

The Lib module in the hocker library should be qualified by `Hocker`

It is currently not qualified by anything more specific, it is at the root of the hierarchy. This is not problematic for building the hocker-* executables but it is if a user wants to use any of the functions exposed by the hocker library in their own code. The reasons why this is problematic are (a) import statements for the Lib module aren't clear (via module hierarchy qualification) that the module is from hocker and (b) modules at the root of a shared module hierarchy without a unique qualifier are highly likely to conflict with a module from some other dependency.

hocker-config and hocker-layer fail the response integrity check

hocker-config and hocker-layer both check the Docker-Content-Digest hash to ensure the integrity of the response body. However, because we only accept the image manifest media type this is affected by a regression (#2395) in behavior in which the Docker-Content-Digest header contains a hash value from the image manifest list even when the image manifest is the only media type the client accepts.

For example, hocker-config will fail with this error message:

hocker-config: Response content hash is 8072a54ebb3bc136150e2f2860f00a7bf45f13eeb917cca2430fcd0054c8e51b
and it does not match the addressable content hash
be0cd392e45be79ffeffa6b05338b98ebb16c87b255f48e297ec7f98e123905c

Relax upper bound on turtle dependency.

Latests nixpkgs provides turtle 1.5.7 in its haskell package set, as such hocker doesn't build. I adjusted the version bounds to >= 1.3.0 && < 1.6 via an override and hocker builds and works fine.

Handle image manifest lists media type

We said we accepted the image manifest list media type when requesting manifests for a docker image. Docker distribution recently started honoring that, which breaks hocker-manifest because hocker-manifest doesn't actually know what to do with that result (hence c37f317).

This ticket tracks the work to support the image manifest list media type which provides a list of platform-specific references to image manifests for a docker image.

The Types module in the hocker library should be qualified by `Hocker`

It is currently not qualified by anything more specific, it is at the root of the hierarchy. This is not problematic for building the hocker-* executables but it is if a user wants to use any of the functions or types exposed by the hocker library in their own code. The reasons why this is problematic are (a) import statements for the Types module aren't clear (via module hierarchy qualification) that the module is from hocker and (b) modules at the root of a shared module hierarchy without a unique qualifier are highly likely to conflict with a module from some other dependency.

docker2nix should fetch the manifest for the user

docker2nix currently only accepts a manifest on stdin or as a filesystem path via --manifest. Which works nicely if you have fetch and store the registry manifest separately from generating the Nix code but we can also make this easier to use by making both of those inputs optional and fetching the manifest for the user if they don't provide a manifest.

Is this project dead?

Hi, I pulled this project and tried building hocker from the release.nix but it fails. It looks like the whole project still relies on 18.09 for packages, too.

Is this project being maintained any longer?

Thanks!

Pinned nixpkgs hash mismatch on Nix 2.0?

When trying to build from the checkout of this repo I receive the following error:

error: while evaluating the file '/shared/Code/hocker/nix/17_09.nix':
while evaluating anonymous function at /shared/Code/hocker/nix/fetchNixpkgs.nix:1:1, called from /shared/Code/hocker/nix/17_09.nix:5:3:
while evaluating 'ifThenElse' at /shared/Code/hocker/nix/fetchNixpkgs.nix:8:16, called from /shared/Code/hocker/nix/fetchNixpkgs.nix:12:1:
hash mismatch in file downloaded from 'https://github.com/NixOS/nixpkgs/archive/3389f23412877913b9d22a58dfb241684653d7e9.tar.gz': expected hash 'sha256:1zf05a90d29bpl7j56y20r3kmrl4xkvg7gsfi55n6bb2r0xp2ma5', got 'sha256:0wgm7sk9fca38a50hrsqwz6q79z35gqgb9nw80xz7pfdr4jy9pf8

It seems the hash specified is of the tarball itself rather than the unpacked contents (AFAIK fetchTarball expects the latter).

docker2nix should provide the fetchdocker Nix code (ala how nixfromnpm does it)

With appropriate options for choosing where to install the Nix code or turning off the behavior entirely. This is similar behavior to the nixfromnpm project and I think it's good because it means if Nix code is updated and the generator needs to change, it can be done in lock-step.

This also means, however, that all of the generated Nix code must be completely overridable.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.