Giter Club home page Giter Club logo

wave's Introduction

Wave containers

Summary

Wave allows provisioning container images on-demand, removing the need to build and upload them manually to a container registry.

Containers provisioned by Wave can be both disposable, i.e. ephemeral containers only accessible for a short period of time, and regular long-term registry-persisted container images.

Features

  • Authenticate the access to remote container registries;
  • Augment container images i.e. dynamically add one or more container layers to existing images;
  • Build container images on-demand for a given container file (aka Dockerfile);
  • Build container images on-demand based on one or more Conda packages;
  • Build container images on-demand based on one or more Spack packages;
  • Build container images for a specified target platform (currently linux/amd64 and linux/arm64);
  • Push and cache built containers to a user-provided container repository;
  • Build Singularity native containers both using a Singularity spec file, Conda package(s) and Spack package(s);
  • Push Singularity native container images to OCI-compliant registries;

How it works

Container provisioning requests are submitted via the endpoint POST /container-token specifying the target container image or container file. The endpoint return the container name provisioned by Wave that can be accessed by a regular Docker client or any other container client compliant via Docker registry v2 API.

When a disposable container is requested, Wave acts as a proxy server between the Docker client and the target container registry. It instruments the container manifest, adding on-demand the new layers specified in the user submitted request. Then the pull flow is managed by Docker client as in any other container, the base image layers are downloaded from the container registry where the image is stored, while the instrumented layers are downloaded via the Wave service.

Requirements

  • Java 19 or later
  • Linux or macOS
  • Docker engine (for development)
  • Kubernetes cluster (for production)

Get started

  1. Clone the Wave repository from GitHub:

    git clone https://github.com/seqeralabs/wave && cd wave
  2. Define one of more of those environment variable pairs depending the target registry you need to access:

    export DOCKER_USER="<Docker registry user name>"
    export DOCKER_PAT="<Docker registry access token or password>"
    export QUAY_USER="<Quay.io registry user name or password>"
    export QUAY_PAT="<Quay.io registry access token>"
    export AWS_ACCESS_KEY_ID="<AWS ECR registry access key>"
    export AWS_SECRET_ACCESS_KEY="<AWS ECR registry secret key>"
    export AZURECR_USER="<Azure registry user name>"
    export AZURECR_PAT="<Azure registry access token or password>"
  3. Setup a local tunnel to make the Wave service accessible to the Docker client (only needed if you are running on macOS):

    npx localtunnel --port 9090

    Then configure in your environment the following variable using the domain name return by Local tunnel, e.g.

    export WAVE_SERVER_URL="https://sweet-nights-report.loca.lt"
  4. Run the service in your computer:

    bash run.sh
    
  5. Submit a container request to the Wave service using the curl tool

    curl \
        -H "Content-Type: application/json" \
        -X POST $WAVE_SERVER_URL/container-token \
        -d '{"containerImage":"ubuntu:latest"}' \
        | jq -r .targetImage
  6. Pull the container image using the name returned in the previous command, for example

    docker pull sweet-nights-report.loca.lt/wt/617e7da1b37a/library/ubuntu:latest

Note You can use the Wave command line tool instead of curl to interact with the Wave service and submit more complex requests.

Debugging

  • To debug http requests made proxy client add the following Jvm setting:

    '-Djdk.httpclient.HttpClient.log=requests,headers'

Related links

wave's People

Contributors

ewels avatar jason-seqera avatar jimmypoms avatar jordeu avatar jorgeaguileraseqera avatar llewellyn-sl avatar marcodelapierre avatar munishchouhan avatar pditommaso avatar swampie avatar t0randr avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wave's Issues

Mail template loading fails in the prod environment

When the container build completes, the Wave backend sends a notification email. see here

http://github.com/seqeralabs/wave/blob/255f78ca76eecfc559d77a16b20348771fd4aa4b/src/main/groovy/io/seqera/wave/service/builder/ContainerBuildServiceImpl.groovy#L184-L184

This works nicely in the dev env, however, it fails in the prod environment with the following stack trace

22:01:01.480 [default-nioEventLoopGroup-2-2] ERROR i.s.w.s.b.ContainerBuildServiceImpl - == Build failed for image: 195996028523.dkr.ecr.eu-west-1.amazonaws.com/wave/build:5a82843ebbf83b9217385aab8e4c3ab20262c8b89f75c89344642be84ef13eee -- cause: java.lang.IllegalArgumentException: Cannot load notification default template -- check classpath resource: /io/seqera/wave/build-notification.html
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Cannot load notification default template -- check classpath resource: /io/seqera/wave/build-notification.html
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
	at io.seqera.wave.service.builder.ContainerBuildServiceImpl.waitImageBuild(ContainerBuildServiceImpl.groovy:96)
	at io.seqera.wave.controller.RegistryProxyController.handleGet(RegistryProxyController.groovy:63)
	at io.seqera.wave.controller.$RegistryProxyController$Definition$Exec.dispatch(Unknown Source)
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:351)
	at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:583)
	at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303)
	at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111)
	at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103)
	at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659)
	at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49)
	at reactor.core.publisher.Flux.subscribe(Flux.java:8469)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:426)
	at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:371)
	at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onSubscribe(ReactorSubscriber.java:50)
	at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68)
	at reactor.core.publisher.Flux.subscribe(Flux.java:8469)
	at io.micronaut.http.server.netty.RoutingInBoundHandler.handleRouteMatch(RoutingInBoundHandler.java:584)
	at io.micronaut.http.server.netty.RoutingInBoundHandler.channelRead0(RoutingInBoundHandler.java:447)
	at io.micronaut.http.server.netty.RoutingInBoundHandler.channelRead0(RoutingInBoundHandler.java:142)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.micronaut.http.netty.stream.HttpStreamsHandler.channelRead(HttpStreamsHandler.java:225)
	at io.micronaut.http.netty.stream.HttpStreamsServerHandler.channelRead(HttpStreamsServerHandler.java:134)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
	at io.netty.handler.codec.http.websocketx.extensions.WebSocketServerExtensionHandler.channelRead(WebSocketServerExtensionHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
	at io.netty.handler.codec.http.HttpServerKeepAliveHandler.channelRead(HttpServerKeepAliveHandler.java:64)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.flow.FlowControlHandler.dequeue(FlowControlHandler.java:200)
	at io.netty.handler.flow.FlowControlHandler.channelRead(FlowControlHandler.java:162)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.IllegalArgumentException: Cannot load notification default template -- check classpath resource: /io/seqera/wave/build-notification.html
	at io.seqera.wave.mail.MailHelper.getTemplateFile(MailHelper.groovy:30)
	at io.seqera.wave.service.mail.MailServiceImpl.buildCompletionMail(MailServiceImpl.groovy:68)
	at io.seqera.wave.service.mail.MailServiceImpl.sendCompletionMail(MailServiceImpl.groovy:43)
	at io.seqera.wave.service.builder.ContainerBuildServiceImpl$1.call(ContainerBuildServiceImpl.groovy:184)
	at io.seqera.wave.service.builder.ContainerBuildServiceImpl$1.call(ContainerBuildServiceImpl.groovy)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	... 1 common frames omitted

I guess something related to the loading of the template file from the classpath.

http://github.com/seqeralabs/wave/blob/255f78ca76eecfc559d77a16b20348771fd4aa4b/wave-mail/src/main/groovy/io/seqera/wave/mail/MailHelper.groovy#L28-L31

Manifest v1 history looks messed

I've noticed some inconsistency in the history of containers with manifest v1.

For example:

$ docker history quay.io/biocontainers/fastqc:0.11.9--0 
IMAGE          CREATED       CREATED BY                                      SIZE      COMMENT
9d444341a7b2   2 years ago   /bin/sh                                         525MB     
<missing>      5 years ago   /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "/bin/…   0B        
<missing>      5 years ago   /bin/sh -c #(nop) ADD file:8583f81843640f66e…   1.94MB    
<missing>      5 years ago   /bin/sh -c #(nop) MAINTAINER Bjoern Gruening…   0B        
<missing>      6 years ago   /bin/sh -c #(nop) CMD ["/bin/sh"]               0B        
<missing>      6 years ago   /bin/sh -c opkg-cl install http://downloads.…   441kB     
<missing>      6 years ago   /bin/sh -c opkg-cl install http://downloads.…   70.6kB    
<missing>      6 years ago   /bin/sh -c #(nop) ADD file:d01ddbb13c1e847e7…   9.15kB    
<missing>      6 years ago   /bin/sh -c #(nop) ADD file:e2c3819e14cb4b8d2…   103B      
<missing>      6 years ago   /bin/sh -c #(nop) ADD file:1fb1c8c23666e2dc3…   220B      
<missing>      6 years ago   /bin/sh -c #(nop) ADD file:317a8c7f54c369601…   4.27MB    
<missing>      6 years ago   /bin/sh -c #(nop) MAINTAINER Jeff Lindsay <p…   0B

However, in the converted one the sizes looks not aligned with the one above:

$ docker history 9090-seqeralabs-nftowercloud-b59bxddmpit.ws-eu34xl.gitpod.io/tw/of2wc6jonfxs6ytjn5rw63tumfuw4zlsom/fastqc:0.11.9--0
IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
788d6e768c1b   2 years ago     /bin/sh                                         525MB     
<missing>      292 years ago                                                   0B        
<missing>      5 years ago     /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "/bin/…   0B        
<missing>      5 years ago     /bin/sh -c #(nop) ADD file:8583f81843640f66e…   0B        
<missing>      5 years ago     /bin/sh -c #(nop) MAINTAINER Bjoern Gruening…   0B        
<missing>      6 years ago     /bin/sh -c #(nop) CMD ["/bin/sh"]               441kB     
<missing>      6 years ago     /bin/sh -c opkg-cl install http://downloads.…   70.6kB    
<missing>      6 years ago     /bin/sh -c opkg-cl install http://downloads.…   9.15kB    
<missing>      6 years ago     /bin/sh -c #(nop) ADD file:d01ddbb13c1e847e7…   103B      
<missing>      6 years ago     /bin/sh -c #(nop) ADD file:e2c3819e14cb4b8d2…   220B      
<missing>      6 years ago     /bin/sh -c #(nop) ADD file:1fb1c8c23666e2dc3…   4.27MB    
<missing>      6 years ago     /bin/sh -c #(nop) ADD file:317a8c7f54c369601…   0B        
<missing>      6 years ago     /bin/sh -c #(nop) MAINTAINER Jeff Lindsay <p…   23.5MB  

Also, the 292 years ago date looks missing. It could be taken from the later timestamp.

Netty direct memory tunning

Netty uses direct-memory buffers outside the Java heap memory pool that can be quite tricky to tune correctly.

This may requires setting the Java -XX:MaxDirectMemorySize option to reserve enough amount of memory that should be used by the Java app

Other tricky implications are described by in this tweet

https://twitter.com/nitsanw/status/1435997586951921668

The goal of this issue is to investigate the best backend mem setting keeping into consideration a 2GB total memory limit for the container running the registry backend

Improve unit tests for manifest and blob cache

It looks like there are some problems when the storage cache expires

http://github.com/seqeralabs/wave/blob/master/src/main/groovy/io/seqera/wave/storage/MemoryStorage.groovy#L20-L24

For example here if the manist is not found in the cache, it will fallback into this code

http://github.com/seqeralabs/wave/blob/master/src/main/groovy/io/seqera/wave/controller/RegistryProxyController.groovy#L95-L95

Not sure it's correct (if the code is correct the log line Blob pulling from.. etc is wrong).

The goal of this issue is to add some unit tests, that check the pull both for manifest and blobs works correctly when the cache expires

Example in the readme not working

I made

make clean pack
bash run 

then launched the proxy, and run this command:

» docker run --rm reg.ngrok.io/library/busybox cat foo.txt
Unable to find image 'reg.ngrok.io/library/busybox:latest' locally
latest: Pulling from library/busybox
aa5434a6d997: Pull complete 
Digest: sha256:c0f7f7b104aed6bd259749f9e8cdfb3a428c34d9e8a26d109235e4a5533928dd
Status: Downloaded newer image for reg.ngrok.io/library/busybox:latest
cat: can't open 'foo.txt': No such file or directory

The image is pulled but the foo.txt is not in the container.

Unexpected error on K8s

When running with Kubernetes the following error is reported. Likely is caused because K8s is trying to access some internal probes

23:15:16.804 [pool-1-thread-14] ERROR io.seqera.controller.RegHandler - Unexpected error
java.io.IOException: headers already sent
	at jdk.httpserver/sun.net.httpserver.ExchangeImpl.sendResponseHeaders(ExchangeImpl.java:206)
	at jdk.httpserver/sun.net.httpserver.HttpExchangeImpl.sendResponseHeaders(HttpExchangeImpl.java:85)
	at io.seqera.controller.RegHandler.handleResp0(RegHandler.groovy:233)
	at io.seqera.controller.RegHandler.handleNotFound(RegHandler.groovy:246)
	at io.seqera.controller.RegHandler.handleGet(RegHandler.groovy:180)
	at io.seqera.controller.RegHandler.doHandle(RegHandler.groovy:87)
	at io.seqera.controller.RegHandler.handle(RegHandler.groovy:39)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:730)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:699)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Migrate to Micronaut 3.3

Re-fact the proxy server using the following technologies for optimal performance and scaling

  • Rewrite core functionality with Java 11
  • Use Micronaut 3.3 for the proxy server
  • Target Graal native compilation
  • Keep the ProxyClient based on Java 11 HTTP client
  • Keep unit tests with Spock

Allow multi-layers

The layer metadata structure should be re-organised to handle multiple layers.

The current structure is the following

{
  "entrypoint": [
    "/opt/fusion/entry.sh"
  ],
  "workingDir": null,
  "cmd": null,
  "env": [
    "FOO=bar"
  ],
  "append": {
    "location": "pack/layers/layer.tar.gzip",
    "gzipDigest": "sha256:dcf9047433a07424e802675341ad5ce27426d9c4223d795b4ae1814bad5bd907",
    "tarDigest": "sha256:23977e7ebbc433a0d859c73d5552f68932d60b1cbd726d1cee01708310953a01"
  }
}

It should be changed into:

{
  "Entrypoint": [
    "/opt/fusion/entry.sh"
  ],
  "WorkingDir": null,
  "Cmd": null,
  "Env": [
    "FOO=bar"
  ],
  "Layers": [
      {
        "Location": "pack/layers/layer.tar.gzip",
        "GzipDigest": "sha256:dcf9047433a07424e802675341ad5ce27426d9c4223d795b4ae1814bad5bd907",
        "TarDigest": "sha256:23977e7ebbc433a0d859c73d5552f68932d60b1cbd726d1cee01708310953a01"
      },
     {
        "Location": "other/layer.tar.gzip",
        "GzipDigest": "sha256:xyz",
        "TarDigest": "sha256:zyz"
      }      
   ]
}

Note: also the use of capitalised field names in the latter, to better match the convention used by Docker manifest data structure.

create a benchmark

in order to know the capabilities of the service it can be good to have some kind of benchmarks

create some documentation

We need a little documentation about the architecture, API , etc, something better than the README

Error "header parser received no bytes"

Some containers pull return the following error message:

09:39:12.816 [pool-1-thread-848] INFO  io.seqera.controller.RegHandler - Request HEAD - /v2/tw/of2wc6jonfxs6ytjn5rw63tumfuw4zlsom/quast/manifests/5.0.2--py37pl526hb5aa323_2
Mar 05, 2022 9:39:12 AM sun.net.httpserver.ExchangeImpl sendResponseHeaders
WARNING: sendResponseHeaders: being invoked with a content length for a HEAD request
09:39:12.839 [pool-1-thread-848] ERROR io.seqera.controller.RegHandler - Unexpected error
java.io.IOException: HTTP/1.1 header parser received no bytes
	at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:565)
	at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
	at io.seqera.proxy.ProxyClient.head0(ProxyClient.groovy:181)
	at io.seqera.proxy.ProxyClient.head(ProxyClient.groovy:159)
	at io.seqera.proxy.ProxyClient.head(ProxyClient.groovy:155)
	at io.seqera.ContainerScanner.resolve(ContainerScanner.groovy:99)
	at io.seqera.controller.RegHandler.handleManifest(RegHandler.groovy:171)
	at io.seqera.controller.RegHandler.doHandle(RegHandler.groovy:90)
	at io.seqera.controller.RegHandler.handle(RegHandler.groovy:39)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:730)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:699)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: HTTP/1.1 header parser received no bytes
	at java.net.http/jdk.internal.net.http.common.Utils.wrapWithExtraDetail(Utils.java:327)
	at java.net.http/jdk.internal.net.http.Http1Response$HeadersReader.onReadError(Http1Response.java:673)
	at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.checkForErrors(Http1AsyncReceiver.java:297)
	at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:263)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	... 3 common frames omitted
Caused by: java.io.IOException: Connection reset by peer
	at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
	at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:233)
	at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
	at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:356)
	at java.net.http/jdk.internal.net.http.SocketTube.readAvailable(SocketTube.java:1153)
	at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(SocketTube.java:821)
	at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowTask.run(SocketTube.java:175)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
	at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.signalReadable(SocketTube.java:763)
	at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent.signalEvent(SocketTube.java:941)
	at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowEvent.handle(SocketTube.java:245)
	at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:957)
	at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:912)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:912)
09:39:13.083 [pool-1-thread-848] INFO  io.seqera.controller.RegHandler - Request GET - /v2/tw/of2wc6jonfxs6ytjn5rw63tumfuw4zlsom/quast/manifests/5.0.2--py37pl526hb5aa323_2
09:39:13.778 [pool-1-thread-848] INFO  io.seqera.controller.RegHandler - Request GET - /v2/tw/of2wc6jonfxs6ytjn5rw63tumfuw4zlsom/quast/manifests/sha256:3a2a1dc6b53dcc86cec588d28685a7f14a903c6d449e088e75be7c9203c33e84
09:39:14.334 [pool-1-thread-848] INFO  io.seqera.controller.RegHandler - Request GET - /v2/tw/of2wc6jonfxs6ytjn5rw63tumfuw4zlsom/quast/blobs/sha256:4857ca6d930b60d4610559919f11d2ed48099ce5e0f4178892304dfa0df4f11a
09:39:14.335 [pool-1-thread-847] INFO  io.seqera.controller.RegHandler - Request GET - /v2/tw/of2wc6jonfxs6ytjn5rw63tumfuw4zlsom/quast/blobs/sha256:43d7b71dfd436cddc8dd4df827dee29f46f000fe35bddabe661f14e4df45c09f
09:39:14.339 [pool-1-thread-846] INFO  io.seqera.controller.RegHandler - Request GET - /v2/tw/of2wc6jonfxs6ytjn5rw63tumfuw4zlsom/quast/blobs/sha256:12fe7046e2c5e38c781d69d3e77e4bc5ab4d95259ebfefb86e0801f5874af7d0
09:39:14.341 [pool-1-thread-849] INFO  io.seqera.controller.RegHandler - Request GET - /v2/tw/of2wc6jonfxs6ytjn5rw63tumfuw4zlsom/quast/blobs/sha256:e6bcdfebf6a068565be0a4f6c4dbdc5f3c4b37bee8a51514663276f8f7514c78

The problem can be replicated using this command:

docker run --rm \
   --platform linux/amd64 \
   -e AWS_REGION=eu-west-1 \
   -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
   -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
   -e NXF_FUSION_BUCKETS=s3://nextflow-ci \
   --privileged  \
   -it \
   reg.staging-tower.xyz/tw/of2wc6jonfxs6ytjn5rw63tumfuw4zlsom/quast:5.0.2--py37pl526hb5aa323_2 

The target container is

quay.io/biocontainers/quast:5.0.2--py37pl526hb5aa323_2

Refactor the v2 controller with independent methods

The V2 controller currently uses a single method to implement the 3 main endpoints:

  • HEAD /v2/<name>/manifests/<reference>
  • GET /v2/<name>/manifests/<reference>
  • GET /v2/<name>/blobs/<digest>

The goal of this issue is to refactor it into 3 separate methods to simplify the implementation and make it more predictable

OOM error while pulling container

The following error is reported in the log file

09:40:06.839 [pool-1-thread-846] ERROR io.seqera.controller.RegHandler - Unexpected error
java.io.IOException: closed
	at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.current(ResponseSubscribers.java:352)
	at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.read(ResponseSubscribers.java:403)
	at java.base/java.io.InputStream.transferTo(InputStream.java:704)
	at io.seqera.controller.RegHandler.handleReply(RegHandler.groovy:125)
	at io.seqera.controller.RegHandler.handleProxy(RegHandler.groovy:107)
	at io.seqera.controller.RegHandler.handleGet(RegHandler.groovy:199)
	at io.seqera.controller.RegHandler.doHandle(RegHandler.groovy:95)
	at io.seqera.controller.RegHandler.handle(RegHandler.groovy:39)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:730)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:699)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:937)
	at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:491)
	at java.base/javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:779)
	at java.base/javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:730)
	at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2497)
	at java.base/sun.security.ssl.SSLCipher$T12GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1655)
	at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240)
	at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197)
	at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
	at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681)
	at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433)
	at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637)
	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader.unwrapBuffer(SSLFlowDelegate.java:481)
	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(SSLFlowDelegate.java:392)
	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(SSLFlowDelegate.java:264)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	... 3 common frames omitted

Allow supporting third party container registries

The proxy registry should be able to pull images from any third party container registry implementing the Docker APIs.

The target registry name should be parametrised instead of being defined in the codebase

https://github.com/seqeralabs/tower-reg/blob/eb412d35af4be4afa7268f338878b4719c205cb3/src/main/groovy/io/seqera/ProxyClient.groovy#L28-L28

The main goal is being about to pull images from quay.io service. Other can be services (e.g. AWS ECR, Azure registry, etc) can be added later.

How dev environment is determined?

I'm not understanding why the server starts in dev mode. AFAIK micronaut does not use a default env if not specified otherwise.

21:22:58.416 [main] INFO  i.m.context.env.DefaultEnvironment - Established active environments: [dev]

externalize build info

I think it's not a good idea to have the build info stored in a source that is under the control version

task buildInfo { doLast {
    def info = """\
                name=${project.name}
                group=${project.group}
                version=${version}
                commitId=${project.property('commitId')}
            """.stripIndent().toString()
    def f = file("src/main/resources/META-INF/build-info.properties")
    f.parentFile.mkdirs()
    f.text = info
} }

This task must write in the build/generated directory (or build/docker) the build info file to avoid versioning it by mistake

refactor ConfigurableAuthProvider

Refactor ConfigurableAuthProvider

  • use Micronaut HttpClient
  • remove the authUrl and service properties and extract from the '/v2' endpoints as LoginValidator is doing right now

Add support Docker image config version 2, schema 1

While performing some tests it was found that some container collections hold images using an old image manifest format described in this document.

When returning the above image manifest the uses use the content-type application/vnd.docker.distribution.manifest.v1+prettyjws which the ContainerScanner is not able to handle and therefore serving those images it fails here

https://github.com/seqeralabs/tower-reg/blob/4f46e70d0ea98ca1a2c5aa96b1865b77f7894d82/src/main/groovy/io/seqera/ContainerScanner.groovy#L163-L165

Although this manifest has been deprecated, the support for it is critical because exists a lot of images with using it in popular collections such as Biocontianers. Therefore users using this service would not be able to use it.

The problem

In principle, it should be possible to use the similar approach used for other manifests, to modify on-fly the manifest, compute a new digest and return it.

However, this format is made more complex because it uses a cryptographic signature that docker verify before continuing the pull of the constant. Therefore any change (even just reformatting it) will cause an error when pulling the container.

Solutions

  • Just remove the signature and change the response type (already tried, apparently docker continue to check it)
  • Strip the original signature, modify the manifest and re-compute the signature. This could be done using the Docker trustlib, but the implementation could be challenging. More details at this link https://github.com/seqeralabs/docker-signature-tool
  • Transform the manifest v2, schema 1 to a the new manifest format already managed by the ContainerScanner. In should be investigated:
    • Exists any tool for this conversion (just the manifest, not all the image binary conversion)
    • Write a conversion tool from scratch.

Tasks

Add compatibility with other clients

Singularity and Podman do not request a HEAD before a GET of the manifest.

This PR #48 partially solves the compatibility problem.

Singularity is fully working.

Podman is able to retrieve the layers but then is reporting a blob size: mismatch error.

$ podman run -it --rm b718-31-221-139-238.ngrok.io/library/debian:sid-slim /bin/bash
Trying to pull b718-31-221-139-238.ngrok.io/library/debian:sid-slim...
Getting image source signatures
Copying blob 2ae18af4a305 done  
Copying blob 877816e5da8c done  
Copying config 96372189bf done  
Error: writing blob: blob size mismatch

Add caching layer

The goal of this issue is to cache the layer blob binary into bucket storage to prevent pulling the same binary from the remote host multiple times, and therefore optimise bandwidth, speed and cost.

This is expected to be an opt-in feature, therefore if no bucket is provided, the cache can be skipped.

Since the pull of a container layer can result in the transfer of a large payload, the ideal solution consists of implementing the save into the cache in a parallel manner, using the same stream connection, ie. while the stream is being transferred from the remote host, is copied both the to sources request request and to saved into the target bucket.

Add support for MySQL DB connection

The support for Tower-provided credentials was implemented as part of #70.

This has been implemented via Micronaut data for JDBC, however, it currently only supports H2 test database.

The goal of this issue is to add also the support for MySQL database. This requires adding a MySQL dialect specialisation for those DAOs

Ideally, those DAOs should be annotated with @Requires(env='mysql') (and the H2 ones with env='h2') to avoid them conflicting each other and allow the configuration of the target database via a MN environment.

Improve error reporting when matching registry is not found

11:38:30.638 [default-nioEventLoopGroup-4-3] INFO  io.seqera.docker.ContainerScanner - Image pditommaso/nf-launcher:j11-22.03.0-edge_1 => digest=Optional.empty
11:38:30.639 [default-nioEventLoopGroup-4-3] INFO  io.seqera.controller.V2Controller - Unexpected response statusCode: 404
io.seqera.proxy.InvalidResponseException: Unexpected response statusCode: 404
	at io.seqera.docker.ContainerScanner.resolve(ContainerScanner.groovy:105)
	at io.seqera.docker.ContainerService.handleManifest(ContainerService.groovy:71)
	at io.seqera.controller.V2Controller.manifestForPath(V2Controller.groovy:96)
	at io.seqera.controller.V2Controller.handleGet(V2Controller.groovy:65)
	at io.seqera.controller.$V2Controller$Definition$Exec.dispatch(Unknown Source)
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:351)
	at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:583)
	at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303)
	at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111)
	at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103)
	at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659)
	at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49)
	at reactor.core.publisher.Flux.subscribe(Flux.java:8469)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:426)
	at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:371)
	at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onSubscribe(ReactorSubscriber.java:50)
	at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68)
	at reactor.core.publisher.Flux.subscribe(Flux.java:8469)
	at io.micronaut.http.server.netty.RoutingInBoundHandler.handleRouteMatch(RoutingInBoundHandler.java:584)
	at io.micronaut.http.server.netty.RoutingInBoundHandler.channelRead0(RoutingInBoundHandler.java:447)
	at io.micronaut.http.server.netty.RoutingInBoundHandler.channelRead0(RoutingInBoundHandler.java:142)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.micronaut.http.netty.stream.HttpStreamsHandler.channelRead(HttpStreamsHandler.java:225)
	at io.micronaut.http.netty.stream.HttpStreamsServerHandler.channelRead(HttpStreamsServerHandler.java:134)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
	at io.netty.handler.codec.http.websocketx.extensions.WebSocketServerExtensionHandler.channelRead(WebSocketServerExtensionHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
	at io.netty.handler.codec.http.HttpServerKeepAliveHandler.channelRead(HttpServerKeepAliveHandler.java:64)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.flow.FlowControlHandler.dequeue(FlowControlHandler.java:200)
	at io.netty.handler.flow.FlowControlHandler.channelRead(FlowControlHandler.java:162)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)

Unable to load region from system settings exception

The support for AWS params store implemented by 29e2a89 causes the following error when running in a non-AWS environment.

The error can be avoid setting the AWS_REGION variable in the launching environent. However this is not the desired beharior. Micronaut should not try to load the AWS configuration client if the ec2 environment is not specified/available.

For this reason the boostrap file has been named bootstreap-ec2.yml, however it looks MN tries to load the client in anycase


Caused by: io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type [software.amazon.awssdk.services.servicediscovery.ServiceDiscoveryAsyncClientBuilder]: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.AwsRegionProviderChain@1150d471: [software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@70029d2d: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@70029d2d: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@6413d7e7: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@68e02f33: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@3c952a33: Unable to contact EC2 metadata service.]]
        at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2365)
        at io.micronaut.context.DefaultBeanContext.createAndRegisterSingletonInternal(DefaultBeanContext.java:3281)
        at io.micronaut.context.DefaultBeanContext.loadContextScopeBean(DefaultBeanContext.java:2664)
        at io.micronaut.context.DefaultBeanContext.initializeContext(DefaultBeanContext.java:1932)
        ... 7 common frames omitted
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.AwsRegionProviderChain@1150d471: [software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@70029d2d: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@70029d2d: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@6413d7e7: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@68e02f33: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@3c952a33: Unable to contact EC2 metadata service.]]

Java heap space issue

When pulling many containers the server runs out quickly of memory with the following exception

Apr-03 22:39:35.952 [default-nioEventLoopGroup-4-2] ERROR i.m.http.server.RouteExecutor - Unexpected error occurred: Java heap space
java.lang.OutOfMemoryError: Java heap space
Apr-03 22:39:50.959 [default-nioEventLoopGroup-4-2] INFO  io.seqera.controller.V2Controller - > Request [HEAD] /v2/tw/of2wc6jonfxs6ytjn5rw63tumfuw4zlsom/bedtools/manifests/2.30.0--hc088bd4_0
Apr-03 22:40:47.750 [default-nioEventLoopGroup-4-2] INFO  io.seqera.controller.V2Controller - Java heap space
java.lang.OutOfMemoryError: Java heap space
Apr-03 22:40:51.842 [default-nioEventLoopGroup-4-3] INFO  io.seqera.controller.V2Controller - > Request [GET] /v2/tw/of2wc6jonfxs6ytjn5rw63tumfuw4zlsom/bioconductor-summarizedexperiment/blobs/sha256:7d0254e3aa18c2f2d5e67e20409f866080b3d2e77638d9cfeb892c1bdb07baaf
Apr-03 22:44:54.105 [default-nioEventLoopGroup-4-3] WARN  i.n.u.c.SingleThreadEventExecutor - An event executor terminated with non-empty task queue (154)
Apr-03 22:44:54.106 [default-nioEventLoopGroup-4-2] WARN  i.n.u.c.AbstractEventExecutor - A task raised an exception. Task: io.netty.channel.AbstractChannel$AbstractUnsafe$1@7b351b1d
java.lang.OutOfMemoryError: Java heap space
Apr-03 22:44:55.091 [default-nioEventLoopGroup-4-2] WARN  io.netty.channel.nio.NioEventLoop - Unexpected exception in the selector loop.
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
	at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:120)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:177)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)
Apr-03 22:45:05.336 [default-nioEventLoopGroup-4-2] INFO  io.seqera.controller.V2Controller - > Request [GET] /v2/tw/of2wc6jonfxs6ytjn5rw63tumfuw4zlsom/bioconductor-summarizedexperiment/blobs/sha256:77c6c00e8b61bb628567c060b85690b0b0561bb37d8ad3f3792877bddcfe2500
Apr-03 22:45:49.522 [default-nioEventLoopGroup-4-2] DEBUG io.seqera.storage.MemoryStorage - getBlob /v2/biocontainers/bioconductor-summarizedexperiment/blobs/sha256:77c6c00e8b61bb628567c060b85690b0b0561bb37d8ad3f3792877bddcfe2500
Apr-03 22:46:27.981 [default-nioEventLoopGroup-4-2] INFO  io.seqera.controller.V2Controller - Java heap space
java.lang.OutOfMemoryError: Java heap space

I've added a compress/decompres ability cached manifest to try to mitigate the problem: 2703d96

Also added a memorized to the layer loading to prevent multiple loads of the same file: 1309fd7

But not sure the cache is the real problem, I'm starting to think there could be a mem leak somewhere

Bash binary disappear when pulling from the Tower reg

I've noticed that with some containers the /bin/bash binary in stripped in the image pulled from the tower registry.

For example:

$ docker run --rm -it quay.io/biocontainers/fastqc:0.11.9--0 ls -la /bin/bash
-rwxr-xr-x    1 root     root       1943912 May  9  2016 /bin/bash

However when pulling through tower reg:

$ docker run --rm -it 9090-seqeralabs-nftowercloud-b59bxddmpit.ws-eu34xl.gitpod.io/tw/of2wc6jonfxs6ytjn5rw63tumfuw4zlsom/fastqc:0.11.9--0 ls -la /bin/bash
ls: /bin/bash: No such file or directory

Env variables in the config file are not resolved

I'm trying to use some env variables in the application-dev.yml file, eg.

  registries:
    - name: docker.io
      host: registry-1.docker.io
      auth:
        username: ${DOCKER_USER}
        password: ${DOCKER_PAT}

However, it looks the are not resolved (see below). Not understanding the reason, we are heavily using this notation on Tower

21:33:05.341 [default-nioEventLoopGroup-1-2] DEBUG io.seqera.controller.V2Controller - Server configuration=DefaultConfiguration[arch=x86_64; registries=Registry[name=docker.io; host=registry-1.docker.io; auth=Auth[service=registry.docker.io; username=${DOCKER_USER}; password=${D****; url=auth.docker

Allow authentication irrespective of target image

The current client requires a different authentication token depending on the container image ie. repository for which the GET request is made.

See https://github.com/seqeralabs/tower-reg/blob/eb412d35af4be4afa7268f338878b4719c205cb3/src/main/groovy/io/seqera/ProxyClient.groovy#L56-L56

Ideally, it should be possible to get an auth token for all images owned by the user account. This would allow reusing the same client instance for multiple requests irrespective of the target image.

This may require using the authentication API v2 described here. However, in my tests, I was able to retrieve a token, but it was not allowing to access the registry APIs.

e.g.

TOKEN=$(curl -X POST https://hub.docker.com/v2/users/login \
   -H "Content-Type: application/json" \
   -d '{"username": "pditommaso", "password": "d213e955-3357-4612-8c48-fa5652ad968b"}' \
   | jq -r .token)


curl -X GET https://registry-1.docker.io/v2/library/hello-world/manifests/latest \
   -H "Content-Type: application/json" \
   -H "Authorization: ${TOKEN}"

Improve error reporting when invalid credentials are used

Ideally it should be reported an actional error message, avoiding the MN error stack trace mess

java.lang.IllegalStateException: Unable to authorize request -- response: {"details":"incorrect username or password"}

        at io.seqera.auth.BaseAuthProvider.getTokenFor(BaseAuthProvider.groovy:74)
        at io.seqera.proxy.ProxyClient.get1(ProxyClient.groovy:165)
        at io.seqera.proxy.ProxyClient.get0(ProxyClient.groovy:127)
        at io.seqera.proxy.ProxyClient$1.get(ProxyClient.groovy:108)
        at io.seqera.proxy.ProxyClient$1.get(ProxyClient.groovy)
        at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:236)
        at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
        at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:75)
        at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:176)
        at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:437)
        at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:115)
        at io.seqera.proxy.ProxyClient.get(ProxyClient.groovy:111)
        at io.seqera.proxy.ProxyClient.getStream(ProxyClient.groovy:70)
        at io.seqera.docker.ContainerService.handleRequest(ContainerService.groovy:64)
        at io.seqera.controller.V2Controller.handleGet(V2Controller.groovy:69)
        at io.seqera.controller.$V2Controller$Definition$Exec.dispatch(Unknown Source)
        at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:351)
        at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:583)
        at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303)
        at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111)
        at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103)
        at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659)
        at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49)
        at reactor.core.publisher.Flux.subscribe(Flux.java:8469)
        at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:426)
        at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57)
        at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398)
        at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:371)
        at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onSubscribe(ReactorSubscriber.java:50)
        at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68)
        at reactor.core.publisher.Flux.subscribe(Flux.java:8469)
        at io.micronaut.http.server.netty.RoutingInBoundHandler.handleRouteMatch(RoutingInBoundHandler.java:584)
        at io.micronaut.http.server.netty.RoutingInBoundHandler.channelRead0(RoutingInBoundHandler.java:447)
        at io.micronaut.http.server.netty.RoutingInBoundHandler.channelRead0(RoutingInBoundHandler.java:142)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:102)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.micronaut.http.netty.stream.HttpStreamsHandler.channelRead(HttpStreamsHandler.java:225)
        at io.micronaut.http.netty.stream.HttpStreamsServerHandler.channelRead(HttpStreamsServerHandler.java:134)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
        at io.netty.handler.codec.http.websocketx.extensions.WebSocketServerExtensionHandler.channelRead(WebSocketServerExtensionHandler.java:99)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
        at io.netty.handler.codec.http.HttpServerKeepAliveHandler.channelRead(HttpServerKeepAliveHandler.java:64)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.flow.FlowControlHandler.dequeue(FlowControlHandler.java:200)
        at io.netty.handler.flow.FlowControlHandler.channelRead(FlowControlHandler.java:162)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)

Improve image build waiting mechanism

Wave can build container on-fly when the container request provides a dockerfile.

http://github.com/seqeralabs/wave/blob/2b24fb330bab6220870141140f62164ed7404e4e/wave-api/src/main/groovy/io/seqera/wave/api/SubmitContainerTokenRequest.groovy#L30-L30

However, the build process can take some minutes. Therefore subsequence image requests need to wait for the build completion before starting to pull the corresponding layers.

Now it's uses a very simple blicking mechanism in the controller handler.

http://github.com/seqeralabs/wave/blob/2b24fb330bab6220870141140f62164ed7404e4e/src/main/groovy/io/seqera/wave/controller/RegistryProxyController.groovy#L62-L63

But this can be very inefficient. Likely a better approach consistent using reactive response.

Also the build response status should be taken into account an return an error when the build fails

Invalid submodules names

The the project has been re-organised in submodels that could be imported by other project.

The expected subprojet ground name should be io.seqera. See here.

However when the project is includes into other gradle builds via a includeBuild, the build is failing because the wave sub-projects are exported with the wave group name

Java TAR binary assembling

The Tar layer assembling and metadata file creation should be implemented in Java instead of a bash script.

This functionally should be made accessible both from with the registry and as a command-line tool to replace the current script.

Support for Tower credentials

The goal of this issue is to allow to authenticate requests for private docker registries by using Container credentials stored in the Tower backend.

In order to fulfil this requirement there are two challenges:

  1. the system should be aware of the user identity (and tower workspace)
  2. the system should be able to fetch matching credentials for the target registry for the current (user, workspace) pair.

General interaction

Screenshot 2022-06-14 at 16 49 18

The sequence diagram above depicts the expected interaction across the different components

  1. Nextflow the interaction submitting a request to validate the request for a private container image. The following data are submitted in the request payload
  • TOWER_ACCESS_TOKEN
  • TOWER_WORKSPACE_ID
  • container_image_name (eg. docker.io/user/image:foo)
  1. The proxy registry validate the TOWER_ACCESS_TOKEN against Tower system
  2. The Tower system ACK the user access token
  3. The registry confirms the request by sending Nextflow unique container_token
  4. The container_token is embedded in the container image name to be requested via the proxy e.g. proxy.reg/tw/<container_token>/user/image:foo
  5. The Docker client submit the request for proxy.reg/tw/<container_token>/user/image:foo
  6. The Proxy registry receives the request and uses the container_token to decode the corresponding user and target container to be accessed
  7. The Proxy registry queries the Tower system to retrieve a corresponding credentials pair for the target registry
  8. The Proxy registry authorises the image request in the target registry e.g. docker.io
  9. The Proxy registry requests the container image to the target registry
  10. The target registry responds with the image assets
  11. The Proxy request forward back the image assets to the Docker client

Allow handling encoded repository names

The goal of this feature is to allow serving arbitrary container repositories irrespective of the registry where they are hosted.

Since Docker repository names have constraints on the characters allowed and their semantics [1], this will require encoding the original repository name using base32 charset to prevent special characters into request URI.

Client encoding

For sake of clarity let's take in consideration the following scenario, the Nextflow client require to use the container with name quay.io/seqeralabs/nf-launcher:j11-22.01.0-edge. The following steps are taken

  1. the name is split by / components

  2. all components before the last / are encoded into base32 and converted to lowercase (padding characters are removed). Therefore quay.io/seqeralabs -> of2wc6jonfxs643fofsxeylmmfrhg

  3. the encoded component is used to form the new image repository name shown below and submitted to the docker client:

    <TOWER REGISTRY NAME>/tw/of2wc6jonfxs643fofsxeylmmfrhg/nf-launcher:j11-22.01.0-edge
    
  4. the docker client will dispatch requests using the following URI

    https://<TOWER REGISTRY NAME>/v2/tw/of2wc6jonfxs643fofsxeylmmfrhg/nf-launcher/manifests/j11-22.01.0-edge
    

Proxy decoding

The proxy registry server needs to listen for image names having the prefix /tw/<base32 string> and decode it recreating the target request. Therefore following the example above the request URI

  1. the encode component /tw/of2wc6jonfxs643fofsxeylmmfrhg in the request URL is matched

  2. the component of2wc6jonfxs643fofsxeylmmfrhg is decoded into quay.io/seqeralabs

  3. the quay.io registry name is used to create the target client (see #2) and a new request URI is composed i.e.

    https://quay.io/v2/seqeralabs/nf-launcher/manifests/j11-22.01.0-edge 
    
  4. the flow continues as in the current implementation

References

  1. https://docs.docker.com/registry/spec/api/#overview

Add endpoint to validate credential of an arbitrary container registry

The goal of this issue is to add an endpoint that given the triple (username, password, registry name) return OK, if the credentials provided allow accessing the registry of an error otherwise.

This "service" will be used to validate the registry credentials provided by a tower user as described here https://github.com/seqeralabs/nf-tower-cloud/issues/2482

The endpoint definition will be:

POST /validate-creds

Exchange object

class ValidateContainerRegistryCreds {
  String userName // mandatory 
  String password // mandatory 
  String registry // nullable, when missing fallback to docker.io 
}

The first implementation can be limited to docker.io and quay.io.

Second iteration it should be able to handle an arbitrary registry or at least AWS ECR, Azure registry and Google artifact registry

use reactive mode for blobs

currently we are requesting the blob to the remote repository and once downloaded we send to the client

using a reactive approach we can improve the download of large blobs images

Consolidate unit tests

Now that we have a fully working prototype, think it's a good idea to fix the existing tests (the ones still have sense) and the new ones whenever needed.

This activity is preliminary to the re-implementation with Micronaut.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.