localhost registry using nodeport gets upgraded to HTTPS and HTTP fallback does not work as expected about containerd HOT 4 CLOSED

This is a behavior change from the historical behavior of localhost with containerd that only presents when configuring the new (recommended) format. Note we can do the override for containerd, something like (assuming using the default nodeport of 31999 in Zarf):

/etc/containerd/certs.d/127.0.0.1:31999/hosts.toml

[host."http://127.0.0.1:31999"]
  capabilities = ["pull", "resolve"]
  insecure_skip_verify = true

but that still breaks Zarf's ability to operate without manipulating K8s nodes directly. Even if we could manipulate the nodes via a daemonset (which is not ideal from a security standpoint), this still would not work because Zarf cannot guarantee an image is available for the daemonset due to being in an air-gapped environment.

The way Zarf accomplishes this today is by mounting a pod with en existing image from the cluster, but there's no guarantee new nodes will have that image, especially when upgrading nodes. This is also true for the pause image.

from containerd.

It requires a TLS server to respond with an http response after the connection is established

The TLS client will see an http response and fail the round trip, which can then be retried as http. If it is a TLS server, the fallback just passes through the response. The error here is that the fallback roundtripper is expecting the TLS client to receive an HTTP response, but in this case gets a handshake timeout. We can treat the handshake timeout as a fallback trigger here. There is also one more optimization to save the last fallback server to continue using http. Otherwise TLS will be attempted on each RoundTrip call. I'm working on these two changes which should solve this issue.

I think that the case mentioned should continue using the fallback and the best way to avoid it would be to add a configuration for the host. The fallback is to match existing expected behavior that Docker does and it is even more common now for services to default to using TLS even on localhost. For just testing/development it makes sense to use TLS on even on localhost since that will be used when moving to another environment. For local services, it makes sense to have a local configuration which could explicitly set the scheme and avoid the fallback.

from containerd.

I'm not sure I understand how the roundTripper fallback is supposed to function. It requires a TLS server to respond with an http response after the connection is established, so an http-only server wouldn't be compatible with this fallback mechanism. Also since this is against 127.0.0.1/localhost, I'm not sure how this would function as-designed without additional out-of-band management either through the host CA Trust or containerd hosts.toml config because you can't normally issue a valid cert to localhost, at least not on public CAs.

from containerd.

Thanks for the detailed reply, that does clear some things up with the behavior we were seeing, definitely the timeout issue.

Digging through this a bit more on our side, our behavior in rust uses rouille, which uses tiny-http and that essentially waits until the connection drops looking for the end of line, which it never sees as it's the beginning of the TLS negotiation.

We will also explore on our side alternative libraries or PRs with those libraries to address this behavior. In go, net/http connection is closed automatically it looks like.

go response:

docker run -d -p 5000:5000 --restart always registry:2

openssl s_client -alpn h2,http/1.1 -connect 127.0.0.1:5000

Connecting to 127.0.0.1
CONNECTED(00000003)
C09E7FD901000000:error:0A0000C6:SSL routines:tls_get_more_records:packet length too long:ssl/record/methods/tls_common.c:655:
C09E7FD901000000:error:0A000139:SSL routines::record layer failure:ssl/record/rec_layer_s3.c:650:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 324 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

curl -kv https://localhost:5000/v2/library/registry/manifests/2.8.3

*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* LibreSSL/3.3.6: error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version
* Closing connection 0
curl: (35) LibreSSL/3.3.6: error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version

our rust response:

openssl s_client -alpn h2,http/1.1 -connect 127.0.0.1:5000

Connecting to 127.0.0.1
CONNECTED(00000003)

***HANGS***

curl -kv https://localhost:5000/v2/library/registry/manifests/2.8.3

*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):

***HANGS***

from containerd.