Comments (16)
tonistiigi
commented on May 28, 2024
1
(This should really be tracked at docker/buildx)
Looks like the issue is that the client side request to auth.docker.io is blocked, presumably because the system is configured to only allow it through proxy. Note that this request is different as client is sending it directly from buildx binary while for example pulling the layers happens between registry and buildkit daemon (inside a VM). This is so that your credentials are not shared with buildkit and only the short-lived token is sent there.
Options:
- Client-side token fetching can be disabled by defining
BUILDKIT_NO_CLIENT_TOKEN=1. Obviously, this does reduce security a bit because your password is sent to buildkit so that it can authenticate for you. This assumes that the proxy is either not needed or already configured for the VM requests. - You can define
HTTP_PROXY=... docker buildx ...when you run the buildx command (you can also configure your system so it is added automatically. Buildx will pick up such variables from the environment if they exist.
There is also old open PR docker/buildx#1487 to allow this case to be configured via config file but have not managed to get it merged yet.
from buildkit.
profnandaa
commented on May 28, 2024
Cross-linking #4643
from buildkit.
TBBle
commented on May 28, 2024
This one's not really area/windows, since this is Docker Desktop's WSL2 environment. And yeah, probably the same as #4643, although in this case it's the BuildKit embedded in buildx, and that one is seeing the issue in buildkitd in a buildx docker-container builder instance.
Edit: Ooops, technically it is area/windows, since it's a Windows build of buildx trying to fetch the token on the client side, not the buildkitd in WSL2, that's hitting the failure.
Edit again: Another oops, this might be unrelated to #4643, as that's shelled out to a command-line, and we explicitly provide an env that doesn't include http proxy info. This issue lies in our calls to the http client, and I noticed we changed HTTP clients between 0.12.1 and 0.13, so it might already be solved, but will need a version of buildx that implements 0.13 to re-test the repro case here.
Edit: @amyboose: Docker BuildX 0.13.0 is out and includes BuildKit 0.13.0. Please retest with that release, as it may have solved the issue due to a change in HTTP client used for the failing operation.
Also, I just noticed docker/buildx#1487 (comment) and docker/buildx#1979 which notes that BuildX doesn't currently get the Docker Desktop proxy configuration. So is the HTTP_PROXY env-var set in the session where you're running BuildX? If not, try setting it and see if that resolves the issue. I expect this would affect other command-line tools like curl (outside the container) but it's possible curl can read other sources of proxy info that maybe Go's HTTP client cannot.
from buildkit.
amyboose
commented on May 28, 2024
I've created buildkitd v13 using docker:
docker run --rm --privileged -d --name buildkit moby/buildkit
buildctl is located on Windows machine.
My command to build image:
buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=.
Result:
[+] Building 0.4s (2/2) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 101B 0.0s
=> ERROR [internal] load metadata for docker.io/library/nginx:1.13-alpine 0.3s
------
> [internal] load metadata for docker.io/library/nginx:1.13-alpine:
------
Dockerfile:1
--------------------
1 | >>> FROM nginx:1.13-alpine
2 | COPY conf /etc/nginx/conf.d/default.conf
--------------------
error: failed to solve: nginx:1.13-alpine: failed to resolve source metadata for docker.io/library/nginx:1.13-alpine: failed to do request: Head "https://registry-1.docker.io/v2/library/nginx/manifests/1.13-alpine": tls: failed to verify certificate: x509: certificate signed by unknown authority
Next time:
[+] Building 30.1s (3/3) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 101B 0.0s
=> ERROR [internal] load metadata for docker.io/library/nginx:1.13-alpine 30.1s
=> [auth] library/nginx:pull token for registry-1.docker.io 0.0s
------
> [internal] load metadata for docker.io/library/nginx:1.13-alpine:
------
Dockerfile:1
--------------------
1 | >>> FROM nginx:1.13-alpine
2 | COPY conf /etc/nginx/conf.d/default.conf
--------------------
error: failed to solve: nginx:1.13-alpine: failed to resolve source metadata for docker.io/library/nginx:1.13-alpine: failed to authorize: failed to fetch oauth token: Post "https://auth.docker.io/token": dial tcp 54.236.113.205:443: i/o timeout
After using debug:
time="2024-03-07T14:07:42+05:00" level=debug msg="commandconn: starting docker with [exec -i buildkit buildctl dial-stdio]"
time="2024-03-07T14:07:42+05:00" level=debug msg="serving grpc connection" spanID=86eca0f6028438bc traceID=2ee034080aedf0c2ecb2727962aab3b7
[+] Building 30.1s (3/3)
[+] Building 30.2s (3/3) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 101B 0.0s
=> ERROR [internal] load metadata for docker.io/library/nginx:1.13-alpine 30.1s
=> [auth] library/nginx:pull token for registry-1.docker.io 0.0s
------
> [internal] load metadata for docker.io/library/nginx:1.13-alpine:
------
time="2024-03-07T14:08:12+05:00" level=debug msg="commandconn (docker):write /dev/stdout: The pipe is being closed.\n"
Dockerfile:1
--------------------
1 | >>> FROM nginx:1.13-alpine
2 | COPY conf /etc/nginx/conf.d/default.conf
--------------------
error: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: nginx:1.13-alpine: failed to resolve source metadata for docker.io/library/nginx:1.13-alpine: failed to authorize: DeadlineExceeded: failed to fetch oauth token: Post "https://auth.docker.io/token": dial tcp 54.236.113.205:443: i/o timeout
8600 v0.13.0 C:\Users\DVEgorov10\Downloads\buildkit-v0.13.0.windows-amd64\bin\buildctl.exe --debug build --frontend=dockerfile.v0 --local context=. --local dockerfile=.
github.com/moby/buildkit/session/auth/authprovider.(*authProvider).FetchToken.func2
/src/session/auth/authprovider/authprovider.go:109
github.com/moby/buildkit/session/auth/authprovider.(*authProvider).FetchToken
/src/session/auth/authprovider/authprovider.go:133
github.com/moby/buildkit/session/auth._Auth_FetchToken_Handler.func1
/src/session/auth/auth.pb.go:954
github.com/moby/buildkit/session.NewSession.ChainUnaryServer.func3.1.1
/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
/src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:326
github.com/moby/buildkit/session.NewSession.filterServer.func1
/src/session/session.go:177
github.com/moby/buildkit/session.NewSession.ChainUnaryServer.func3.1.1
/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/moby/buildkit/session.NewSession.ChainUnaryServer.func3
/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/session/auth._Auth_FetchToken_Handler
/src/session/auth/auth.pb.go:956
google.golang.org/grpc.(*Server).processUnaryRPC
/src/vendor/google.golang.org/grpc/server.go:1343
google.golang.org/grpc.(*Server).handleStream
/src/vendor/google.golang.org/grpc/server.go:1737
google.golang.org/grpc.(*Server).serveStreams.func1.1
/src/vendor/google.golang.org/grpc/server.go:986
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1650
1 v0.13.0 buildkitd
github.com/moby/buildkit/session.grpcClientConn.ChainUnaryClient.func8.1.1
/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:72
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryClientInterceptor.func1
/src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:110
github.com/moby/buildkit/session.grpcClientConn.filterClient.func6
/src/session/session.go:186
github.com/moby/buildkit/session.grpcClientConn.ChainUnaryClient.func8.1.1
/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:72
github.com/moby/buildkit/session.grpcClientConn.ChainUnaryClient.func8
/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:81
google.golang.org/grpc.(*ClientConn).Invoke
/src/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/session/auth.(*authClient).FetchToken
/src/session/auth/auth.pb.go:869
github.com/moby/buildkit/session/auth.FetchToken.func1
/src/session/auth/auth.go:59
github.com/moby/buildkit/session.(*Manager).Any
/src/session/group.go:83
github.com/moby/buildkit/session/auth.FetchToken
/src/session/auth/auth.go:56
github.com/moby/buildkit/util/resolver.(*authHandler).fetchToken
/src/util/resolver/authorizer.go:341
github.com/moby/buildkit/util/resolver.(*authHandler).doBearerAuth.func1
/src/util/resolver/authorizer.go:307
github.com/moby/buildkit/util/flightcontrol.(*call[...]).run
/src/util/flightcontrol/flightcontrol.go:121
sync.(*Once).doSlow
/usr/local/go/src/sync/once.go:74
sync.(*Once).Do
/usr/local/go/src/sync/once.go:65
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1650
1 v0.13.0 buildkitd
github.com/moby/buildkit/client/llb/sourceresolver.(*imageMetaResolver).ResolveImageConfig
/src/client/llb/sourceresolver/imageresolver.go:51
github.com/moby/buildkit/frontend/gateway/forwarder.(*BridgeClient).ResolveImageConfig
/src/frontend/gateway/forwarder/forward.go:101
github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb.toDispatchState.toDispatchState.func3.func4
/src/frontend/dockerfile/dockerfile2llb/convert.go:429
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1650
1 v0.13.0 buildkitd
github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb.toDispatchState.toDispatchState.func3.func4
/src/frontend/dockerfile/dockerfile2llb/convert.go:437
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1650
8600 v0.13.0 C:\Users\DVEgorov10\Downloads\buildkit-v0.13.0.windows-amd64\bin\buildctl.exe --debug build --frontend=dockerfile.v0 --local context=. --local dockerfile=.
google.golang.org/grpc.getChainUnaryInvoker.func1
/src/vendor/google.golang.org/grpc/clientconn.go:519
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryClientInterceptor.func1
/src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:110
github.com/moby/buildkit/client.New.filterInterceptor.func5
/src/client/client.go:387
google.golang.org/grpc.DialContext.chainUnaryClientInterceptors.func3
/src/vendor/google.golang.org/grpc/clientconn.go:507
google.golang.org/grpc.(*ClientConn).Invoke
/src/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/frontend/gateway/pb.(*lLBBridgeClient).Solve
/src/frontend/gateway/pb/gateway.pb.go:3078
github.com/moby/buildkit/client.(*gatewayClientForBuild).Solve
/src/client/build.go:94
github.com/moby/buildkit/frontend/gateway/grpcclient.(*grpcClient).Solve
/src/frontend/gateway/grpcclient/client.go:415
main.buildAction.func5.2
/src/cmd/buildctl/build.go:378
github.com/moby/buildkit/frontend/gateway/grpcclient.(*grpcClient).Run
/src/frontend/gateway/grpcclient/client.go:218
github.com/moby/buildkit/client.(*Client).Build.func2
/src/client/build.go:59
github.com/moby/buildkit/client.(*Client).solve.func3
/src/client/solve.go:300
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1650
1 v0.13.0 buildkitd
github.com/moby/buildkit/frontend/gateway.(*llbBridgeForwarder).Return
/src/frontend/gateway/gateway.go:975
github.com/moby/buildkit/control/gateway.(*GatewayForwarder).Return
/src/control/gateway/gateway.go:146
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Return_Handler.func1
/src/frontend/gateway/pb/gateway.pb.go:3441
main.main.func3.ChainUnaryServer.func2.1.1
/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
/src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:326
main.unaryInterceptor.func1
/src/cmd/buildkitd/main.go:686
main.main.func3.ChainUnaryServer.func2.1.1
/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
main.main.func3.ChainUnaryServer.func2
/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Return_Handler
/src/frontend/gateway/pb/gateway.pb.go:3443
google.golang.org/grpc.(*Server).processUnaryRPC
/src/vendor/google.golang.org/grpc/server.go:1343
google.golang.org/grpc.(*Server).handleStream
/src/vendor/google.golang.org/grpc/server.go:1737
google.golang.org/grpc.(*Server).serveStreams.func1.1
/src/vendor/google.golang.org/grpc/server.go:986
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1650
8600 v0.13.0 C:\Users\DVEgorov10\Downloads\buildkit-v0.13.0.windows-amd64\bin\buildctl.exe --debug build --frontend=dockerfile.v0 --local context=. --local dockerfile=.
google.golang.org/grpc.getChainUnaryInvoker.func1
/src/vendor/google.golang.org/grpc/clientconn.go:519
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryClientInterceptor.func1
/src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:110
github.com/moby/buildkit/client.New.filterInterceptor.func5
/src/client/client.go:387
google.golang.org/grpc.DialContext.chainUnaryClientInterceptors.func3
/src/vendor/google.golang.org/grpc/clientconn.go:507
google.golang.org/grpc.(*ClientConn).Invoke
/src/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
/src/api/services/control/control.pb.go:2234
github.com/moby/buildkit/client.(*Client).solve.func2
/src/client/solve.go:274
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1650
8600 v0.13.0 C:\Users\DVEgorov10\Downloads\buildkit-v0.13.0.windows-amd64\bin\buildctl.exe --debug build --frontend=dockerfile.v0 --local context=. --local dockerfile=.
github.com/moby/buildkit/client.(*Client).solve.func2
/src/client/solve.go:290
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1650
from buildkit.
TBBle
commented on May 28, 2024
Try running the buildkit daemon with
docker run --rm --privileged -d --name buildkit --env "HTTP_PROXY=<YourProxyDetailsHere>" --env "HTTPS_PROXY=<YourProxyDetailsHere>" moby/buildkit
or if you already have the HTTP_PROXY and/or HTTPS_PROXY env-vars set in the relevant shell,
docker run --rm --privileged -d --name buildkit --env HTTP_PROXY --env HTTPS_PROXY moby/buildkit
from buildkit.
amyboose
commented on May 28, 2024
Try running the buildkit daemon with
docker run --rm --privileged -d --name buildkit --env "HTTP_PROXY=<YourProxyDetailsHere>" --env "HTTPS_PROXY=<YourProxyDetailsHere>" moby/buildkitor if you already have the
HTTP_PROXYand/orHTTPS_PROXYenv-vars set in the relevant shell,docker run --rm --privileged -d --name buildkit --env HTTP_PROXY --env HTTPS_PROXY moby/buildkit
The same result. It's not the error of only my PC. Other computers also has the same error message. I think it can be the error on SSL sertificate.
I've tested a command
docker run -it mcr.microsoft.com/dotnet/aspnet:8.0 openssl s_client -connect auth.docker.io:443
and it returns result:
CONNECTED(00000003)
depth=1 O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
verify return:1
depth=0 CN = *.docker.com
verify return:1
---
Certificate chain
0 s:CN = *.docker.com
i:O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 7 08:47:15 2023 GMT; NotAfter: Sep 5 08:47:15 2024 GMT
1 s:O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
i:O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Mar 10 08:13:42 2014 GMT; NotAfter: Mar 5 08:13:42 2034 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
some_Value
-----END CERTIFICATE-----
subject=CN = *.docker.com
issuer=O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 10785 bytes and written 384 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---
DONE
from buildkit.
TBBle
commented on May 28, 2024
That error shows it's not getting or using the proxy settings, and is instead being transparently redirected to your corporate firewall (or possibly the local computer's firewall), which relies on having the firewall's own certificate trusted by the computer as a variant of a man-in-the-middle attack. The SSL setup in side the container does not trust this certificate of course, so it fails as you see.
The real docker.com certificate is not signed by Kaspersky's firewall product, but by Amazon.
So, try:
docker run -it mcr.microsoft.com/dotnet/aspnet:8.0 openssl s_client -connect auth.docker.io:443 -proxy <YourProxyDetailsHere>
and if that still doesn't work, then I think <YourProxyDetailsHere> is incorrect in some way.
Once that's working, you can test with curl in a container, passing the --env "HTTP_PROXY=<YourProxyDetailsHere>" --env "HTTPS_PROXY=<YourProxyDetailsHere>" parameters to docker run.
Once that is working, there's no reason I'm aware of that
docker run --rm --privileged -d --name buildkit --env "HTTP_PROXY=<YourProxyDetailsHere>" --env "HTTPS_PROXY=<YourProxyDetailsHere>" moby/buildkit
would not work, but if it still fails, then we've possibly identified a real bug somewhere, which will probably be a hassle to reproduce and diagnose from outside a corporately-firewalled network. >_< But we can cross that bridge when we come to it.
from buildkit.
amyboose
commented on May 28, 2024
Recently my organization disabled firewall for .docker.com host.
The error from header is still exists.
Executing docker with openssl on auth.docker.io:
docker run -it --rm mcr.microsoft.com/dotnet/aspnet:8.0 openssl s_client -connect auth.docker.io:443
docker run -it --rm mcr.microsoft.com/dotnet/aspnet:8.0 openssl s_client -proxy my_proxy_here -connect auth.docker.io:443
Result for both:
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, CN = Amazon RSA 2048 M01
verify return:1
depth=0 CN = *.docker.com
verify return:1
---
Certificate chain
0 s:CN = *.docker.com
i:C = US, O = Amazon, CN = Amazon RSA 2048 M01
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Oct 2 00:00:00 2023 GMT; NotAfter: Oct 31 23:59:59 2024 GMT
1 s:C = US, O = Amazon, CN = Amazon RSA 2048 M01
i:C = US, O = Amazon, CN = Amazon Root CA 1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Aug 23 22:21:28 2022 GMT; NotAfter: Aug 23 22:21:28 2030 GMT
2 s:C = US, O = Amazon, CN = Amazon Root CA 1
i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: May 25 12:00:00 2015 GMT; NotAfter: Dec 31 01:00:00 2037 GMT
3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 2 00:00:00 2009 GMT; NotAfter: Jun 28 17:39:16 2034 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
some data
-----END CERTIFICATE-----
subject=CN = *.docker.com
issuer=C = US, O = Amazon, CN = Amazon RSA 2048 M01
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5693 bytes and written 453 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
Executing docker with openssl on reddit:
docker run -it --rm mcr.microsoft.com/dotnet/aspnet:8.0 openssl s_client -connect reddit.com:443
Result:
CONNECTED(00000003)
depth=1 O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
verify return:1
depth=0 C = US, ST = California, L = SAN FRANCISCO, O = "REDDIT, INC.", CN = *.reddit.com
verify return:1
---
Certificate chain
0 s:C = US, ST = California, L = SAN FRANCISCO, O = "REDDIT, INC.", CN = *.reddit.com
i:O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Oct 6 03:31:43 2023 GMT; NotAfter: Oct 4 03:31:43 2024 GMT
1 s:O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
i:O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Mar 10 08:13:42 2014 GMT; NotAfter: Mar 5 08:13:42 2034 GMT
---
Server certificate
some data
-----END CERTIFICATE-----
subject=C = US, ST = California, L = SAN FRANCISCO, O = "REDDIT, INC.", CN = *.reddit.com
issuer=O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 7740 bytes and written 380 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---
Kaspersky (firewall) is still exists.
Executing docker build:
docker build . --build-arg HTTP_PROXY=http://my_proxy_here --build-arg HTTPS_PROXY=http://my_proxy_here
Result:
ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: nginx:1.13-alpine: failed to authorize: DeadlineExceeded: failed to fetch oauth token: Post "https://auth.docker.io/token": dial tcp 54.236.113.205:443: i/o timeout
Executing curl with no proxy:
docker run -it --rm --env "HTTP_PROXY=http://my_proxy_here" --env "HTTPS_PROXY=http://my_proxy_here" mcr.microsoft.com/dotnet/aspnet:8.0 /bin/bash -c "apt update && apt upgrade -y && apt install curl -y && curl https://reddit.com/"
docker run -it --rm --env "HTTP_PROXY=http://my_proxy_here" --env "HTTPS_PROXY=http://my_proxy_here" mcr.microsoft.com/dotnet/aspnet:8.0 /bin/bash -c "apt update && apt upgrade -y && apt install curl -y && curl https://reqbin.com/echo"
Result:
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
Executing curl with auth.docker.io and no proxy:
docker run -it --rm --env "HTTP_PROXY=http://my_proxy_here" --env "HTTPS_PROXY=http://my_proxy_here" mcr.microsoft.com/dotnet/aspnet:8.0 /bin/bash -c "apt update && apt upgrade -y && apt install curl -y && curl https://auth.docker.io/token"
Result: success request
Executing curl with proxy on context.reverso.net:
docker run -it --rm --env "HTTP_PROXY=http://my_proxy_here" --env "HTTPS_PROXY=http://my_proxy_here" mcr.microsoft.com/dotnet/aspnet:8.0 /bin/bash -c "apt update && apt upgrade -y && apt install curl -y && curl -x http://my_proxy_here https://context.reverso.net/"
Result:
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
Most of the time curl -x works good.
Executing buildctl:
docker run --rm --privileged -d --name buildkit --env "HTTP_PROXY=http://my_proxy_here" --env "HTTPS_PROXY=http://my_proxy_here" moby/buildkit
buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=.
Result:
[+] Building 31.0s (3/3) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 158B 0.0s
=> ERROR [internal] load metadata for docker.io/library/nginx:1.13-alpine 30.8s
=> [auth] library/nginx:pull token for registry-1.docker.io 0.0s
------
> [internal] load metadata for docker.io/library/nginx:1.13-alpine:
------
Dockerfile:1
--------------------
1 | >>> FROM nginx:1.13-alpine
2 | RUN openssl verify /usr/share/ca-certificates/nuget.crt
3 | COPY conf /etc/nginx/conf.d/default.conf
--------------------
error: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: nginx:1.13-alpine: failed to resolve source metadata for docker.io/library/nginx:1.13-alpine: failed to authorize: DeadlineExceeded: failed to fetch oauth token: Post "https://auth.docker.io/token": dial tcp 54.198.86.24:443: i/o timeout
I think that http client ignores proxy when build image based on another image
from buildkit.
brecode
commented on May 28, 2024
I am facing the exact same issue when running behind a corporate proxy. Everything else works fine but build doesn't honor proxy settings
from buildkit.
amyboose
commented on May 28, 2024
@TBBle Can you look at my last comment, please?
from buildkit.
kunZooberg
commented on May 28, 2024
It looks like a discussion started from 2019, may I know if there is some inner reason why we don't support it yet?
#971
from buildkit.
amyboose
commented on May 28, 2024
(This should really be tracked at docker/buildx)
Looks like the issue is that the client side request to auth.docker.io is blocked, presumably because the system is configured to only allow it through proxy. Note that this request is different as client is sending it directly from
buildxbinary while for example pulling the layers happens between registry and buildkit daemon (inside a VM). This is so that your credentials are not shared with buildkit and only the short-lived token is sent there.Options:
- Client-side token fetching can be disabled by defining
BUILDKIT_NO_CLIENT_TOKEN=1. Obviously, this does reduce security a bit because your password is sent to buildkit so that it can authenticate for you. This assumes that the proxy is either not needed or already configured for the VM requests.- You can define
HTTP_PROXY=... docker buildx ...when you run the buildx command (you can also configure your system so it is added automatically. Buildx will pick up such variables from the environment if they exist.There is also old open PR docker/buildx#1487 to allow this case to be configured via config file but have not managed to get it merged yet.
I've tried HTTP_PROXY, http_proxy, HTTPS_PROXY, https_proxy in many ways to solve the problem. Nothing works when FROM <some image> is on the start of Dockerfile.
And I confirm that my internet traffic goes only through proxy. I guess that docker pull use proxy but FROM instruction ignores proxy all the time
from buildkit.
tonistiigi
commented on May 28, 2024
Post how you are setting the proxy and what exact error you get.
from buildkit.
amyboose
commented on May 28, 2024
Post how you are setting the proxy and what exact error you get.
You can find almost all information above in my comment and in the head.
Also I want to summarize my configuration:
- Docker 4.28 and 4.29
- Windows 10
- Proxies is configured by docker settings for both http and https. Proxy works from containers because my corporate internet traffic goes only through proxy. And proxy works only when I build image, based on other images from cache.
- I've tried build-args both for docker build and docker-compose for building image based on non-cached image. Nothing works.
- I've checked traffic using Wireshark. It shows that docker get DNS addresses of auth.docker.io and then send HTTP traffic directly to these addresses with no proxy.
During some time my organization disabled traffic filtering and SSL bump for me. It also didn't help solve the problem.
Also I did many tests using buildx with no docker. You can find all my test results above. Unfortunately now I can't do tests within a week.
I've solved all my issues on corporate network using proxy setting, CA certificates and build-args. This is the only problem left. And I'm almost sure that there is a bug
from buildkit.
tonistiigi
commented on May 28, 2024
You can find almost all information above in my #4725 (comment) and in the head.
I can't spot any instance trying either solution from #4725 (comment)
from buildkit.
amyboose
commented on May 28, 2024
You can find almost all information above in my #4725 (comment) and in the head.
I can't spot any instance trying either solution from #4725 (comment)
Ok, I will try solutions from your comment and let you know.
from buildkit.
Related Issues (20)
- wishlist: annotations on docker/dockerfile images HOT 2
- WCOW fails to load custom frontend HOT 3
- Proposal: Use referrers api to import/export inline type cache
- bake does not merge compose services/<service>/build/x-bake setting with serivice extensions
- copy --parents silently ignores nonexistent paths HOT 2
- buildkit WCOW cannot seemingly run `RUN powershell ...`, while vanilla dockerd can HOT 4
- Proposal: csv syntax for git repos HOT 3
- Support extracting `ADD --checksum=.. https://.. ..` HOT 6
- 0.13.2 corrupts cache and cannot start (`go.etcd.io/bbolt.(*freelist).read`, `panic: invalid freelist page: 0, page type is unknown<0`) HOT 5
- Local and Registry cache not used or *invalidating* cache unnecessarily HOT 3
- can't add CA cert for gha remotecache?
- Inconsistencies in RUN mount options parsing/handling HOT 1
- dockerfile: Add integration test for ARG with empty value and default
- [v0.13] It seems that "registry.insecure=true" doesn't work. HOT 7
- buildkit remote cache fails if manifest is larger than 1MB
- Windows buildkit: copy context error HOT 6
- Proposal: lint disable controls HOT 2
- windows differ is not implemented HOT 1
- `FROM --platform=$BUILDPLATFORM a as b` not overriding `TARGETPLATFORM` for multi-stage builds HOT 7
- `checksum`ing "local files" for remote builder contexts HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from buildkit.