Are you use the envd server? <li class="task-lis

bug: use-proxy not working about envd HOT 5 OPEN

ghpu commented on August 16, 2024
bug: use-proxy not working
from envd.
Comments (5)

kemingy commented on August 16, 2024 2
From the debug messages, it seems envd can get the base image metadata through the proxy (on your host), but the buildkitd in the container cannot access the Internet.
Is your proxy a localhost address?
Can you try envd v1 with moby? You can create the moby context with envd context create --name moby-test --builder moby-worker --use. v1
from envd.
gaocegege commented on August 16, 2024
Thanks for raising this!
from envd.
kemingy commented on August 16, 2024
Can you run envd --debug up --use-proxy to get the debug messages?
from envd.
ghpu commented on August 16, 2024
Here is the redacted version, X for project PATH, Z for proxy ip and port
DEBU[2023-12-01T10:19:35+01:00] /home/X/.config/envd/id_rsa_envd.pub already present 
DEBU[2023-12-01T10:19:35+01:00] /home/X/config/envd/id_rsa_envd already present 
DEBU[2023-12-01T10:19:35+01:00] home manager initialized                      cache-dir=/home/X/cache/envd cache-map="map[]" cache-status=/home/X/.cache/envd/cache.status config-file=/home/mclf0736/.config/envd/config.envd context="{default [{default docker-container envd_buildkitd docker <nil>}]}" context-file=/home/X/.config/envd/contexts
DEBU[2023-12-01T10:19:35+01:00] telemetry initialization                      UID=Y
DEBU[2023-12-01T10:19:35+01:00] sending telemetry                            
DEBU[2023-12-01T10:19:35+01:00] tag not specified, using default             
DEBU[2023-12-01T10:19:35+01:00] starting up command                           builder-options="{/home/X/projects/flyte/build.envd /home/X/.config/envd/config.envd plain flyte:dev /home/X/projects/flyte build /home/X/.config/envd/id_rsa_envd.pub    true }" container-name=flyte detach=false
DEBU[2023-12-01T10:19:35+01:00] getting exporter entry                        entry="[]"
DEBU[2023-12-01T10:19:35+01:00] commandconn: starting docker with [exec -i envd_buildkitd buildctl dial-stdio] 
DEBU[2023-12-01T10:19:35+01:00] starting buildkitd                            buildkit-config="&{[]}" container=envd_buildkitd tag="docker.io/moby/buildkit:v0.10.6"
DEBU[2023-12-01T10:19:35+01:00] commandconn (docker):Error response from daemon: No such container: envd_buildkitd 
DEBU[2023-12-01T10:19:35+01:00] waiting to start                              container=/envd_buildkitd
DEBU[2023-12-01T10:19:36+01:00] the container is running                      container=/envd_buildkitd
DEBU[2023-12-01T10:19:36+01:00] container is running, check if it's ready at docker-container://envd_buildkitd...  container=envd_buildkitd driver=docker-container image="docker.io/moby/buildkit:v0.10.6" socket=envd_buildkitd
DEBU[2023-12-01T10:19:36+01:00] waiting to connect to buildkitd               container=envd_buildkitd driver=docker-container image="docker.io/moby/buildkit:v0.10.6" socket=envd_buildkitd
DEBU[2023-12-01T10:19:37+01:00] commandconn: starting docker with [exec -i envd_buildkitd buildctl dial-stdio] 
DEBU[2023-12-01T10:19:38+01:00] connected to buildkitd                        container=envd_buildkitd driver=docker-container image="docker.io/moby/buildkit:v0.10.6" socket=envd_buildkitd
DEBU[2023-12-01T10:19:38+01:00] evaluating config file                        language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:38+01:00] interpret the file                            filename=/home/X/.config/envd/config.envd
DEBU[2023-12-01T10:19:38+01:00] interpret the file                            filename=/home/X/projects/flyte/build.envd
DEBU[2023-12-01T10:19:38+01:00] Execute build func                           
DEBU[2023-12-01T10:19:38+01:00] rule `base` is invoked, image=ghcr.io/flyteorg/flytekit:py3.11-1.10.1, dev=true  frontend=starlark
DEBU[2023-12-01T10:19:38+01:00] rule `install.python_packages` is invoked, name=[transformers], requirements=, local_wheels=[]  frontend=starlark
DEBU[2023-12-01T10:19:38+01:00] rule `install.apt_packages` is invoked, name=[git]  frontend=starlark
DEBU[2023-12-01T10:19:38+01:00] rule `runtime.environ` is invoked, env: map[Debug:True PYTHONPATH:/root _F_IMG_ID:localhost:30000/flytekit:CHRFCcdhdRIt8Zy3BHJfzg.. http_proxy:http://Z https_proxy:http://Z], extra_path: []  frontend=starlark
DEBU[2023-12-01T10:19:38+01:00] rule `config.pip_index` is invoked, index=https://pypi.org/simple, extraIndex=, trust=false  frontend=starlark
DEBU[2023-12-01T10:19:38+01:00] rule `install.python` is invoked, version=3.11  frontend=starlark
DEBU[2023-12-01T10:19:38+01:00] failed to check manifest update: image with hash 7502135eecfb2e36c99aff8968531891 not found  language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:38+01:00] compile LLB                                   gid=1000 uid=1000
DEBU[2023-12-01T10:19:38+01:00] compile base image                            image="ghcr.io/flyteorg/flytekit:py3.11-1.10.1" language="[{python 0xc000f8a140}]"
DEBU[2023-12-01T10:19:38+01:00] Using registries.d directory /etc/containers/registries.d 
DEBU[2023-12-01T10:19:38+01:00] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[2023-12-01T10:19:38+01:00] Trying to access "ghcr.io/flyteorg/flytekit:py3.11-1.10.1" 
DEBU[2023-12-01T10:19:38+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /run/user/1000/containers/auth.json 
DEBU[2023-12-01T10:19:38+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /home/X/.config/containers/auth.json 
DEBU[2023-12-01T10:19:38+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /home/X/.docker/config.json 
DEBU[2023-12-01T10:19:38+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /home/X/.dockercfg 
DEBU[2023-12-01T10:19:38+01:00] No credentials for ghcr.io/flyteorg/flytekit found 
DEBU[2023-12-01T10:19:38+01:00]  No signature storage configuration found for ghcr.io/flyteorg/flytekit:py3.11-1.10.1, using built-in default file:///home/X/.local/share/containers/sigstore 
DEBU[2023-12-01T10:19:38+01:00] Looking for TLS certificates and private keys in /etc/docker/certs.d/ghcr.io 
DEBU[2023-12-01T10:19:38+01:00] GET https://ghcr.io/v2/                      
DEBU[2023-12-01T10:19:39+01:00] Ping https://ghcr.io/v2/ status 401          
DEBU[2023-12-01T10:19:39+01:00] GET https://ghcr.io/token?scope=repository%3Aflyteorg%2Fflytekit%3Apull&service=ghcr.io 
DEBU[2023-12-01T10:19:39+01:00] Increasing token expiration to: 60 seconds   
DEBU[2023-12-01T10:19:39+01:00] GET https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1 
DEBU[2023-12-01T10:19:39+01:00] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json" 
DEBU[2023-12-01T10:19:39+01:00] Using registries.d directory /etc/containers/registries.d 
DEBU[2023-12-01T10:19:39+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /run/user/1000/containers/auth.json 
DEBU[2023-12-01T10:19:39+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /home/X/.config/containers/auth.json 
DEBU[2023-12-01T10:19:39+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /home/X/.docker/config.json 
DEBU[2023-12-01T10:19:39+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /home/X/.dockercfg 
DEBU[2023-12-01T10:19:39+01:00] No credentials for ghcr.io/flyteorg/flytekit found 
DEBU[2023-12-01T10:19:39+01:00]  No signature storage configuration found for ghcr.io/flyteorg/flytekit:py3.11-1.10.1, using built-in default file:///home/mclf0736/.local/share/containers/sigstore 
DEBU[2023-12-01T10:19:39+01:00] Looking for TLS certificates and private keys in /etc/docker/certs.d/ghcr.io 
DEBU[2023-12-01T10:19:39+01:00] GET https://ghcr.io/v2/                      
DEBU[2023-12-01T10:19:40+01:00] Ping https://ghcr.io/v2/ status 401          
DEBU[2023-12-01T10:19:40+01:00] GET https://ghcr.io/token?scope=repository%3Aflyteorg%2Fflytekit%3Apull&service=ghcr.io 
DEBU[2023-12-01T10:19:40+01:00] Increasing token expiration to: 60 seconds   
DEBU[2023-12-01T10:19:40+01:00] HEAD https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1 
DEBU[2023-12-01T10:19:40+01:00] GET https://ghcr.io/v2/flyteorg/flytekit/manifests/sha256:709d2f9d14b9a49cb3d319bbd75473ae6c8c50ee09cf6ab4698373fffb84c883 
DEBU[2023-12-01T10:19:40+01:00] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json" 
DEBU[2023-12-01T10:19:40+01:00] GET https://ghcr.io/v2/flyteorg/flytekit/manifests/sha256:1fbae966f0baa54da59da84f5db73f5ce051d8a23fd8992f48912a314878f04b 
DEBU[2023-12-01T10:19:40+01:00] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json" 
DEBU[2023-12-01T10:19:40+01:00] Downloading /v2/flyteorg/flytekit/blobs/sha256:bab837efcb3df968715ca678c859e552ce70ce7156ee66008ad0b9611bc9cce0 
DEBU[2023-12-01T10:19:40+01:00] GET https://ghcr.io/v2/flyteorg/flytekit/blobs/sha256:bab837efcb3df968715ca678c859e552ce70ce7156ee66008ad0b9611bc9cce0 
DEBU[2023-12-01T10:19:41+01:00] inherit envs from base image: [PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LANG=C.UTF-8 GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D PYTHON_VERSION=3.11.4 PYTHON_PIP_VERSION=23.1.2 PYTHON_SETUPTOOLS_VERSION=65.5.1 PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/0d8570dc44796f4369b652222cf176b3db6ac70e/public/get-pip.py PYTHON_GET_PIP_SHA256=96461deced5c2a487ddc65207ec5a9cffeca0d34e7af7ea1afc470ff0d746207 PYTHONPATH=/root FLYTE_INTERNAL_IMAGE=ghcr.io/flyteorg/flytekit:py3.11-1.10.1] 
DEBU[2023-12-01T10:19:41+01:00] apt/pypi calculated cacheID: /var/cache/apt/flyte-cpu 
DEBU[2023-12-01T10:19:41+01:00] apt/pypi calculated cacheID: /var/lib/apt/flyte-cpu 
DEBU[2023-12-01T10:19:41+01:00] using custom PyPI index                       index="https://pypi.org/simple"
DEBU[2023-12-01T10:19:41+01:00] Configure pip install statements              command="python -m pip install transformers"
DEBU[2023-12-01T10:19:41+01:00] apt/pypi calculated cacheID: /root/.cache/pip/flyte-cpu 
DEBU[2023-12-01T10:19:41+01:00] compiled build.envd                           language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:41+01:00] building envd image                           language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:41+01:00] build image with buildkit                     language-version=v1 tag="flyte:dev" type=docker
DEBU[2023-12-01T10:19:41+01:00] loading image to docker host                  language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:41+01:00] serving grpc connection                      
DEBU[2023-12-01T10:19:41+01:00] running BuildFunc for envd                    language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:41+01:00] final entrypoint: {[horust]}                  language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:41+01:00] setting image config: {"architecture":"amd64","os":"linux","config":{"User":"envd","ExposedPorts":{"2222/tcp":{}},"Env":["https_proxy=http://Z","http_proxy=http://Z","GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D","PYTHONPATH=/root","_F_IMG_ID=localhost:30000/flytekit:CHRFCcdhdRIt8Zy3BHJfzg..","PYTHON_VERSION=3.11.4","PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/0d8570dc44796f4369b652222cf176b3db6ac70e/public/get-pip.py","PYTHON_GET_PIP_SHA256=96461deced5c2a487ddc65207ec5a9cffeca0d34e7af7ea1afc470ff0d746207","FLYTE_INTERNAL_IMAGE=ghcr.io/flyteorg/flytekit:py3.11-1.10.1","PYTHON_PIP_VERSION=23.1.2","PYTHON_SETUPTOOLS_VERSION=65.5.1","USER=envd","SHELL=bash","ENVD_WORKDIR=/home/envd/flyte","Debug=True","LANG=C.UTF-8","PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/conda/envs/envd/bin:/opt/conda/bin:/home/envd/.local/bin","LC_ALL=en_US.UTF-8","LANG=C.UTF-8"],"Entrypoint":["horust"],"WorkingDir":"/home/envd/flyte","Labels":{"ai.tensorchord.envd.apt.packages":"[\"git\"]","ai.tensorchord.envd.build.context":"/home/X/projects/flyte","ai.tensorchord.envd.build.digest":"7502135eecfb2e36c99aff8968531891","ai.tensorchord.envd.container.name":"flyte","ai.tensorchord.envd.graph.general":"{\"Languages\":[{\"Name\":\"python\",\"Version\":\"3.11\"}],\"EnvdSyntaxVersion\":\"v1\",\"Image\":\"ghcr.io/flyteorg/flytekit:py3.11-1.10.1\",\"User\":\"envd\",\"Shell\":\"bash\",\"Dev\":true,\"CUDA\":null,\"CUDNN\":\"8\",\"NumGPUs\":0,\"ShmSize\":0,\"UbuntuAPTSource\":null,\"CRANMirrorURL\":null,\"JuliaPackageServer\":null,\"PyPIIndexURL\":\"https://pypi.org/simple\",\"PyPIExtraIndexURL\":null,\"PyPITrust\":false,\"PublicKeyPath\":\"/home/X/.config/envd/id_rsa_envd.pub\",\"PyPIPackages\":[[\"transformers\"]],\"RequirementsFile\":null,\"PythonWheels\":null,\"RPackages\":[],\"JuliaPackages\":[],\"SystemPackages\":[\"git\"],\"VSCodePlugins\":null,\"UserDirectories\":[],\"Exec\":[],\"Copy\":null,\"Mount\":null,\"HTTP\":null,\"Entrypoint\":null,\"Repo\":{},\"EnvironmentName\":\"flyte\",\"EnvironmentPath\":\"/home/X/projects/flyte\",\"WorkingDir\":\"/home/envd/flyte\",\"DisableMergeOp\":false,\"environ\":{\"Debug\":\"True\",\"FLYTE_INTERNAL_IMAGE\":\"ghcr.io/flyteorg/flytekit:py3.11-1.10.1\",\"GPG_KEY\":\"A035C8C19219BA821ECEA86B64E628F8D684696D\",\"LANG\":\"C.UTF-8\",\"PATH\":\"/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"PYTHONPATH\":\"/root\",\"PYTHON_GET_PIP_SHA256\":\"96461deced5c2a487ddc65207ec5a9cffeca0d34e7af7ea1afc470ff0d746207\",\"PYTHON_GET_PIP_URL\":\"https://github.com/pypa/get-pip/raw/0d8570dc44796f4369b652222cf176b3db6ac70e/public/get-pip.py\",\"PYTHON_PIP_VERSION\":\"23.1.2\",\"PYTHON_SETUPTOOLS_VERSION\":\"65.5.1\",\"PYTHON_VERSION\":\"3.11.4\",\"SHELL\":\"bash\",\"USER\":\"envd\",\"_F_IMG_ID\":\"localhost:30000/flytekit:CHRFCcdhdRIt8Zy3BHJfzg..\",\"http_proxy\":\"http://Z\",\"https_proxy\":\"http://Z\"},\"env_paths\":[\"/usr/local/sbin\",\"/usr/local/bin\",\"/usr/sbin\",\"/usr/bin\",\"/sbin\",\"/bin\",\"/opt/conda/envs/envd/bin:/opt/conda/bin:/home/envd/.local/bin\"],\"Platform\":{\"architecture\":\"amd64\",\"os\":\"linux\"}}","ai.tensorchord.envd.graph.runtime":"{\"environ\":{\"Debug\":\"True\",\"FLYTE_INTERNAL_IMAGE\":\"ghcr.io/flyteorg/flytekit:py3.11-1.10.1\",\"GPG_KEY\":\"A035C8C19219BA821ECEA86B64E628F8D684696D\",\"LANG\":\"C.UTF-8\",\"PATH\":\"/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"PYTHONPATH\":\"/root\",\"PYTHON_GET_PIP_SHA256\":\"96461deced5c2a487ddc65207ec5a9cffeca0d34e7af7ea1afc470ff0d746207\",\"PYTHON_GET_PIP_URL\":\"https://github.com/pypa/get-pip/raw/0d8570dc44796f4369b652222cf176b3db6ac70e/public/get-pip.py\",\"PYTHON_PIP_VERSION\":\"23.1.2\",\"PYTHON_SETUPTOOLS_VERSION\":\"65.5.1\",\"PYTHON_VERSION\":\"3.11.4\",\"SHELL\":\"bash\",\"USER\":\"envd\",\"_F_IMG_ID\":\"localhost:30000/flytekit:CHRFCcdhdRIt8Zy3BHJfzg..\",\"http_proxy\":\"http://Z\",\"https_proxy\":\"http://Z\"},\"env_paths\":[\"/usr/local/sbin\",\"/usr/local/bin\",\"/usr/sbin\",\"/usr/bin\",\"/sbin\",\"/bin\",\"/opt/conda/envs/envd/bin:/opt/conda/bin:/home/envd/.local/bin\"]}","ai.tensorchord.envd.ports":"[{\"name\":\"ssh\",\"port\":2222}]","ai.tensorchord.envd.pypi.commands":"[\"transformers\"]","ai.tensorchord.envd.r.packages":"[]","ai.tensorchord.envd.repo":"{}","ai.tensorchord.envd.syntax.version":"v1","ai.tensorchord.envd.vendor":"envd"}},"rootfs":{"type":"layers","diff_ids":null}}  language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:41+01:00] stopping session                             
#1 [internal] setting pip cache mount permissions
ERRO[2023-12-01T10:19:41+01:00] Buildkit error: failed to solve: failed to load cache key: failed to do request: Head "https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1": dial tcp 140.82.121.34:443: connect: connection refused
(1) attached stack trace
  -- stack trace:
  | github.com/tensorchord/envd/pkg/builder.generalBuilder.build.func1
  |     /home/runner/work/envd/envd/pkg/builder/build.go:265
  | golang.org/x/sync/errgroup.(*Group).Go.func1
  |     /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75
  | runtime.goexit
  |     /opt/hostedtoolcache/go/1.19.10/x64/src/runtime/asm_amd64.s:1594
Wraps: (2) Buildkit error
Wraps: (3) failed to solve: failed to load cache key: failed to do request: Head "https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1": dial tcp 140.82.121.34:443: connect: connection refused
  | (1) failed to solve: failed to load cache key: failed to do request: Head "https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1": dial tcp 140.82.121.34:443: connect: connection refused
  | Error types: (1) *builder.BuildkitdErr
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *builder.BuildkitdErr 
#1 DONE 0.0s

#2 docker-image://ghcr.io/flyteorg/flytekit:py3.11-1.10.1
#2 resolve ghcr.io/flyteorg/flytekit:py3.11-1.10.1 0.1s done
#2 ERROR: failed to do request: Head "https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1": dial tcp 140.82.121.34:443: connect: connection refused

#3 docker-image://docker.io/tensorchord/horust:v0.2.3
#3 resolve docker.io/tensorchord/horust:v0.2.3 0.1s done
#3 CANCELED

#4 docker-image://docker.io/mambaorg/micromamba:1.0.0
#4 resolve docker.io/mambaorg/micromamba:1.0.0 0.1s done
#4 CANCELED

#5 docker-image://docker.io/tensorchord/starship:v0.0.1
#5 resolve docker.io/tensorchord/starship:v0.0.1 0.1s done
#5 CANCELED

#6 docker-image://docker.io/tensorchord/envd-sshd-from-scratch:v0.3.40
#6 resolve docker.io/tensorchord/envd-sshd-from-scratch:v0.3.40 0.1s done
#6 CANCELED
------
 > docker-image://ghcr.io/flyteorg/flytekit:py3.11-1.10.1:
------
ERRO[2023-12-01T10:19:41+01:00] failed to load docker image: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.42/images/load?quiet=1": context canceled  language-version=v1 tag="flyte:dev"
FATA[2023-12-01T10:19:41+01:00] failed to build the image: failed to build: failed to wait error group: Buildkit error: failed to solve: failed to load cache key: failed to do request: Head "https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1": dial tcp 140.82.121.34:443: connect: connection refused
from envd.
ghpu commented on August 16, 2024
Proxy is not a localhost address,
Using envd1 works to access the images through the proxy ( I got another error later about groupadd and GID 1000 already existing, but that's another issue :)- )
from envd.
bug: use-proxy not working about envd HOT 5 OPEN

Comments (5)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent
