Comments (8)
yeazelm
commented on May 24, 2024
Thanks @g4-otayyan for cutting this issue. containerd should already have support and this might just be an issue with docker configuration. I'm working on confirming we see the same failure to pull, but in the mean time can you provide some logs that showed it failed to pull? Can you confirm which image you are using (aws-ecs-2, aws-k8s-1.29, etc) and where you are hosting the containers just to make sure we are looking at the same issue?
from bottlerocket.
yeazelm
commented on May 24, 2024
I was able to launch a container compressed with zstd both on an ECS and K8s Bottlerocket node.
$ docker buildx imagetools \
inspect --raw xxxxxxx.dkr.ecr.us-west-2.amazonaws.com/openai-zstd:latest
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"digest": "sha256:c2024fb4633e7afbd539120e14858ee0eba9432dd638edd283777bd5e9f9e82f",
"size": 8007
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:70da26ed5fce52ad4694154b867025c165f3aed37f4cf545335a1235e8a6a28e",
"size": 23944047
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:6ca97bcc9a7b671aa544cfbf31450af64373002beb0b8182cbd4910a5a90b1e6",
"size": 3306189
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:95d577cb9f9748f1e365a5b519dcdcd79622f31f646b67b20ecaa7e4cc19165a",
"size": 11961355
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:189fc3f6aad3b836ebded8e2bf3fb6a411e019c3be6c34434679136f44a6cf8d",
"size": 220
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:e827925d59450c588ed8d80f7d6bad3dd127b73907ab2423db2f2b62c93fc12d",
"size": 2641900
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:8ee8de0e5f0dbcf6c96bdfab23b1204db866f8435c66c033e25169be4457d2b8",
"size": 40192381
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:f97f6e46221bdab6e8d0f0e74cad080567528a203625d61eb67ece43e7a4d9f5",
"size": 226266438
}
]
ECS from the host
bash-5.1# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
55694a74ad8b xxxxx.dkr.ecr.us-west-2.amazonaws.com/openai-zstd:latest "sh -c 'sleep infini…" 2 minutes ago Up About a minute ecs-openai-zstd-1-openai-zstd-f4a6f79eea9397803b00
@g4-otayyan it might be useful to see the logs from the instance since I think it is working right now.
from bottlerocket.
g4-otayyan
commented on May 24, 2024
@yeazelm I am using the latest version of the aws-ecs-2 AMI taken directly from /aws/service/bottlerocket/aws-ecs-2/arm64/latest/image_id
The use case I have is that I need to create a pre-baked AMI with few docker images cached in it, the I was able to achieve this with the following code
for i in "${IMAGES_TO_PULL[@]}"; do
echo "pulling: $i"
apiclient exec admin bash -c "sheltie ctr -n moby image pull --user 'AWS:${DOCKER_ECR_TEMP_PASSWORD}' $i"
apiclient exec admin bash -c "sheltie ctr -n moby image export /local/img.tar $i"
apiclient exec admin bash -c "sheltie ctr -n moby image rm $i"
apiclient exec admin bash -c "sheltie docker image load -i /local/img.tar"
done
The reason I cannot use docker pull directly is because bottlerocket doesn't allow docker login to run in this context - which I need for my private repo - as it raises a Error saving credentials: mkdir /root/.docker: read-only file system error.
I checked the version of containerd used in the project and it's 1.6.x which doesn't support zstd compressed images, but the latest versions of containerd do.
EDIT: fixed the reason docker pull cannot be used
from bottlerocket.
yeazelm
commented on May 24, 2024
I checked the version of containerd used in the project and it's 1.6.x which doesn't support zstd compressed images, but the latest versions of containerd do
1.6.x does have zstd support, it was backported a while ago to older releases: https://github.com/containerd/containerd/releases/tag/v1.5.0. And in fact I was using 1.19.2 and aws-ecs-2 as well for my test, so I can confirm that the normal process of pulling zstd containers does work in this setup.
As for the series of commands, I'm not sure if this approach is expected to work. We typically recommend users launch ECS tasks on an host to get the pulls to complete normally, then snapshot from that. It might still be worth digging into why this seems to not work via your pattern. Do you have logs to show where this fails?
from bottlerocket.
g4-otayyan
commented on May 24, 2024
My apologies, in the latest bottlerocket version these are the logs that result out of calling apiclient exec admin bash -c "sheltie ctr -n moby image pull --user 'AWS:${DOCKER_ECR_TEMP_PASSWORD}' $i" on a zstd compressed image:
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
x.dkr.ecr.x.amazonaws.com/x:x: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:bd2cc1d7877ebc1414aab51d21a951ea3a4a14acc3e00f1ab5fd5421c2d0c38a: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:1d682a82b32cad30f4a8b73d0f66a7c18db6cfcc1174115f9293302dbc2cf49c: done |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:11c6b967d91d8f0368ec7ad64026df6a3998b64a8160bba0220ba7b64fe429a3: done |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:964c971e724254153b0c0416376d984c910b19de50047c7a7b88cefad2505f27: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:e0eb58eea5522872205ef2fe24d94e2f9537d868fde6f0cb1c0be94b6bd0f76e: done |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:4c3e6c4b7b998a58a0b55f47bf704d8da6e6e2aa1dcd66a755edd385bbd0ce29: done |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:7121ad798546b898cd1ce9ea799d02852c4434cced81dd4f5ca565d262a416f2: done |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:9ae9d6828c71fe034c1a0747a2c6580739ab00b84c590e37899ad07a24e73ddb: done |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:93e5719156f0a876f129f3ee626d80a9420ef9056503404abdd57e772e115e85: done |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:c3cf754191dcaf0ea966cecf4cb794c0c4582164d9bf191a76810bc46625e1a4: done |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:a4179a9859a0df9c349ed7f0729a5d8042b75e9e56d2a293004cab52e803fb5b: done |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:8f35caee9685f6316b1a1674265ef603ea318fa597d1a9a6c47191a7c553b3f9: done |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:ab1435a78319c6a1296266c286dc842908adf3c748875d13f696a977fc14ad0d: done |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:176adfaabeeeab89f4577dcf4c81775beba37dc13f98c010844480381c8794ab: done |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:9d48b3f86785929da90f57918147e12c16a38e442e7453aeab78243c513f8613: done |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:ba14d2ff578bcdce5845a40a14921c98e54eed53ff42072d388f296ab0b26735: done |++++++++++++++++++++++++++++++++++++++|
unpacking linux/amd64 sha256:bd2cc1d7877ebc1414aab51d21a951ea3a4a14acc3e00f1ab5fd5421c2d0c38a...
ctr: mismatched image rootfs and manifest layers
from bottlerocket.
g4-otayyan
commented on May 24, 2024
Perhaps the issue is with the manifest of the multi-arch image and not zstd compression?
from bottlerocket.
g4-otayyan
commented on May 24, 2024
I actually managed to make it work, here's a code snippet if anyone needs this later:
#!/bin/bash
set -euo pipefail
ROOT_URL=x.dkr.ecr.x.amazonaws.com
declare -a IMAGES_TO_PULL=(
"$ROOT_URL/x:x"
"$ROOT_URL/xx:xx"
)
apiclient exec admin bash -c "sheltie mkdir -p /tmp/docker/"
apiclient exec admin bash -c "sheltie docker --config /tmp/docker/ login --username AWS --password ${DOCKER_ECR_TEMP_PASSWORD} ${ROOT_URL}"
for i in "${IMAGES_TO_PULL[@]}"; do
echo "pulling: $i"
apiclient exec admin bash -c "sheltie docker --config /tmp/docker/ pull $i"
done
apiclient exec admin bash -c "sheltie rm -rf /tmp/docker/"
from bottlerocket.
yeazelm
commented on May 24, 2024
Thanks @g4-otayyan for the update! Glad you figured this out. I'll resolve this issue then.
from bottlerocket.
Related Issues (20)
- Support for User Namespaces in Kubernetes 1.30 HOT 2
- don't use bootconfig for systemd's unified cgroup hierarchy HOT 1
- v1.19.5 💘 Tracking Issue HOT 1
- pytorch could not detect Nvidia driver on bottlerocket HOT 6
- occasional build failures after extracting subpackages HOT 1
- Looking for aws-dev variant AMI ID HOT 1
- Fail to detect GPU on Bottlerocket v1.19 within AWS g4dn instance HOT 8
- v1.20.0 🐫 Tracking Issue HOT 1
- v1.20.0 update eni-max-pods mapping file HOT 1
- ootb: apiclient needs to be model agnostic HOT 1
- v1.20.0 Host container updates
- Is there any documentation for making bottlerocket work without the internet access to the instances security group ? HOT 1
- kernel-parameters does not accept single-word config options, specifying them causes reboot-loops HOT 3
- BottleRocket NVIDIA EKS Node group wont join EKS Cluster HOT 2
- nvidia-container-cli timeout error when running ECS tasks
- Changes to kernel module compression can break certain workflows HOT 5
- Cilium-agent does not start after upgrading to bottlerocket OS 1.20.0 HOT 1
- Host Container Unable to Create Container Task HOT 4
- Collecting logs from EKS Worker Nodes running Bottlerocket AMI when no SSH is enabled HOT 1
- Create symlinks to devices using the device name configured for EBS volumes
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bottlerocket.