BottleRocket in local VM - Cannot SSH into admin container about bottlerocket HOT 6 CLOSED

ecr-public allows for unauthenticated pulls, which is why that message is only a warning, and I believe may be a red herring.

Looking through the comments I don't think I see anything confirming if this VM has access to the internet, and if that's not enabled that could explain this behavior since we need to pull the container.

Could you please confirm that the networking for QEMU is configured to allow internet access, and that the VM is able to ping public.ecr.aws?

Thank you this was really helpful. You were right I was going down the wrong path with the unauthenticated pull warning. ping didn't work, then I realized this was probably a proxy setting issue. It was. Adding the proxy info to my user-data.toml file fixed it.

[settings.network]
https-proxy = "address:port"
no-proxy = ["localhost", "127.0.0.1"]

from bottlerocket.

Taking a look. It is to be expected that you cannot find an ssh service from the console since ssh runs in the admin container.

Can you try checking that the admin container is running with systemctl status [email protected]?

How confident are you that you have the correct public key, base64 encoded, in your user-data.toml file and that you have the corresponding private key loaded in your ssh-agent?

Can you show us the net.toml file?

Thank you.

from bottlerocket.

Taking a look. It is to be expected that you cannot find an ssh service from the console since ssh runs in the admin container.

Can you try checking that the admin container is running with systemctl status [email protected]?

How confident are you that you have the correct public key, base64 encoded, in your user-data.toml file and that you have the corresponding private key loaded in your ssh-agent?

Can you show us the net.toml file?

Thank you.

I'm confident the base64 encoded public key in user-data is correct and I've verified the corresponding private key is loaded in my ssh-agent.

Here's the output from the admin container status. I see "unauthenticated pull". The issue appears to be that I cannot pull "public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.1" due to failed authorization. Any idea how to troubleshoot this? I have my AWS credentials configured with AWS CLI. I can use AWS CLI commands no problem.

`bash-5.2# systemctl status [email protected]
● [email protected] - Host container: admin
Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/[email protected]; enabled; preset: enabled)
Active: active (running) since Thu 2024-03-28 18:11:46 UTC; 13s ago
Main PID: 1755 (host-ctr)
Tasks: 11 (limit: 38164)
Memory: 51.2M
CPU: 55ms
CGroup: /system.slice/system-host\x2dcontainers.slice/[email protected]
└─1755 /usr/bin/host-ctr run --container-id=admin --source=public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.1 --superpowered=true --registry-config=/etc/host-containers/host-ctr.toml

Mar 28 18:11:46 10.0.2.15 systemd[1]: Started Host container: admin.
Mar 28 18:11:46 10.0.2.15 host-ctr[1755]: time="2024-03-28T18:11:46Z" level=info msg="Image does not exist, proceeding to pull image from source." ref="public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.1"
Mar 28 18:11:59 10.0.2.15 host-ctr[1755]: time="2024-03-28T18:11:59Z" level=warning msg="ecr-public: failed to get authorization token, falling back to default resolver (unauthenticated pull)"`

Here is my net.toml file:

version = 1
[enp0s16]
dhcp4 = true

Thank you.

from bottlerocket.

hmm, what do you see with ctr containers list --address /run/host-containerd/containerd.sock, are there any containers running? It is weird that systemctl status <> shows the unit as active.

from bottlerocket.

hmm, what do you see with ctr containers list --address /run/host-containerd/containerd.sock, are there any containers running? It is weird that systemctl status <> shows the unit as active.

No, I don't see any containers running.

bash-5.2# ctr containers list
CONTAINER IMAGE RUNTIME

Here's the current output from systemctl status.

`bash-5.2# systemctl status [email protected]
● [email protected] - Host container: admin
Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/[email protected]; enabled; preset: enabled)
Active: active (running) since Fri 2024-03-29 20:49:14 UTC; 2min 56s ago
Main PID: 1751 (host-ctr)
Tasks: 11 (limit: 38164)
Memory: 52.5M
CPU: 105ms
CGroup: /system.slice/system-host\x2dcontainers.slice/[email protected]
└─1751 /usr/bin/host-ctr run --container-id=admin --source=public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.1 --superpowered=true --registry-config=/etc/host-containers/host-ctr.toml

Mar 29 20:49:27 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:49:27Z" level=warning msg="ecr-public: failed to get authorization token, falling back to default resolver (unauthenticated pull)"
Mar 29 20:49:57 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:49:57Z" level=info msg="trying next host" error="failed to do request: Head "https://public.ecr.aws/v2/bottlerocket/bottlerocket-admin/manifests/v0.11.1\": dial tcp 99.83.145.10:443: i/o timeout" host=public.ecr.aws
Mar 29 20:49:57 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:49:57Z" level=warning msg="failed to pull image. waiting 4.139s before retrying..." error="failed to resolve reference "public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.1": failed to do request: Head "https://public.ecr.aws/v2/bottlerocket/bottlerocket-admin/manifests/v0.11.1\": dial tcp 99.83.145.10:443: i/o timeout"
Mar 29 20:50:14 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:50:14Z" level=warning msg="ecr-public: failed to get authorization token, falling back to default resolver (unauthenticated pull)"
Mar 29 20:50:44 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:50:44Z" level=info msg="trying next host" error="failed to do request: Head "https://public.ecr.aws/v2/bottlerocket/bottlerocket-admin/manifests/v0.11.1\": dial tcp 99.83.145.10:443: i/o timeout" host=public.ecr.aws
Mar 29 20:50:44 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:50:44Z" level=warning msg="failed to pull image. waiting 6.393s before retrying..." error="failed to resolve reference "public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.1": failed to do request: Head "https://public.ecr.aws/v2/bottlerocket/bottlerocket-admin/manifests/v0.11.1\": dial tcp 99.83.145.10:443: i/o timeout"
Mar 29 20:51:02 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:51:02Z" level=warning msg="ecr-public: failed to get authorization token, falling back to default resolver (unauthenticated pull)"
Mar 29 20:51:32 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:51:32Z" level=info msg="trying next host" error="failed to do request: Head "https://public.ecr.aws/v2/bottlerocket/bottlerocket-admin/manifests/v0.11.1\": dial tcp 99.83.145.10:443: i/o timeout" host=public.ecr.aws
Mar 29 20:51:32 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:51:32Z" level=warning msg="failed to pull image. waiting 8.815s before retrying..." error="failed to resolve reference "public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.1": failed to do request: Head "https://public.ecr.aws/v2/bottlerocket/bottlerocket-admin/manifests/v0.11.1\": dial tcp 99.83.145.10:443: i/o timeout"
Mar 29 20:51:54 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:51:54Z" level=warning msg="ecr-public: failed to get authorization token, falling back to default resolver (unauthenticated pull)"`

from bottlerocket.

ecr-public allows for unauthenticated pulls, which is why that message is only a warning, and I believe may be a red herring.

Looking through the comments I don't think I see anything confirming if this VM has access to the internet, and if that's not enabled that could explain this behavior since we need to pull the container.

Could you please confirm that the networking for QEMU is configured to allow internet access, and that the VM is able to ping public.ecr.aws?

from bottlerocket.