kubelet and containerd in endless loop for CreateContainer with unexpected media type octet-stream about kubernetes HOT 7 CLOSED

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

from kubernetes.

The error message I see in the logs are probably come from here:
https://github.com/containerd/containerd/blob/v1.7.15/images/image.go#L238
Not sure about exact containerd version but I think it's 1.7.x

from kubernetes.

In the pod events we see thousands of events like:
"Container image "xyz" already present on machine" Pulled.
This seems to originate from here:
https://github.com/kubernetes/kubernetes/blob/v1.28.7/pkg/kubelet/images/image_manager.go#L138

So kubelet thinks that image is already present and successful pulled, but then containerd fails with:
"failed to create containerd container: error unpacking image: unexpected media type application/octet-stream for sha:xyz: not found: CreateContainerError"

from kubernetes.

The containerd fails probably here:
https://github.com/containerd/containerd/blob/v1.7.10/images/image.go#L238

Looks like something with the local manifest bookkeeping is not correct, which hinders container creation.

Maybe containerd can detect that image layer/manifest is read from local cache and remove erroneous cached local image layers

from kubernetes.

/sig node

from kubernetes.

kubelet and or containerd should detect the wrong layer/manifest state and delete the incomplete/erroneous download and pull again from container registry.

containerd should identiffy the corrupt layer and remove it, that's nothing the kubelet should do. The kubelet only tells the runtime to pull images when they're not present on disk.

Do you mind moving that issue to the containerd repo?

from kubernetes.

opened in containerd as containerd/containerd#10136

from kubernetes.