container exec may hang on containerd version 1.7.14 about containerd HOT 7 CLOSED

Fixed in main (for 2.0.0 and above) by #9999
Backported to release/1.7 in #10037
Backported to release/1.6 in #10038

Will close when all are merged.

from containerd.

Given #9927, I think containerd version 1.6.30 may have the same problem.

from containerd.

Checked the commit for 1.6.x based on #9927 ,I think that 1.6.29 also has this problem.

$ git tag --contain 65915f0a2c0b4d1cb590f39c6cc3b78e7bb75fcb
v1.6.29
v1.6.30
v1.6.31

from containerd.

Closed it since we released 1.7.15

from containerd.

We just noticed that there is also an issue with the probes and this bug. Using exec probes result in a flaky status, see

❯ kubectl describe pod redis-master-0 | grep -A99 Events:
Events:
  Type     Reason     Age                      From     Message
  ----     ------     ----                     ----     -------
  Warning  Unhealthy  7m36s (x1069 over 5d6h)  kubelet  Liveness probe failed: command "sh -c /health/ping_liveness_local.sh 5" timed out
  Warning  Unhealthy  2m5s (x1967 over 5d6h)   kubelet  Readiness probe failed: command "sh -c /health/ping_readiness_local.sh 1" timed out

And we can see the following logs in the containerd logs

Apr 16 13:52:49 my-cluster-1 containerd[2646]: time="2024-04-16T13:52:49.461803674Z" level=error msg="Failed to delete exec process \"a1dd9aebf1ae39ee151caf36f1df76906459abe7df8f6fb3392f5d6af9d742a2\" for container \"ec071cda32b637ee003e8f49f9cb585b4fef27c3739458fdd9b594317b420579\"" error="context deadline exceeded"
Apr 16 13:52:49 my-cluster-1 containerd[2646]: time="2024-04-16T13:52:49.461905796Z" level=error msg="ExecSync for \"ec071cda32b637ee003e8f49f9cb585b4fef27c3739458fdd9b594317b420579\" failed" error="rpc error: code = DeadlineExceeded desc = failed to exec in container: timeout 6s exceeded: context deadline exceeded"

Updating to containerd v1.7.15 fixes this issue. Maybe we should mention this in the release notes.

from containerd.

Updating to containerd v1.7.15 fixes this issue. Maybe we should mention this in the release notes.

This was the core reason we released v1.7.15 and the one callout in the release notes is for that issue, stated as a regression in v1.7.14: "The fifteenth patch release for containerd 1.7 contains various fixes; one for a
regression introduced in v1.7.14 in the way process exits were handled."

Let us know if there would have been a better way to make it clear we found a regression.

from containerd.

Let us know if there would have been a better way to make it clear we found a regression.

Thanks for asking to make the release notes more clear. The notes were actually helpful at first, but we still weren't sure if it was the problem.
Adding examples such as exec into your container and Kubernetes probes via exec could enhance clarity and understanding.

from containerd.