Preload images of aws-node, kube-proxy and possible others about bottlerocket HOT 5 OPEN

Hello @runningman84, thanks for cutting this issue! I believe this is a similar request as #614 and #2023 where you would like images cached/preloaded before boot. The list you provided is a good list but I don't think every use cases needs all of these images. I think the better approach might be to provide a mechanism to do this caching for the containers you care about.

from bottlerocket.

I think in aws everything but the guardduty stuff is default in any cluster. Even if one or two images are not needed that would not hurt because unused images would be cleaned up if the disk is too full.

from bottlerocket.

This would be a way you could do it yourself: https://github.com/aws-samples/bottlerocket-images-cache/blob/main/README.md

from bottlerocket.

Yes that is possible but I would rather like a default solution without much complexity. Because AWS is releasing bottlerocket and other updates in a regular interval it would make sense to have it included out of the box.

You could even argue that this would help in the marketing because AWS might be the fastest provider to spin up new kubernetes nodes serving customer workloads...

from bottlerocket.

This would be a way you could do it yourself: https://github.com/aws-samples/bottlerocket-images-cache/blob/main/README.md

While this might be viable option for smaller images, it showed to be suboptimal for caching larger ones (1GB+) as container cold start improvements would be masked due to underlying stack of how snapshots are working - backed by s3.
So you will see your Pod in running state quite fast in few seconds, but our tests showed that our cold-start performance dropped by a half with snapshot pre-caching opposed to fresh download of an image.
Once you start Pod it needs to access certain volume blocks from snapshot (saved on s3), and from that point on it starts downloading them from s3 (warming that part of volume).
So result for our 4GB image:

• direct download from ECR ~3 mins to start processing
• with baked images into snapshot ~7+ minutes to start processing
  Although there is EBS FSR available, it is too expensive for such cases.

from bottlerocket.