1.28-1.15.0 image causes node to fail on EKS 1.28 about bottlerocket HOT 8 CLOSED

I can't give any specific dates, but 1.15.1 should be very soon. 😉

from bottlerocket.

Hey, @pat-s thanks for letting us know about this. How are you deploying your Bottlerocket clusters? Are you using eksctl, karpenter, or neither of them? Also, could you please provide us the AMI ID and the region where you are experiencing this problem, if any?

from bottlerocket.

Using the terraform-eks module from hashicorp. latest version.

After having reverted to 1.27 I don't have the exact AMI ID at hand - but we're filtering for bottlerocket-aws-k8s-<version>-aarch* within the 149721548608 owner.
We're running a mixed cluster but it appeared on aarch64 and x86_64 at the same time.

We had to do this quite in a hurry as our ASG went to the limits after the nodes didn't join the cluster in time...

from bottlerocket.

Thanks! By any chance, is there anything special about your configurations to set up the DATA partition? Are you creating it based off an existing snapshot with a filesystem already in it?

from bottlerocket.

I had a hunch on what might cause this. We moved the data partition to XFS for new variants including aws-k8s-1.28. This is expecting an XFS-formatted data partition on boot. I posted #3499 which should allow systemd to mount /local/ even if it doesn't match the filesystem specified in the image so if one brings an ext4 (or other filesystem) data volume.

from bottlerocket.

Good catch you two! Yes indeed, we are adding another disk from a snapshot to all nodes. And yes this one should be ext4 I guess.

from bottlerocket.

Addressed by #3499 and will be included in the next release!

from bottlerocket.

Is there an ETA for 1.15.1? Would be great to align node versions with the control plane again 🙂

from bottlerocket.