Bottlerocket merging bootstrap_extra_args are adding extra quotes when using the official eks terraform module about bottlerocket HOT 7 OPEN

@Jinkxed Thanks for raising the issue. We will look in to this and get back to you.

from bottlerocket.

While going through the issue description, I noticed "double quotes" around the keys in

[settings.kubernetes]
"cluster-name" = "test-cluster"
"cluster-dns-ip" = "172.20.0.10"
"max-pods" = 110
[settings.kubernetes.node-labels]
"eks.amazonaws.com/nodegroup-image" = "ami-0d34ea6796ecf0833"
"eks.amazonaws.com/capacityType" = "SPOT"
"eks.amazonaws.com/sourceLaunchTemplateVersion" = "8"
"eks.amazonaws.com/nodegroup" = "eks-arm-spot-20240116231525619800000001"
"role" = "spot"
"eks.amazonaws.com/sourceLaunchTemplateId" = "lt-0f69ebb1047f87a70"

As referred here, the bootstrap_extra_args have been passed straight into user-data.
Can you confirm if the double quotes are required around the keys?

from bottlerocket.

Let me confirm, but I'm pretty sure you will get TOML errors if they aren't.

from bottlerocket.

Sorry, actually just re-read your question. That's how they come in without setting anything custom in the terraform module.

I know when I add additional keys that if I don't double quote them it will give TOML errors, the documentation shows it both ways? https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/18.17.0#eks-managed-node-groups

from bottlerocket.

Well that's how it was previously anyway. Just tried to a test in our dev environment.

Brand new cluster with no bootstrap_extra_args everything is double quoted.

[settings.kubernetes]
"cluster-name" = "us-east-1-dev"
"cluster-dns-ip" = "172.20.0.10"
"max-pods" = 17
[settings.kubernetes.node-labels]
"eks.amazonaws.com/nodegroup-image" = "ami-0dbfa71c674062f95"
"eks.amazonaws.com/capacityType" = "SPOT"
"eks.amazonaws.com/sourceLaunchTemplateVersion" = "1"
"eks.amazonaws.com/nodegroup" = "eks-arm-spot-2024011821273310770000001b"
"role" = "spot"
"eks.amazonaws.com/sourceLaunchTemplateId" = "lt-08bd91da980bf4d78"

If I add bootstrap_extra_args

    bootstrap_extra_args = <<-EOT
      [settings.kubernetes]
      max-pods = 300
    EOT

It comes in like this:

settings.kubernetes.cluster-name = 'us-east-1-dev'
settings.kubernetes.cluster-dns-ip = '172.20.0.10'
settings.kubernetes.max-pods = 300
settings.kubernetes.node-labels.'eks.amazonaws.com/sourceLaunchTemplateVersion' = '2'
settings.kubernetes.node-labels.role = 'spot'
settings.kubernetes.node-labels.'eks.amazonaws.com/nodegroup-image' = 'ami-0dbfa71c674062f95'
settings.kubernetes.node-labels.'eks.amazonaws.com/capacityType' = 'SPOT'
settings.kubernetes.node-labels.'eks.amazonaws.com/nodegroup' = 'eks-arm-spot-2024011821273310770000001b'
settings.kubernetes.node-labels.'eks.amazonaws.com/sourceLaunchTemplateId' = 'lt-08bd91da980bf4d78'

Which fails.

from bottlerocket.

The flattened form you end up with has all the keys in their dotted form, where dots delimit different levels in the hierarchy. I think the extra quotes come in for most of the node-labels, because they contain dots . in the key name. If they were not quoted the . in the key names would probably be interpreted as other layers of the settings tree. I do not think that kubernetes has extra suboptions eks.amazonaws within node-labels, so the quotation there seems right and aligns with TOML's grammar.

So the quotes you have in your extra user data around the keys seem to be necessary (at least for the node-labels) so that TOML interpreters do not mistake the dots in them as delimiters for the key hierarchy. It also seems correct that your output after merging has the extra quotes around them to ensure the dots in them are not mistaken for hierarchical delimiters in the TOML.

There is one difference though between the two cases from the TOML perspective. Strings with single quotation marks ' in toml are literal strings, while Strings with double quotations marks " are "basic strings". I do not see where that would make a difference in the example provided, but I am not completely sure.

Having not reproduced the issue I am not quite clear at what step you are running into issues with this. Is the instance never coming up properly or has it issues parsing the user data? Is is bailing out before even starting the instance? Could you elaborate on what you mean with "Which fails."

from bottlerocket.

FYI - this is not related to the EKS module; its how EKS managed nodegroups merge user data provided by users with the user data it provides. Thats where the changing to the flattened TOML is taking place

I don't now if that is causing an issue, or if there is an issue (that was unclear to me), but that is where the change in TOML form is coming from

from bottlerocket.