Support for ignition to enable usage of OS's like Flatcar about cluster-api-provider-packet HOT 9 OPEN

Yeah, indeed it turns out be an issue of userdata starting with ## jinja-template.
Removing that line, the invalid character error was gone.
Yesterday we (vbatts and me) could not see it fixed just because existing clusterctl binaries and manifests were still pointing to old container repos of cluster-api. (Thanks @vbatts ! )

Question is, how do we proceed to fix the issue.
Since the upstream cluster-api apparently uses Jinja templates, it could be difficult to remove the jinja header from cluster-api.
Is it possible for the Packet provisioning system to accept Jinja templates in userdata?

from cluster-api-provider-packet.

We added that when we started using the jinja templating functionality in newer versions of cloud-init based on the cloud-init docs here: https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html#using-instance-data

Based on the docs I linked, if ## jinja-template is not the first line, then the functionality will not be enabled.

from cluster-api-provider-packet.

from cluster-api-provider-packet.

Hello!
Sorry, I lost this issue for some reason! I am glad the conversation is going well. For what I understand reading your comments you didn't track it down as a cluster-api-provider-packet issue yet.

• Add a way to override or extend this cloudinit content via the Cluster/machine spec

We thought about this when implementing the provider but it looks like a complicated approach to me, will love to stay away from it because I am sure it will be hard to get a procedure that can cover a good amount of formats and possibilities supported by the cloud-init. And I think this should be something made available from cluster-api project itself in this way it will work across providers.

from cluster-api-provider-packet.

I think there are two (or more) ways to go about this issue:

• Add support for something like ignition, as opposed to hardcoded cloudinit

I like this idea long term, but it is also a non-trivial amount of work, since there are cases where existing infrastructure providers (cluster-api-provider-aws, for example) that are assuming cloud-init and we might have to solve additional tangential issues as we sort those out (such as the way we are injecting bootstrapping data in cluster-api-provider-aws as a secret to avoid exposing certificate material in UserData as well as avoiding UserData size limits).

• Flatcar cloudinit is a compatible implementation in golang (not the upstream python one), so we could add a fix to ignore this setting (I don't like this option. Hacky, but most accessible)

I would expect that this work would likely also have to account for supporting the jinja2-based substitution values as well rather than just ignoring the header (if it doesn't already). Obviously this would be relatively non-trivial as well if the support for template substitution isn't already present.

• Add a way to override or extend this cloudinit content via the Cluster/machine spec

I'm hesitant to add more assumptions that the bootstrapping data is in cloud-init format, only because it makes the eventual support for additional formats more difficult. That said, if it gives us the ability to solve problems quicker, we should definitely leave it on the table.

Before we go too much further into how we can resolve the issue, I'd like to step back and go into the why do we have this issue in the first place.

Due to the way the Kubernetes cloud provider integration works, we need to ensure that the hostnames of the instances we are spinning up match the expected values found in the appropriate metadata fields used for lookups in the various cloud provider APIs, for example in AWS, we need to ensure that the hostname matches the private dns name of the instance.

To further complicate things, by default the hostname of the instance does not necessarily match this expected value, and what the value should be varies depending on the cloud provider, so we can't necessarily automate detection of this value at runtime in a way that works across cloud providers. We also can't guess what the value should be (at least for all cloud providers, even if it is deterministic for a subset of cloud providers).

However, we then discovered the templating feature for cloud-init, which allows us to customize each of the default templates on a per-provider basis, filling in the appropriate metadata information at runtime.

Which brings us back around to the current problem, where we'll need to ensure that any solution we pursue does not re-introduce the original issue, or that we find another way to address the issue that will work across various bootstrapping mechanisms.

from cluster-api-provider-packet.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

from cluster-api-provider-packet.

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

from cluster-api-provider-packet.

/lifecycle frozen

from cluster-api-provider-packet.

I believe ignition is supported by upstream cluster-api now, so that's nice. The question of how many OS's to support is a weighted one, however. As we have limited time/contributors. So it'd be good to know how heavily desired support for flatcar or any other OS is.

from cluster-api-provider-packet.