massdriver-cloud / aws-eks-cluster Goto Github PK

The external-dns service is configured to use TXT DNS records to keep track of the A and CNAME records it has created in the hosted zone. The default name of the TXT record is the same as the name of the A/CNAME record.

This causes problems in scenarios where TXT records already exist, because external-dns generates an error in that case (see sample below). One reason that a TXT record might already exist is to comply with DNS validation checks for 3rd party services, such as Google Workspace or GMail.

To assist with this problem, external-dns supports a txt-prefix and txt-suffix parameter.

I propose that these values become available in the Massdriver configure UI when managing a bundle. It probably makes sense to place them behind an "advanced" configuration setting. I further propose that a prefix of edns. becomes the default. This would avoid the surprise that I ran into this week for newcomers, while also allowing folks the option of reverting to the previous behavior. It's worth noting that the external-dns docs discourage changing the txt-prefix or txt-suffix values after external-dns is installed in a cluster, because it may affect its ability to make DNS changes. For that reason, it might make sense to make these values read only after the bundle has been deployed.

(Note: The actual domain and hostedzone id have been redacted.)

│ time="2023-10-28T06:51:11Z" level=info msg="Applying provider record filter for domains: [example.io. .example.io.]"                                         │
│ time="2023-10-28T06:51:11Z" level=info msg="Desired change: DELETE www.example.io A [Id: /hostedzone/Z9999999999999]"                                 │
│ time="2023-10-28T06:51:11Z" level=info msg="Desired change: DELETE www.example.io TXT [Id: /hostedzone/Z9999999999999]"                               │
│ time="2023-10-28T06:51:11Z" level=info msg="Desired change: CREATE example.io A [Id: /hostedzone/Z9999999999999]"                                     │
│ time="2023-10-28T06:51:11Z" level=info msg="Desired change: CREATE example.io TXT [Id: /hostedzone/Z9999999999999]"                                   │
│ time="2023-10-28T06:51:11Z" level=error msg="Failure in zone example.io. [Id: /hostedzone/Z9999999999999]"                                            │
│ time="2023-10-28T06:51:11Z" level=error msg="InvalidChangeBatch: [Tried to create resource record set [name='example.io.', type='TXT'] but it already exists │
│ time="2023-10-28T06:51:11Z" level=error msg="failed to submit all changes for the following zones: [/hostedzone/Z9999999999999]"                      │
│

Issue: When a scaling event needs a node in a specific availability zone and otherwise has capacity, the cluster will not scale up. One of our customers found this when running an old version of Mongo. The EBS volume is zonal and mongo can't be scheduled because of this without manual intervention.

Fix: one autoscaler per availability zone

kubernetes/kubernetes#75402 (comment)