Option to create individual EC2 instances as Vault cluster rather than an ASG about terraform-aws-vault HOT 5 CLOSED

You make some pretty good points. However, a few thoughts in favor of the ASG:

• Vault Enterprise supports auto-unseal, so auto recovery may be quite useful.
• Relying on the IPs of EC2 Instances isn't a good idea, as the IP will change if the instance crashes and is replaced. You can attach an ENI and rely on that as your static IP, but that's doable with an ASG as well by running a script on each Instance to attach the ENI during boot.
• You can attach DNS records to instances in an ASG as well by running a script on each Instance to make the Route 53 changes.
• One of the advantages of an ASG is that it has integration with an ELB. Combined with auto-unseal, this would actually make it possible to do a zero-downtime, rolling update that replaces one Vault node at a time, waits for it to pass health checks, and then repeats the process with the next one. We have code to do this for some other distributed systems (e.g., kafka) and hope to open source it in the future.

That said, I can see arguments in either direction.

@KFishner Is there someone from the Vault team who could chime in with thoughts?

from terraform-aws-vault.

@amanoske what are your thoughts?

from terraform-aws-vault.

There are definitely circumstances where an ASG makes the most sense. Anytime you have auto-unseal set up it's a clear win, but I suspect that the auto-unseal use case is the minority. Most folks probably don't have Vault Pro/Enterprise, and some who do have it prefer the "threshold of unseal key holders" security model anyways (we would be in that camp).

Relying on the IPs of EC2 Instances isn't a good idea, as the IP will change if the instance crashes and is replaced. You can attach an ENI and rely on that as your static IP, but that's doable with an ASG as well by running a script on each Instance to attach the ENI during boot.

If you're not using an ASG, what replaces the crashed instance? The intended workflow here is that on instance crash, you run a terraform apply to create a replacement instance with a new IP, create a new ELB attachment, and update the route 53 record with the new IP. Then, you unseal the new instance. We're using the ASG + ENI attach on start pattern elsewhere (for Zookeeper+Kafka), but didn't think the ENI added much if there is no ASG.

In regards to both the ENI attachment and the DNS setup, we generally prefer for AWS modifications to be explicitly stated in and tracked by Terraform rather than applied dynamically by instances at launch time. With an ASG the benefits of the dynamic configuration can outweigh that preference, but for individual instances I'm not so sure.

In any case, I'm certainly not arguing for the elimination of the ASG option :-). There are a lot of tradeoffs here - some users will be best off with an ASG, some might not.

from terraform-aws-vault.

If you're not using an ASG, what replaces the crashed instance? The intended workflow here is that on instance crash, you run a terraform apply to create a replacement instance with a new IP, create a new ELB attachment, and update the route 53 record with the new IP. Then, you unseal the new instance.

Out of curiosity, how do you roll an update out across the instances? Do you modify them in place (e.g., using Chef or Ansible)?

In any case, I'm certainly not arguing for the elimination of the ASG option :-). There are a lot of tradeoffs here - some users will be best off with an ASG, some might not.

Well, we built this repo out of a bunch of individual modules specifically so we have this flexibility. I'd accept a PR that adds a new module that mimics the vault-cluster module, but uses individual instances rather than an ASG. One thing I'm a bit apprehensive about is that to make the cluster size configurable in this new module, you'd need to create aws_instance resources with the count parameter, which (a) may make it tricky to do deployments and (b) due to hashicorp/terraform#14275, will make it impossible to safely resize the cluster, though that's probably a rare use case.

from terraform-aws-vault.

re: updates, we baked new AMIs and then did Terraform deploys with -target to selectively replace instances.

We've revisited this, and ended up deciding to do an ASG where each instance creates a DNS record for itself on startup. We explored the ENI option, but the security groups were a little tricky to set up using this module. We really just wanted the DNS names more than the static IPs, and it was easy to just do that.

Example userdata snippet:

# Get internal VPC ip address and AZ of this instance.
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
MY_AZ=$(curl http://169.254.169.254/latest/meta-data/placement/availability-zone)
MY_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)

# Create a DNS record to help human operators.
cat <<EOF > dns-req.json
{
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "${MY_AZ}.${VAULT_DOMAIN}",
        "Type": "A",
        "TTL": 10,
        "ResourceRecords": [
          {
            "Value": "${MY_IP}"
          }
        ]
      }
    }
  ]
}
EOF
aws route53 change-resource-record-sets \
 --hosted-zone-id ${HOSTED_ZONE_ID} \
 --change-batch file://dns-req.json

Accordingly, I'm going to close this out.

from terraform-aws-vault.