https://www.hashicorp.com/blog/announcing-terraform-0-12

for finding out what providers are compatible with 0.12

hashicorp/terraform#21235 (comment)

https://github.com/terraform-providers/terraform-provider-scaleway/issues/125

currently certs are created with cert authority as a local ip

unfortunately, it is not available for arm64/aarch64 yet

https://community.cloudflare.com/t/request-for-arm64-architecture-binary/16687?u=balupton

perhaps installing it via go get instead of rpm would be an option

vault is failing according to consul, however vault is running fine, with the exception of this error message:

check unable to talk with Consul backend: error="Unexpected response code: 500 (CheckID "vault:10.10.36.185:8200:vault-sealed-check" does not have associated TTL)

Summary:

Generating the nomad certs on origin did not work as the nomad machines would then have certs which did not include their private_ips in the certs ip_sans, which would cause the cert to be rejected from the local instance.

I then tried to generate the nomad certs on the nomad machines. That fixes the ip_sans issue, but then prevents nomad to nomad communication as each nomad instance then has different certs.

Solving this seems to require a certificate issuer and update service.

Possible Solutions:

Local polling + local issuance:

1. Create a poll service on each machine that polls a vault secret (that contains issued pki combo json) every 30 seconds, if there is a change, then reconfigure the local nomad service.
2. When a new nomad service is required, append another vault secret with the new private_ip, then generate a new pki combo with all the private_ips from the earlier secret, put that combo json into the secret at step 1.
3. To setup the vault secrets, vault policies and tokens would need to be created for the polling and writing requirement. Or just use the cluster_token in memory.

Developer issuance in pre:

1. For each new server that was just issued but not yet configured, the terraform script then remotes into existing services and updates their TLS cert to include the new server's private_ip.
2. Generation of the PKI bundle could occur locally or on origin, then propagated.

Developer issuance in post:

1. All services have TLS off at the start
2. Then once all servers are deployed and running, remote into origin, generate the certs containing all their private_ips, then remote into each server and inject the cert, and reconfigure their services.

Abandon local TLS entirely for Cloudflare Argo Tunnel:

1. Cloudflare Argo Tunnel only allows connections from cloudflare servers and users you give access to via Cloudflare Access. Argo Tunnel also encrypted all traffic by generating a local certificate on the machine that then interfaces with the Cloudflare endpoint. Accomplished by #8

Assessment:

Local polling allows short TTL on local TLS. Accomplishes #4

Local polling AND dev issuance in pre, would involve reloading for all existing servers, when each new server added.

Dev issuance in post, would involve reloading for all servers, but only once in post.

Reloading may induce downtime if not timed to be simultaneous.

Conclusion:

Argo Tunnel should be explored. It could turn out to be easiest and most secure. And may turn out to be able to be used with service TLS.

At a later point, implement service TLS. It would require 1-3 weeks by estimate to get the options for it going.

Notable Changes

Links

• Terraform
  • Current Version: 0.11.3
  • Latest Version: 0.15.0 (Unreleased)
  • Changelog
    • 0.12
    • 0.13
    • 0.14
    • 0.15
  • Notable Links
    • https://www.terraform.io/upgrade-guides/0-12.html
  • Notable Providers
    • https://github.com/cloudflare/terraform-provider-cloudflare
• Consul
  • Current Version: 1.0.7
  • Latest Version: 1.9.0
  • Changelog
    • 1.1.0
    • 1.2.0
    • 1.3.0
    • 1.4.0
    • 1.5.0
    • 1.6.0
    • 1.7.0
    • 1.8.0
    • 1.9.0
  • Notable Links
    • https://www.hashicorp.com/blog/consul-1-2-service-mesh
    • https://www.consul.io/use-cases/multi-platform-service-mesh
    • https://learn.hashicorp.com/tutorials/consul/gossip-encryption-secure?in=consul/security-networking
    • https://learn.hashicorp.com/tutorials/consul/tls-encryption-secure?in=consul/security-networking
    • https://learn.hashicorp.com/tutorials/consul/access-control-setup-production
    • https://www.consul.io/docs/connect/gateways/mesh-gateway
    • https://www.consul.io/docs/connect/gateways/mesh-gateway/wan-federation-via-mesh-gateways
• Vault
  • Current Version: 0.10.1
  • Latest Version: 1.6.0 RC
  • Changelog
    • 1.0.0
    • 1.1.0
    • 1.2.0
    • 1.3.0
    • 1.4.0
    • 1.5.0
  • Notable Links
    • https://www.vaultproject.io/docs/concepts/integrated-storage
    • https://www.vaultproject.io/docs/concepts/seal
    • https://docs.atlas.mongodb.com/tutorial/manage-programmatic-access/
    • https://github.com/hashicorp/vault-plugin-database-couchbase
• Nomad
  • Current Version: 0.8.1
  • Latest Version: 1.0.0 (Unreleased)
  • Changelog
    • 0.9.0
    • 0.10.0
    • 0.11.0
    • 0.12.0
  • Notable Links
    • https://www.nomadproject.io/docs/integrations/consul-connect
    • https://www.nomadproject.io/docs/integrations/vault-integration
• Boundary
  • Current Version: not yet used
  • Latest Version: 0.1.1
  • Changelog
• Upgrade Issues:
  • automate tls
  • nomad tls
  • tls certificate issuer and updater
  • terraform upgrade
  • use boundary

Terraform

HCL2 solves a lot of the issues we were having.

Terraform Cloud also solves the encrypted/secure env var issues.

Terraform Modules

• consul
• nomad
• vault

HashiCorp now has full starter kits for a secure environment on AWS. If only they did this when this project was started, it would have saved me months of figuring out how to do it securely and integrated myself.

Consul, Vault, Nomad

The new versions support federation and automatic TLS, which was the most difficult part of this entire setup back when this project was being authored. Upgrading to them should save a lot of time.

Boundary

Boundary is a new project that simplifies ssh and access into the machines.

https://www.boundaryproject.io