This is a demo of using Boundary, Consul, and Vault to secure an application on Kubernetes.

Boundary controls user access to databases and test endpoints. Consul secures service-to-service communication. Vault secures the Consul cluster and issues temporary credentials for an application to access a database

• Navigation
• Prerequisites
• Setup
• Explore
• Clean Up
• Credits
• Additional References

Each folder contains a few different configurations.

• Terraform Configurations

  • infrastructure/: All the infrastructure to run the system.

    • VPC (3 private subnets, 3 public subnets)
    • Boundary cluster (controllers, workers, and AWS RDS PostgreSQL database)
    • AWS Elastic Kubernetes Service cluster
    • AWS RDS (PostgreSQL) database for demo application
    • HashiCorp Virtual Network (peered to VPC)
    • HCP Consul
    • HCP Vault

  • boundary: Configures Boundary with two projects, one for operations and the other for development teams.

  • vault/setup/: Deploy a Vault cluster via Helm chart and set up Kubernetes auth method

  • certs/: Sets up offline root CA and signs intermediate CA in Vault for Consul-related certificates.

  • vault/consul/: Set up Consul-related secrets engines.

  • consul/setup/: Deploys a Consul cluster via Helm chart. For demonstration of Vault as a secrets backend, deploys Consul servers + clients.

  • consul/config/: Sets up external service to database.

  • vault/app/: Set up secrets engines for applications. Archived in favor of consul/cts/.

• Other

  • consul/cts/: Deploys CTS to Kubernetes for setting up Vault database secrets based on database service's address

  • application/: Deploys the HashiCorp Demo Application (AKA HashiCups) to Kubernetes

  • database/: Configures HashiCorp Demo Application database

• Terraform 1.2.2
• Consul 1.12.0 (on Kubernetes)
• HashiCorp Cloud Platform (HCP) Vault 1.10.3
• HashiCorp Cloud Platform (HCP) Consul 1.11.6
• Boundary 0.8.1

• Terraform Cloud
• AWS Account
  • Create an AWS EC2 keypair.
• HashiCorp Cloud Platform account
  • You need access to HCP Consul and Vault.
  • Create a service principal for the HCP Terraform provider.
• jq installed
• Fork this repository.

Note: When you run this, you might get the error Provider produced inconsistent final plan. This is because we're using default_tags. Re-run the plan and apply to resolve the error.

First, set up the Terraform workspace.

1. Create a new Terraform workspace.
2. Choose "Version control workflow".
3. Connect to GitHub.
4. Choose your fork of this repository.
5. Name the workpsace infrastructure.
6. Select the "Advanced Options" dropdown.
7. Use the working directory infrastructure.
8. Select "Create workspace".

Next, configure the workspace's variables.

1. Variables should include:

   • client_cidr_block (sensitive): list including the public IP address of your machine, in [00.00.00.00/32] form. You get it by running curl ifconfig.me in your terminal.

2. Environment Variables should include:

   • HCP_CLIENT_ID: HCP service principal ID
   • HCP_CLIENT_SECRET (sensitive): HCP service principal secret
   • AWS_ACCESS_KEY_ID: AWS access key ID
   • AWS_SECRET_ACCESS_KEY (sensitive): AWS secret access key
   • AWS_SESSION_TOKEN (sensitive): If applicable, the token for session

If you have additional variables you want to customize, including region, make sure to update them in the infrastructure/terraform.auto.tfvars file.

Finally, start a new plan and apply it. It can take more than 15 minutes to provision!

First, set up the Terraform workspace.

1. Create a new Terraform workspace.
2. Choose "Version control workflow".
3. Connect to GitHub.
4. Choose your fork of this repository.
5. Name the workpsace boundary.
6. Select the "Advanced Options" dropdown.
7. Use the working directory boundary.
8. Select "Create workspace".

Next, configure the workspace's variables. This Terraform configuration retrieves a set of variables using terraform_remote_state data source.

1. Variables should include:

   • tfc_organization: your Terraform Cloud organization name

2. Environment Variables should include:

   • AWS_ACCESS_KEY_ID: AWS access key ID
   • AWS_SECRET_ACCESS_KEY (sensitive): AWS secret access key
   • AWS_SESSION_TOKEN (sensitive): If applicable, the token for session

Queue to plan and apply. This creates an organization with two scopes:

• core_infra, which allows you to SSH into EKS nodes
• product_infra, which allows you to access the PostgreSQL database

Only product users will be able to access product_infra. operations users will be able to access both core_infra and product_infra.

First, set up the Terraform workspace.

1. Create a new Terraform workspace.
2. Choose "Version control workflow".
3. Connect to GitHub.
4. Choose your fork of this repository.
5. Name the workpsace vault-setup.
6. Select the "Advanced Options" dropdown.
7. Use the working directory vault/setup.
8. Select "Create workspace".

Next, configure the workspace's variables. This Terraform configuration retrieves a set of variables using terraform_remote_state data source.

1. Variables should include:

   • tfc_organization: your Terraform Cloud organization name

2. Environment Variables should include:

   • HCP_CLIENT_ID: HCP service principal ID
   • HCP_CLIENT_SECRET (sensitive): HCP service principal secret
   • AWS_ACCESS_KEY_ID: AWS access key ID
   • AWS_SECRET_ACCESS_KEY (sensitive): AWS secret access key
   • AWS_SESSION_TOKEN (sensitive): If applicable, the token for session

Terraform will set up Kubernetes authentication method and deploy the Vault Helm chart to the cluster.

As a best practice, store root CAs away from Vault. To demonstrate this, we generate a root CA offline. We use three separate root CAs:

• Cluster Root CA

  • Level 1 Intermediate CA (server root)
  • Level 2 Intermediate CA (server intermediate)

• Service Mesh Root CA for mTLS: This requires three levels because we will need to reconfigure the CA for the correct SPIFFE URI.

  • Level 1 Intermediate CA
  • Level 2 Intermediate CA (service mesh root)
  • Level 3 Intermediate CA (service mesh intermediate)

• API Gateway Root CA

  • Level 1 Intermediate CA (gateway root)
  • Level 2 Intermediate CA (gateway intermediate)

NOTE: This is a local Terraform command in order to secure the offline root CA.

Run the command to create a root CA as well as the intermediate CAs, and store the intermediate CAs in Vault.

make configure-certs

First, set up the Terraform workspace.

1. Create a new Terraform workspace.
2. Choose "Version control workflow".
3. Connect to GitHub.
4. Choose your fork of this repository.
5. Name the workpsace vault-consul.
6. Select the "Advanced Options" dropdown.
7. Use the working directory vault/consul.
8. Select "Create workspace".

Next, configure the workspace's variables. This Terraform configuration retrieves a set of variables using terraform_remote_state data source.

1. Variables should include:

   • tfc_organization: your Terraform Cloud organization name`

2. Environment Variables should include:

   • HCP_CLIENT_ID: HCP service principal ID
   • HCP_CLIENT_SECRET (sensitive): HCP service principal secret

Terraform will set up the PKI secrets engine for TLS in the Consul cluster (not the service mesh).

Using kustomize, deploy the Gateway CRDs.

make configure-kubernetes

Then, set up the Terraform workspace.

1. Create a new Terraform workspace.
2. Choose "Version control workflow".
3. Connect to GitHub.
4. Choose your fork of this repository.
5. Name the workpsace consul-setup.
6. Select the "Advanced Options" dropdown.
7. Use the working directory consul/setup.
8. Select "Create workspace".

Next, configure the workspace's variables. This Terraform configuration retrieves a set of variables using terraform_remote_state data source.

1. Variables should include:

   • tfc_organization: your Terraform Cloud organization name

2. Environment Variables should include:

   • HCP_CLIENT_ID: HCP service principal ID
   • HCP_CLIENT_SECRET (sensitive): HCP service principal secret
   • AWS_ACCESS_KEY_ID: AWS access key ID
   • AWS_SECRET_ACCESS_KEY (sensitive): AWS secret access key
   • AWS_SESSION_TOKEN (sensitive): If applicable, the token for session

3. Queue to plan and apply. This deploys Consul clients and a terminating gateway via the Consul Helm chart to the EKS cluster to join the HCP Consul servers.

The Helm chart will get stuck because of this issue. Patch the API gateway to resolve.

make configure-api-gateway

API Gateway requires a SPIFFE-compliant URI in the service mesh certificate. To bypass this issue, you will need to reconfigure the root CA with a SPIFFE URI that contains the correct Consul cluster ID.

make configure-certs-spiffe

This forces a root certificate rotation for Consul service mesh.

Update the terminating gateway with a write policy to the database.

make configure-terminating-gateway

Then, set up the Terraform workspace.

1. Create a new Terraform workspace.
2. Choose "Version control workflow".
3. Connect to GitHub.
4. Choose your fork of this repository.
5. Name the workpsace consul-config.
6. Select the "Advanced Options" dropdown.
7. Use the working directory consul/config.
8. Select "Create workspace".

Next, configure the workspace's variables. This Terraform configuration retrieves a set of variables using terraform_remote_state data source.

1. Variables should include:

   • tfc_organization: your Terraform Cloud organization name

2. Environment Variables should include:

   • AWS_ACCESS_KEY_ID: AWS access key ID
   • AWS_SECRET_ACCESS_KEY (sensitive): AWS secret access key
   • AWS_SESSION_TOKEN (sensitive): If applicable, the token for session

3. Queue to plan and apply. This does a few things, including:

   • registers the database as an external service to Consul
   • deploys the Consul API Gateway
   • sets up the application intentions.

To add data, you need to log into the PostgreSQL database. However, it's on a private network. You need to use Boundary to proxy to the database.

1. Set up all the variables you need in your environment variables.

   source set_terminal.sh

2. Run the following commands to log in and load data into the products database.

   make configure-db

If you try to log in as a user of the products team, you can print out the tables.

make postgres-products

You can use Consul-Terraform-Sync to read the database address from Consul and automatically configure a database secrets engine in Vault using a Terraform module.

To do this, deploy CTS to Kubernetes.

make configure-cts

To deploy the example application, run make configure-application.

You can check if everything by checking the pods in Kubernetes.

$ kubectl get pods

NAME                                                          READY   STATUS    RESTARTS   AGE
## omitted for clarity
frontend-5d7f97456b-2fznv                      2/2     Running   0          15m
nginx-59c9dbb9ff-j9xhc                         2/2     Running   0          15m
payments-67c89b9bc9-kbb9r                      2/2     Running   0          16m
product-55989bf685-ll5t7                       3/3     Running   0          5m5s
public-64ccfc4fc7-jd7v7                        2/2     Running   0          8m17s

Port forward the nginx service to http://localhost:8080.

kubectl port-forward svc/nginx 8080:80

You'll get a UI where you can order your coffee.

To set up a route on the API gateway, deploy an HTTPRoute.

make configure-route

To use Boundary, use your terminal in the top level of this repository.

1. Set the BOUNDARY_ADDR environment variable to the Boundary endpoint.

   source set_terminal.sh

2. Use the example command in top-level Makefile to SSH to the EKS nodes as the operations team.

   make ssh-operations

3. Go to the Boundary UI and examine the "Sessions". You should get an active session in the Boundary list because you accessed the EKS node over SSH.

Delete applications.

make clean-application

Revoke Vault credentials for applications.

make clean-vault

Disable CTS task, remove resources, and delete CTS.

make clean-cts

Go into Terraform Cloud and destroy resources for the consul-config workspace.

Remove additional Consul resources.

make clean-consul

Go into Terraform Cloud and destroy resources for the consul-setup workspace.

Remove API Gateway manifests.

make clean-kubernetes

Go into Terraform Cloud and destroy resources for the vault-consul workspace.

Remove certificates for Consul from Vault.

make clean-certs

Go into Terraform Cloud and destroy resources for the vault-setup workspace.

Go into Terraform Cloud and destroy resources for the boundary workspace.

Go into Terraform Cloud and destroy resources for the infrastructure workspace.

• The demo application comes from the HashiCorp Demo Application.

• portal.cloud.hashicorp.com/sign-up
• consul.io/docs/k8s/installation/vault
• vaultproject.io/docs/secrets/pki
• consul.io/docs/nia
• vaultproject.io/docs/auth/kubernetes
• consul.io/docs/security/acl/auth-methods/kubernetes
• hashi.co/k8s-vault-consul
• hashi.co/k8s-consul-api-gateway