We are excited to announce the release of the latest version of the Terraform Cloud Operator for Kubernetes, now available at hashicorp/terraform-cloud-operator. This update introduces a more streamlined set of CRDs, along with new features such as Agent Pools, the capability to deploy and scale agents, Run Tasks, and Workspace Notifications. To learn more about these enhancements and to get started, please checkout our announcement blog
The Terraform Cloud Operator for Kubernetes provides first-class integration between Kubernetes and Terraform Cloud by extending the Kubernetes control plane to enable lifecycle management of cloud and on-prem infrastructure through Kubernetes manifests. Manifests can be deployed and managed using kubectl, Terraform, Gitops tools, or any other tool that allows you to manage Kubernetes custom resources.
This operator provides a unified way to manage a Kubernetes application and its infrastructure dependencies through a single Kubernetes CustomResourceDefinition (CRD). After the infrastructure dependencies are created, pertinent information such as endpoints and credentials are returned from Terraform Cloud to Kubernetes.
- Manage the lifecycle of cloud and on-prem infrastructure through a single Kubernetes custom resource
- Install the operator from the corresponding Helm Chart to enable the management of infrastructure services from any Kubernetes cluster.
- Provision and manage infrastructure from any provider, such as AWS, Azure, GCP, and any of the hundreds of other Terraform providers, to use them with your existing application configurations, through Terraform Cloud or Terraform Enterprise.
- Deploy and Manage your Kubernetes and infrastructure resources in a single git repository, separate git repositories, or directly from a module in the Terraform Registry, to match your existing operating model.
- Provide governance for your infrastructure resources using policy-as-code with OPA Gatekeeper and HashiCorp Sentinel.
You can read more about this project and its potential use cases on our blog.
Terraform also enables you to create and publish custom infrastructure providers through the Terraform SDK. Once you create a new Terraform provider, publish it to the Terraform Registry and then you can consume it with the operator.
Join us in the #terraform-providers channel on the Kubernetes Slack to discuss this, and other Terraform and Kubernetes projects (Sign up here).
Note: This project is versioned separately from Terraform. Supported Terraform versions must be version 0.12 or above. By versioning this project separately, we can iterate on Kubernetes integrations more quickly and release new versions without forcing Terraform users to do a full Terraform upgrade.
We take Terraform's security and our users' trust very seriously. If you believe you have found a security issue in the Terraform Cloud Operator for Kubernetes, please responsibly disclose by contacting us at [email protected].
Create the namespace where you will deploy the operator, Secrets, and Workspace resources.
$ kubectl create ns $NAMESPACEThe operator must authenticate to Terraform Cloud. Note that the operator must run within the cluster, which means that it already handles Kubernetes authentication.
-
Generate a Terraform Cloud Team API token at
https://app.terraform.io/app/$ORGANIZATION/settings/teams, where$ORGANIZATIONis your organization name. -
Create a file for storing the API token and open it in a text editor.
-
Insert the generated token (
$TERRAFORM_CLOUD_API_TOKEN) into the text file formatted for Terraform credentials.credentials app.terraform.io { token = "$TERRAFORM_CLOUD_API_TOKEN" }
-
Create a Kubernetes secret named
terraformrcin the namespace. Reference the credentials file ($FILENAME) created in the previous step.$ kubectl create -n $NAMESPACE secret generic terraformrc --from-file=credentials=$FILENAME
Ensure
terraformrcis the name of the secret, as it is the default secret name defined under the Helm valuesyncWorkspace.terraformRC secretNamein thevalues.yamlfile.
If you have the free tier of Terraform Cloud, you will only be able to generate a token for the one team associated with your account. If you have a paid tier of Terraform Cloud, create a separate team for the operator with "Manage Workspaces" access.
Note that a Terraform Cloud Team API token is a broad-spectrum token. It allows the token holder to create workspaces and execute Terraform runs. You cannot limit the access it provides to a single workspace or role within a team. In order to support a first-class Kubernetes experience, security and access control to this token must be enforced by Kubernetes Role-Based Access Control (RBAC) policies.
Sensitive variables in Terraform Cloud workspaces often take the form of credentials for cloud providers or API endpoints. They enable Terraform Cloud to authenticate against a provider and apply changes to infrastructure.
Create the secret for the namespace that contains all of the sensitive variables required for the workspace.
$ kubectl create -n $NAMESPACE secret generic workspacesecrets --from-literal=SECRET_KEY=$SECRET_KEY --from-literal=SECRET_KEY_2=$SECRET_KEY_2 ...Ensure workspacesecrets is the name of the secret, as it is the default secret name defined under the Helm value syncWorkspace.sensitiveVariables.secretName in the values.yaml file.
In order to support a first-class Kubernetes experience, security and access control to these secrets must be enforced by Kubernetes Role-Based Access Control (RBAC) policies.
By default, the operator will create a Terraform Cloud workspace with a pinned version of Terraform.
Override the Terraform version that will be set for the workspace by changing the Helm value syncWorkspace.terraformVersion to the Terraform version of choice.
Use the Helm chart repository to deploy the Terraform Operator to the namespace you previously created.
$ helm repo add hashicorp https://helm.releases.hashicorp.com
$ helm search repo hashicorp/terraform
$ helm install --namespace ${NAMESPACE} hashicorp/terraform --generate-nameThe Workspace CustomResource defines a Terraform Cloud workspace, including variables, Terraform module, and outputs.
Here are examples of Workspace CustomResource..
The Workspace Spec includes the following parameters:
-
organization: The Terraform Cloud organization you would like to use. -
secretsMountPath: The file path defined on the operator deployment that contains the workspace's secrets.
Additional parameters are outlined below.
The Workspace will only execute Terraform configuration in a module. It will not execute
*.tffiles.
Information passed to the Workspace CustomResource will be rendered to a template Terraform configuration that uses the module block. Specify a module with remote source. Publicly available VCS repositories, the Terraform Registry, and private module registry are supported. In addition to source, specify a module version.
module:
source: "hashicorp/hello/random"
version: "3.1.0"The above Kubernetes definition renders to the following Terraform configuration.
module "operator" {
source = "hashicorp/hello/random"
version = "3.1.0"
}Variables for the workspace must equal the module's input variables. You can define Terraform variables in two ways:
-
Inline
variables: - key: hello value: world sensitive: false environmentVariable: false
-
With a Kubernetes ConfigMap reference
variables: - key: second_hello valueFrom: configMapKeyRef: name: say-hello key: to sensitive: false environmentVariable: false
The above Kubernetes definition renders to the following Terraform configuration.
variable "hello" {}
variable "second_hello" {}
module "operator" {
source = "hashicorp/hello/random"
version = "3.1.0"
hello = var.hello
second_hello = var.second_hello
}The operator pushes the values of the variables to the Terraform Cloud
workspace. For secrets, set sensitive to be true. The workspace sets them as write-only. Denote workspace environment variables by setting environmentVariable as true.
Sensitive variables should already be initialized as per Workspace Sensitive Variables. You can define them by setting sensitive: true. Do not define the value or use a ConfigMap reference, as the read from file will override the value you set.
variables:
- key: AWS_SECRET_ACCESS_KEY
sensitive: true
environmentVariable: trueSSH keys can be used to clone private modules. To apply an SSH key to the workspace, specify sshKeyID in the Workspace Custom Resource. The SSH key ID can be found in the Terraform Cloud API.
apiVersion: app.terraform.io/v1alpha1
kind: Workspace
metadata:
name: $WORKSPACE
spec:
sshKeyID: $SSHKEYID
In order to retrieve Terraform outputs, specify the outputs
section of the Workspace CustomResource. The key represents the output key you expect from terraform output and moduleOutputName denotes the module's output key name.
outputs:
- key: my_pet
moduleOutputName: petThe above Kubernetes definition renders to the following Terraform configuration.
output "my_pet" {
value = module.operator.pet
}The values of the outputs can be consumed from two places:
- Kubernetes status of the workspace.
$ kubectl describe -n $NAMESPACE workspace $WORKSPACE_NAME
- ConfigMap labeled
$WORKSPACE_NAME-outputs. Kubernetes deployments can consume these output values.$ kubectl describe -n $NAMESPACE configmap $WORKSPACE_NAME-outputs
Deploy the workspace after configuring its module, variables, and outputs.
$ kubectl apply -n $NAMESPACE -f workspace.ymlThe following changes updates and executes new runs for the Terraform Cloud workspace:
organizationmodulesource or versionoutputs- Non-sensitive or ConfigMap reference
variables.
Updates to sensitive variables will not trigger a new execution because sensitive variables are write-only for security purposes. The operator is unable to reconcile the upstream value of the secret with the value stored locally. Similarly, ConfigMap references do not trigger updates as the operator does not read the value for comparison.
After updating the configuration, re-deploy the workspace.
$ kubectl apply -n $NAMESPACE -f workspace.ymlWhen deleting the Workspace CustomResource, the command line will wait for a few moments.
$ kubectl delete -n $NAMESPACE workspace.app.terraform.io/$WORKSPACE_NAMEThis is because the operator is running a finalizer. The finalizer will execute before the workspace officially deletes in order to:
- Stop all runs in the workspace, including pending ones
terraform destroy -auto-approveon resources in the workspace- Delete the workspace.
Once the finalizer completes, Kubernetes deletes the Workspace CustomResource.
Check the status and outputs of the workspace by examining its Kubernetes status. This provides the run ID and workspace ID to debug in the Terraform Cloud UI.
$ kubectl describe -n $NAMESPACE workspace $WORKSPACE_NAMEWhen workspace creation, update, or deletion fails, check errors by examining the logs of the operator.
$ kubectl logs -n $NAMESPACE $(kubectl get pods -n $NAMESPACE --selector "component=sync-workspace" -o jsonpath="{.items[0].metadata.name}")If Terraform Cloud returns an error that the Terraform configuration is incorrect, examine the Terraform configuration at its ConfigMap.
$ kubectl describe -n $NAMESPACE configmap $WORKSPACE_NAMEThe Helm chart does not include secrets management or injection. Instead, it expects to find secrets mounted as volumes to the operator's deployment. This supports secrets management approaches in Kubernetes that use a volume mount for secrets.
In order to support a first-class Kubernetes experience, security and access control to these secrets must be enforced by Kubernetes Role-Based Access Control (RBAC) policies.
For the Terraform Cloud Team API token, the entire credentials file with the Terraform Cloud API Token is mounted to the filepath specified by TF_CLI_CONFIG_FILE. In an equivalent Kubernetes configuration, the following example creates a Kubernetes secret and mount it to the operator at the filepath specified by TF_CLI_CONFIG_FILE.
---
# not secure secrets management
apiVersion: apps/v1
kind: Secret
metadata:
name: terraformrc
type: Opaque
data:
credentials: |-
credentials app.terraform.io {
token = "$TERRAFORM_CLOUD_API_TOKEN"
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: terraform-k8s
spec:
# some sections omitted for clarity
template:
metadata:
labels:
name: terraform-k8s
spec:
serviceAccountName: terraform-k8s
containers:
- name: terraform-k8s
env:
- name: TF_CLI_CONFIG_FILE
value: "/etc/terraform/.terraformrc"
volumeMounts:
- name: terraformrc
mountPath: "/etc/terraform"
readOnly: true
volumes:
- name: terraformrc
secret:
secretName: terraformrc
items:
- key: credentials
path: ".terraformrc"Similar to the Terraform Cloud API Token, the Helm chart mounts them to the operator's deployment for use. It does not mount workspace sensitive variables to the Workspace Custom Resource. This ensures that only the operator has access to read and create sensitive variables as part of the Terraform Cloud workspace.
Examine the deployment in templates/sync-workspace-deployment.yaml. The deployment mounts a volume containing the sensitive variables. The file name is the secret's key and file contents is the secret's value. This supports secrets management approaches in Kubernetes that use a volume mount for secrets.
---
# not secure secrets management
apiVersion: apps/v1
kind: Secret
metadata:
name: workspacesecrets
type: Opaque
data:
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
GOOGLE_APPLICATION_CREDENTIALS: ${GOOGLE_APPLICATION_CREDENTIALS}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: terraform-k8s
spec:
# some sections omitted for clarity
template:
metadata:
labels:
name: terraform-k8s
spec:
serviceAccountName: terraform-k8s
containers:
- name: terraform-k8s
volumeMounts:
- name: workspacesecrets
mountPath: "/tmp/secrets"
readOnly: true
volumes:
- name: workspacesecrets
secret:
secretName: workspacesecretsThe Helm chart consists of several components. The Kubernetes configurations associated with the Helm chart are located under crds/ and templates/.
Helm starts by deploying the Custom Resource Definition for the Workspace. Custom Resource Definitions extend the Kubernetes API. It looks for definitions in the crds/ of the chart.
The Custom Resource Definition under crds/app.terraform.io_workspaces_crd.yaml defines that the Workspace Custom Resource schema.
In order to scope the operator to a namespace, Helm assigns a role and service account to the namespace. The role has access to Pods, Secrets, Services, and ConfigMaps. This configuration is located in templates/.
To ensure the operator does not have access to secrets or resource beyond the namespace, the Helm chart scopes the operator's deployment to a namespace.
apiVersion: apps/v1
kind: Deployment
metadata:
name: terraform-k8s
spec:
# some sections omitted for clarity
template:
metadata:
labels:
name: terraform-k8s
spec:
serviceAccountName: terraform-k8s
containers:
- name: terraform-k8s
command:
- /bin/terraform-k8s
- "--k8s-watch-namespace=$(POD_NAMESPACE)"
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespaceWhen deploying, ensure that the namespace is passed into the
--k8s-watch-namespace option. Otherwise, the operator will attempt to access across all namespaces (cluster scope).
![hashicorp-copywrite[bot] avatar](https://avatars.githubusercontent.com/u/761456?v=4 "hashicorp-copywrite[bot]")
![hashicorp-tsccr[bot] avatar](https://avatars.githubusercontent.com/in/312369?v=4 "hashicorp-tsccr[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent