flux-iac / tofu-controller Goto Github PK

We should use compress/gzip to compress the byte content of tfplan so that it's possible for us to store a larger plan, as the current limitation of a Secret is 1 MB.

When storing tfplan, we must add an annotation to tell that we use GZIP to encode the content.

metadata:
   annotations:
     encoding: gzip

When setting .spec.approvePlan="disable",
the plan and the apply process would be skipped.

This mode is call the "drift detection" only mode.

The value of the Ready condition must be one of these:

1. Reason="NoDrift", Status=True
2. Reason="DriftDetected", Status=False

The loop is still performed every .spec.interval duration, but no plan or apply action.

The .spec.approvePlan="disable" behavior must have the higher precedence than .spec.force=true to prevent a bad thing to happen.

Implement runner main program and package it into another container image.

The base manifests should use a ServiceAccount specific to this controller rather than the default ServiceAccount fo the namespace.

Terraform Runner is a sub system of TF-controller.
It is responsible for running Terraform commands at the different stages.
Terraform Runner is a part of supporting multi-tenancy #43

Requirements

• It must be an gRPC server wrapping around TFExec #68
• Its local version must be able to use inside the TF-controller, and all current tests pass #68
• the gRPC server must support executing a command for each step in the life-cycle #68
  • receive tar.gz
  • init
  • plan for drift detection
  • plan
  • apply
  • show
  • output
• Separate runner as a Pod #70
• #71
• #72
• #73
• #105
• #106

High Level Diagram

Implement a drift detection logic.

Problem

Source change not trigger the re-plan. The current behaviour is that the controller wait for pending plan forever

How to fix

When revison != last attempted revision, it means that source is changed.
And if there's no intend to apply, it's safe to clear pending plan.

There's a condition that,

• change was made.
• TF source was actually not touched
• the previous plan was applied
• but the current revision was attempted

 Status:                                                                                                                                                                │
│   Conditions:                                                                                                                                                          │
│     Last Transition Time:   2022-01-05T17:19:05Z                                                                                                                       │
│     Message:                Terraform Plan No Changed                                                                                                                  │
│     Reason:                 TerraformPlannedNoChanges                                                                                                                  │
│     Status:                 True                                                                                                                                       │
│     Type:                   Ready                                                                                                                                      │
│     Last Transition Time:   2022-01-05T17:12:38Z                                                                                                                       │
│     Message:                Terraform Plan No Changed                                                                                                                  │
│     Reason:                 TerraformPlannedNoChanges                                                                                                                  │
│     Status:                 False                                                                                                                                      │
│     Type:                   Plan                                                                                                                                       │
│     Last Transition Time:   2022-01-05T16:45:14Z                                                                                                                       │
│     Message:                Terraform Applied Successfully                                                                                                             │
│     Reason:                 TerraformAppliedSucceed                                                                                                                    │
│     Status:                 True                                                                                                                                       │
│     Type:                   Apply                                                                                                                                      │
│   Last Applied Revision:    main/32b2b08efbdf32662a535248a7cfbdb81060a32a                                                                                              │
│   Last Attempted Revision:  main/9ff16f4778bcf56afaffda7aede6e2d482a3df38                                                                                              │
│   Observed Generation:      4                                                                                                                                          │
│   Plan:                                                                                                                                                                │
│     Last Applied:  plan-main-32b2b08efbdf32662a535248a7cfbdb81060a32a 

need to have a decision on how to deal with this kind of behavior.

is EKS IRSA possible for authentication to aws-api instead of using static credentials?

• Prepare a security checklist to review for every release
• Prepare a process for discussing the security aspect for all features, including the new ones being added.

Part of #59

Requirements

• It must be an gRPC server wrapping around TFExec
• Its local version must be able to use inside the TF-controller, and all current tests pass.
• the gRPC server must support executing a command for each step in the life-cycle
  • receive tar.gz
  • init
  • plan for drift detection
  • plan
  • apply
  • show
  • output

To guide new contributors to get local environment set up and start contributing quickly

relates to #72

We currently could do the following to import a TFSTATE for a specify Terraform object.

gzip terraform.tfstate

NAME=tf-controller-ng-cc5a1ac4
kubectl create secret \
  generic tfstate-default-${NAME} \
  --from-file=tfstate=terraform.tfstate.gz \
  --dry-run=client -o=yaml \
  yq e '.metadata.annotations["encoding"]="gzip"' - > tfstate-default-${NAME}.yaml
  
kubectl apply -f tfstate-default-${NAME}.yaml

Need a good UX / design / approach to tackle this feature.

The new .spec.disableDriftDetection requires a full specification as the following.

• a Terraform object with "auto" mode performs "plan" and "apply", with interval like 10s. We set 'disableDriftDetection" to "true"
• then we wait 20s.
• the Ready condition should NOT be changed to "NoDrift", as the detection is disabled.

Seeing plan details is an very important feature for the plan step.
We need to have a mechanism to show plan details somewhere (via a notification? via a PR?)

Like this: https://github.com/fluxcd/kustomize-controller/blob/main/docs/api/kustomize.md

Relate to #34

After #72

To help with a correct variable generation, we need a flag to specify that a variable is an HCL expression. It's a string by default.

Some resources might be very expensive to detect drift, for example a large cluster or complex Terraform resources.

It would be great to have a to disable drift detection.

.spec.disableDriftDetection would be suffice to support this feature.

https://github.com/aquasecurity/trivy

From @phoban01 on #44

Might want to consider adding a successThreshold and failureThreshold to avoid transient network errors triggering health-check failures?

Generally, the application of tcp/http healthchecks could be limited, as it might often be the case that there's no network connectivity between the resources being created and the tf-controller. If outputs could be passed to exec then something like the following would be more practical:

healthChecks:
- name: bucket
  exec: "aws s3api head-bucket --bucket {{ outputs.bucket_name }}"

Just the only test case I've seen failing repeatedly across job runs. Need a proper way to fix it.

Relate to #11

Upgrade The Source controller to v0.21.1
https://github.com/fluxcd/source-controller/releases/tag/v0.21.1

is it an option to move the tf-controller with examples, helm-chart to https://github.com/fluxcd-community ?

Health check for external resources like Redis, RDS or other managed services like Mongo Atlas requires an extra step to perform a TCP connection to ensure that the target service is already up and running.

The Flagger project has some generic way to declare this test step, from which we could learn.

Our resources maybe something like TCP services, HTTP services. Here's a health check spec proposal.

apiVersion: infra.contrib.fluxcd.io/v1alpha1
kind: Terraform
spec:
  path: ./terraform
  healthChecks:
  - name: rds
    kind: TCP
    address: ${output.rds_ip_address}:3306
    timeout: 15m
  - name: nginx
    kind: HTTP
    address: https://${output.nginx_ip_address}/ping
    timeout: 15m