chanzuckerberg / cztack Goto Github PK

The CZI infrastructure stack.

Home Page: https://www.chanzuckerberg.com/

License: MIT License

HCL 76.44% Shell 2.37% Go 20.11% Python 0.13% Makefile 0.96%

cztack's Introduction

CZtack

// bump Cztack (pronounced "stack") is CZI's collection of Terraform modules. We use these as way to scale our infrastructure work.

These modules are compatible with Terraform 0.12 and up.

Contributing

Making a release

Release-please managages all the releases. To make a release, merge the release PR.

Adding a new module

To create a new module, copy the module-template directory and modify as you see fit. And make sure to add the module to the list of modules to test in .github/workflows/ci.yml.

Writing tests

A few notes on writing test for this repo. Note that this is new ground for us, so this will be a work in progress.

To make modules testable, all fields that have a unique constraint need to be parameterizeable. Otherwise concurrent tests will conflict.
It is tempting in testing module A to use module B to set up some context, but because terraform will just store the statefile locally, you can have a conflict.
- We've tried to avoid this for now and set up context more directly
- And also to not run tests in parrallel
- and to clean up state files before and after each run
our linter requires a test for each module. At the very least run init so that its syntax is checked. See an example here.
AWS IAM is eventually consistent and supposedly is homed in us-east-1, so its probably best to run all tests that use IAM in that region.

Test Dependencies

go
terraform
terraform-provider-bless - Place it in the same directory as your terraform executable. dirname \which terraform``

cztack's People

Contributors

Stargazers

Watchers

cztack's Issues

figure out why we only do ipv6 if we are doing redirects

cztack/aws-ecs-service/domain.tf

Line 25 in d2f1501

resource "aws_route53_record" "ipv6" {

aws ecs service module not reading source_security_group_id

We are using the aws-ecs-service module and saw this error with the configuration, from the source code it looks like the security group for the container should be referenced in the alb, but somehow an error was raised for not setting it. (can provide links to our terraform code and run history via direct message)

raw output from terraform:
Error: One of ['cidr_blocks', 'ipv6_cidr_blocks', 'self', 'source_security_group_id', 'prefix_list_ids'] must be set to create an AWS Security Group Rule

on .terraform/modules/napari-hub.web-service.alb-sg/main.tf line 469, in resource "aws_security_group_rule" "egress_with_source_security_group_id":
469: resource "aws_security_group_rule" "egress_with_source_security_group_id" {

link all tagging docs

aws-redis-node apply test

Support Security Groups for RDS Security Groups in our RDS Modules

We support CIDR ranges for access control when defining a DB, but we should really support security groups so that small services don't have their DBs exposed to the whole VPC.

Example: https://github.com/chanzuckerberg/shared-infra/blob/bb21c424b5eb1bea14a3a995529a3764f1a218da/terraform/modules/aurora-postgres/main.tf#L24

Moved from https://github.com/chanzuckerberg/shared-infra/issues/332

aws-aurora-mysql apply test

acm-cert

Terraform modules snowflake-XXX-grant-all should not have OWNERSHIP privileges

All the terraform modules have in their privileges list the OWNERSHIP role, for example on the snowflake-warehouse-grant-all module :

"privileges": [
"MODIFY",
"MONITOR",
"OPERATE",
"OWNERSHIP",
"USAGE"
]

This is incorrect, because the actual result of the Snowflake command GRANT ALL ON WAREHOUSE TO ROLE will give the following privileges : MODIFY, MONITOR, OPERATE and USAGE.

add examples to module docs

move aurora modules

figure out naming scheme for params stuff

cloudwatch log group

move redis-node

Snowflake grant-all modules have conflicting shares and on_future

This is the issue brought up in this discussion topic: Snowflake-Labs/terraform-provider-snowflake#318

@jhettler @ryanking

release 0.10.0

cztack CI

role infraci source_account should be source_account_id

fogg.json as suggested does not work

I wasn't able to get the provider working by just defining it in terraform, so I tried the suggested workflow with fogg in an empty repo:

user@MacBook-Pro bless-test % fogg apply                                 
fogg.yml has error(s):
ERROR: could not open fogg.yml config file
user@MacBook-Pro bless-test % fogg version
0.76.0

The docs suggested I use these contents for fogg.json:

{
  "defaults":{},
  "modules":{},
  "accounts":{},
  "envs":{},
  "plugins": {
    "terraform_providers": {
      "terraform-provider-bless": {
        "url": "https://github.com/chanzuckerberg/terraform-provider-bless/releases/download/v0.2.5/terraform-provider-bless_0.2.5_linux_amd64.tar.gz",
        "format": "tar"
      }
    }
  }
}

I'm guessing v0.76 requires a yaml and not json anymore, so I tried converting the suggested json to fogg.yaml:

defaults: {}
modules: {}
accounts: {}
envs: {}
plugins:
    terraform_providers:
        terraform-provider-bless:
            url: 'https://github.com/chanzuckerberg/terraform-provider-bless/releases/download/v0.2.5/terraform-provider-bless_0.2.5_linux_amd64.tar.gz'
            format: tar

This also produced errors, so I'm guessing the specified information is not sufficient to get started.

user@MacBook-Pro bless-test % fogg apply    
fogg.yml has error(s):
UNKNOWN ERROR: 4 errors occurred:
        * Key: 'Config.Version' Error:Field validation for 'Version' failed on the 'required' tag
        * global must have a valid owner set at either the global or defaults level
        * global must have a valid project set at either the global or defaults level
        * global must have a valid terraform version set at either the global or defaults level

aws-iam-ecs-task-role should take either policy text or policy arn

Currently it takes policy which is the text of a policy to be created. It should take either that or a policy_arn.

cztack/aws-iam-ecs-task-role/variables.tf

Line 25 in fbc1fa1

variable "policy" {

Provider invalid for bless module: "Failed to query available provider packages"

Thanks for running this project.
When getting started today for the first time with the provided example, and defining the provider like so:

terraform {
  required_providers {
    bless = {
      source = "registry.terraform.io/chanzuckerberg/bless"
      version = "~> 0.5"
    }
  }
  required_version = ">= 0.13"
}

I get an error on terraform init:

│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider hashicorp/bless: provider registry registry.terraform.io does not have a provider
│ named registry.terraform.io/hashicorp/bless
│ 
│ Did you intend to use chanzuckerberg/bless? If so, you must specify that source address in each module which requires that provider. To see
│ which modules are currently depending on hashicorp/bless, run the following command:
│     terraform providers
╵

It appears the bless module is pointing at an incorrect url for the intended provider:

Providers required by configuration:
.
├── provider[registry.terraform.io/chanzuckerberg/bless] ~> 0.5
├── provider[registry.terraform.io/hashicorp/null] ~> 3.0
├── provider[registry.terraform.io/hashicorp/aws]
└── module.bless
    ├── provider[registry.terraform.io/hashicorp/random]
    ├── provider[registry.terraform.io/hashicorp/bless]
    ├── provider[registry.terraform.io/hashicorp/aws]
    ├── module.lambda
    │   └── provider[registry.terraform.io/hashicorp/aws] >= 3.0.0
    └── module.logs_policy
        └── provider[registry.terraform.io/hashicorp/aws]