1. Learn how to provision computing resources for running Big Data analyses using the Infrastructure as Code (IaC) approach.
2. Learn how to set up opinionated CI/CD pipelines to deploy cloud infrastructure.
3. Learn how to utilize linters for detecting security vulnerabilities in cloud infrastructure.
4. Learn how to run Apache Spark code in a distributed way on Hadoop cluster using Vertex AI notebooks and Dataproc services on GCP.
5. Learn how to use Workload Identity Federation for a secure authentication from GitHub Actions to Google Cloud.

• Google Cloud SDK ~>424.0.0
• gsutil ~>5.21
• pre-commit ~>2.15.0
• Terraform ~>1.4.0
• Python ~>3.8
• Linux/MacOS
• pre-commit-terraform dependencies

• Redeem a GCP coupon to create a billing account
• Authenticate to GCP to obtain the default credentials used for running the code

# first remove the stored credentials if exist
gcloud auth application-default revoke
# login and get the new application credentials
gcloud auth application-default login

1. Export shared environment variables

export TF_VAR_tbd_semester=2023L
# format: 20xx for teachers, student ID number for students 
export TF_VAR_user_id=2003
# use your own billing account id
export TF_VAR_billing_account=016F99-F0B167-9A895D

2. Enter bootstrap folder then init project and Terraform state bucket

cd bootstrap
terraform init
terraform apply
cd ..

3. CI/CD (Github Actions setup using Workload Identity Federation)

• Edit env/backend.tfvars file and set bucket variable with the Terraform state bucket
• Edit env/project.tfvars file and set project_name, iac_service_account variables using the output from the bootstrap phase, e.g.:
• Edit cicd_bootstrap/conf/github_actions.tfvars to set github_org and github_repo, e.g.:

  github_org  = "mwiewior"
  github_repo = "tbd-workshop-1-public"

• Init state file and set env variables

cd cicd_bootstrap
terraform init -backend-config=../env/backend.tfvars

• Apply

# authenticate Docker backend with GCP
gcloud auth configure-docker
# create CI/CD integration using Workload Identity
terraform apply -var-file ../env/project.tfvars -var-file conf/github_actions.tfvars -compact-warnings
cd ..

4. Use output variables for configuring Github Actions workflow: .github/workflows/pull-request.yml,e.g. : Please do not edit and hardcode these values in a YAML but set the Github Actions secrets instead while preserving the secret names, i.e. GCP_WORKLOAD_IDENTITY_PROVIDER_NAME and GCP_WORKLOAD_IDENTITY_SA_EMAIL.

5. Install and configure pre-commit

pre-commit install

6. Run all linters locally:

pre-commit run --all-files --config .pre-commit-config.yaml --verbose

TASKS to do

• If pre-commit linters report any issues please try to fix them 🛠️.

(base) ➜  tbd-workshop-1-public git:(feat/init-workshop-1) ✗ pre-commit run --all-files --config .pre-commit-config.yaml
Terraform fmt............................................................Passed
Terraform validate.......................................................Passed
Terraform docs...........................................................Passed
Terraform validate with tflint...........................................Passed
Checkov..................................................................Failed
- hook id: terraform_checkov
- exit code: 1

terraform scan results:

Passed checks: 44, Failed checks: 1, Skipped checks: 14

Check: CKV_GCP_89: "Ensure Vertex AI instances are private"
        FAILED for resource: module.vertex_ai_workbench.google_notebooks_instance.tbd_notebook
        File: /modules/vertex-ai-workbench/main.tf:48-61
        Calling File: /main.tf:29-39
        Guide: https://docs.bridgecrew.io/docs/ensure-gcp-vertex-ai-workbench-does-not-have-public-ips

                48 | resource "google_notebooks_instance" "tbd_notebook" {
                49 |   depends_on   = [google_project_service.notebooks]
                50 |   location     = local.zone
                51 |   machine_type = "e2-standard-2"
                52 |   name         = "${var.project_name}-notebook"
                53 |   container_image {
                54 |     repository = var.ai_notebook_image_repository
                55 |     tag        = var.ai_notebook_image_tag
                56 |   }
                57 |   network             = var.network
                58 |   subnet              = var.subnet
                59 |   instance_owners     = [var.ai_notebook_instance_owner]
                60 |   post_startup_script = "gs://${google_storage_bucket_object.post-startup.bucket}/${google_storage_bucket_object.post-startup.name}"
                61 | }

dockerfile scan results:

Passed checks: 103, Failed checks: 1, Skipped checks: 2

Check: CKV_DOCKER_1: "Ensure port 22 is not exposed"
        FAILED for resource: /modules/docker_image/resources/Dockerfile.EXPOSE
        File: /modules/docker_image/resources/Dockerfile:30-30
        Guide: https://docs.bridgecrew.io/docs/ensure-port-22-is-not-exposed

                30 | EXPOSE 8080 16384 16385 4040 22
github_actions scan results:

Passed checks: 99, Failed checks: 0, Skipped checks: 0




Lint Dockerfiles.........................................................Failed
- hook id: hadolint
- exit code: 1

modules/docker_image/resources/Dockerfile:22 DL3013 warning: Pin versions in pip. Instead of `pip install <package>` use `pip install <package>==<version>` or `pip install --requirement <requirements file>`

• Modify Terraform code to use your custom Docker image in Vertex AI Workbench

Commit changes, push to a branch and open a PR to YOUR repository main/master branch. If you see a warning like this -- please enable the workflows: ...and repush your changes!

Once all Pull Requests checks have passed please merge your PR and wait until your release job finishes. 7. Navigate to the Vertex AI Workbench menu item, find your notebook on the list, press CONNECT and follow the instructions

8. In your Jupyterlab enviroment add Python3.8 kernel:

python3.8 -m ipykernel install --user --name pyspark

9. Run a Hello-world PySpark application in a YARN-client mode:

10. Additional tasks using Terraform:

1. Add support for arbitrary machine types and worker nodes for a Dataproc cluster and JupyterLab instance
2. Add support for preemptible/spot instances in a Dataproc cluster
3. Perform additional hardening of Jupyterlab environment, i.e. disable sudo access and enable secure boot
4. (Optional) Get access to Apache Spark WebUI

11. IMPORTANT ❗ ❗ ❗ Please remember to destroy all the resources after the workshop:

terraform init -backend-config=env/backend.tfvars
terraform destroy -no-color -var-file env/project.tfvars 

No providers.

No resources.

No outputs.