Here is my solution to the DevOps Engineering Challenge

The service is exposed (insecurely) : http://35.226.36.32

A list of items that should be completed are located in the TODO.md

I like to use Mermaid Diagrams for documentation. If you'd like this to render in Chrome, you can use the Mermaid plugin

graph LR
    gop[GoServer Deployment]
    goc[GoServer container]
    sqlc[SQLProxy container]
    svc[Load Balancer Service]
    sql[Cloud SQL]
    sqlsecret[Cloud SQL Service Account Key]
    gosecret[Go server secret]
    Kubernetes --> gop
    gop --> goc
    gop --> sqlc
    Kubernetes --> svc
    sqlc --> sql

    sqlc --> sqlsecret
    goc --> gosecret
    subgraph K8s Secrets
      gosecret
      sqlsecret
    end

• Go related details are located here
• Terraform related details are located here

ERD to understand interdependencies:

Invalid Deliveries SQL File

In order to deliver a working solution, I needed to learn/review quite a few technologies just to set up the framework.

The first day or two, I spent creating "hello, world" versions within each technology.

• Initialize

  • Create GCP project (done)
  • Configure VS Code for Go (done)
  • Configure cloud_sql_proxy locally for Postgres (done)

• Go Server

  • Develop hello world (done)
  • Implement Cloud Build pipeline (done)
  • Build/push image (done)

• K8s

  • Manual
    • Create cluster (done)
    • Create deployment (done)
    • Create service (done)
  • Helm
    • Create helm chart (done)
    • Configure deployment (done)
    • Configure service (done)
    • Configure ingress (not needed?)
  • Terraform
    • Create cluster (done)
    • Deploy helm chart (done)

Finally, I got around to developing a solution.

• Load data manually to Postgres (done)
• Explore data and create SQL query (done)
• Create go program
  • Getting started with go (done)
  • Query database (done)
  • Map objects for response (done)

All of the pieces exists, now we need to make sure it is all automated and works like we want.

• Create infrastructure
  • Terraform
    • K8s (done)
    • Helm (done)
      • cloud_sql_proxy
      • go_server
    • Database (done)
      • Postgres
    • Secrets (done)
    • Service Accounts (done)

That data needs to be loaded manually after the CloudSQL instance is created... how should we automate this?

gcloud sql import sql <cloudsql-name> gs://vorto-dropbox/coffee.sql --database public --user postgres

• Using issues to track stories
• Branching based on stories
• I'm sure there's a lot more I need to learn!

Expected behavior:

• Connecting to cloud sql instance with service account succeeds

Actual behavior

• Service account not able to access cloud sql using cloudsql-proxy.

Solution:

• "Create another service account after Enable the Cloud SQL API"
• Referecne: GoogleCloudPlatform/kubernetes-engine-samples#68
• I ended up just creating a separate service account for Cloud SQL connections

Expected behavior:

• When running terraform destroy/apply, the Cloud SQL database should be re-created.

Actual behavior:

• Database does not get created and TF eventually times out

Solution:

• Need to modify the instance name to have some unique name each time it is re-run
• "It seems we can not reuse the same instance name for up to a week after deleting an instance"
• Reference: hashicorp/terraform-provider-google#5101

Expected behavior:

• CloudSQL Proxy deployment connects successfully to CloudSQL instance

Actual behavior:

• Get "cannot fetch token" error

Solution:

• Redeploy kubernets cluster
• Reference: GoogleCloudPlatform/cloud-sql-proxy#9