Problem:
On the first session we have created the high level architecture using diagram as code, but we want to deploy this application in multiple environments to reproduce a real case scenario where we run automated tests on the CI/CD pipelines.

Change the diagram so we can show the different environments, with the following rules:

• We have 3 environments (test, acceptance and production) with the prefixes tst, acc and prd
• The CI pipeline should run automated tests in test environment
• The CD pipeline should:
  • Deploy to acceptance
  • Run automated tests in acceptance
  • Deploy to production
  • Run smoke tests in production
• The Azure Container Registry (ACR) is shared between environments

Describe the cloud infrastructure on Azure as code using Terraform.

• Setup Azure for test:
  • Create test subscription
  • az login
• Setup Terraform:
  • Initial backend on Storage Account
  • Configure provider
  • Resource group(s)
  • Azure Container Registry
  • Storage Account with Blob Containers bronze, silver and gold
• Create CI pipeline:
  • Setup
  • Lint
  • Validate
  • Plan
• Create CD pipeline:
  • Setup
  • Plan
  • Manual approval
  • Apply
• Update docs:
  • Requirements
  • Step by step on how to setup from scratch
  • Architecture

Problem:
We've created the diagram for the high level architecture, but it doesn't contain the details about the logical flow to actually Extract, Load and Transform data to be used by the Web App.

Create a diagram that shows the logical flow for the ELT cron job and how data is accessed on the Web App.

• Data storage consists of three layers (all under the same Azure Storage Account, but in different Azure Blob Containers):
  • Bronze: raw data from source systems
  • Silver: filtered, cleaned and augmented data
  • Gold: aggregated data ready for reporting
• The ELT cron job consists of three sequential tasks:
  • Extract:
    • Export data from Azure Cost Management REST API
    • Write raw data into Bronze layer
  • Load:
    • Read data from Bronze layer
    • Apply schema into dataset
    • Write data into Silver layer
  • Transform:
    • Reads data from Silver layer
    • Run transformations
    • Write data into Gold layer
• The ELT cron job has an Azure Service Principal as Managed Identity, which has Storage Blob Data Contributor role on all blob containers
• The Web App has an Azure Service Principal as Managed Identity, which has Storage Blob Data Reader role on the Gold blob container