Hi all,

The pre-provisioning is setting up the UDR routes. And the cluster provisioning diagram is also mentioning that egress traffic is going through the AZ FW.

But the terraform files don't have that configuration.
As far as I can see terraform is deploying a cluster with "Standard Load Balancer" and "VMSS" which implies the SLB with a public IP address. Would the UDR work in that case, as we would have a LB with public IP that is overriding the UDR?
Im guessing the outbound_type is missing in the terraform config. https://www.terraform.io/docs/providers/azurerm/r/kubernetes_cluster.html#outbound_type

https://github.com/Azure/sg-aks-workshop/blob/master/cluster-pre-provisioning/README.md
https://github.com/Azure/sg-aks-workshop/blob/master/cluster-provisioning/README.md
https://docs.microsoft.com/en-us/azure/aks/egress-outboundtype

Regrads,
Zarko

Once not in preview anymore, could be useful to add a small paragraph in Deploy App > Image Vulnerability Scanning and Management, to say that we have "embedded" image scanning within ACR.

If we quote every product on the market except "our" solution... :)

Would be good to include the personas (dev, operations, networking, security, etc) in the roles that play a part in the Governance + Security Scenario

kubernetes_namespace.flux: Creating...

Error: Post http://localhost/api/v1/namespaces: dial tcp 127.0.0.1:80: connectex: No connection could be made because the target machine actively refused it.

on flux.tf line 13, in resource "kubernetes_namespace" "flux":
13: resource "kubernetes_namespace" "flux" {

We have deployed the Falco k8s_audit_rules.yaml section as mentioned in the Cluster-config folder which is linked to Flux.
After the deployment, It seems only the falco_rules.yaml section is working. We were able to query the log results from Log analytics as well.

These are some falco rules that are working.

• Log files were tampered
• Shell history had been deleted or renamed
• Privileged container started
• Error File below / or /root opened for writing
• Notice A shell was spawned in a container with an attached terminal
• A shell configuration file has been modified
• Docker or kubernetes client executed in container
• Unexpected connection to K8s API Server from container

But for K8s-audit rules, the most important rules are as follows and they doesn't seem to be working.

1. Detect any attempt to create a namespace outside of a set of known namespaces
2. Detect any attempt to create a pod in the kube-system or kube-public namespaces
3. Detect any attempt to create a serviceaccount in the kube-system or kube-public namespaces
4. Detect any attempt to modify/delete a ClusterRole/Role starting with system
5. Detect any attempt to create a ClusterRoleBinding to the cluster-admin user
6. Detect any attempt to create a Role/ClusterRole with wildcard resources or verbs
7. Detect any attempt to create a Role/ClusterRole that can perform write-related actions
8. Detect any attempt to create a Role/ClusterRole that can exec to pods
9. Detect any attempt to create a deployment
10. Detect any attempt to delete a deployment
11. Detect any attempt to create a service
12. Detect any attempt to delete a service
13. Detect any attempt to create a configmap
14. Detect any attempt to delete a configmap
15. Detect any attempt to create a namespace
16. Detect any attempt to delete a namespace
17. Detect any attempt to create a service account
18. Detect any attempt to delete a service account
19. Detect any attempt to create a cluster role/role
20. Detect any attempt to delete a cluster role/role
21. Detect any attempt to create a clusterrolebinding
22. Detect any attempt to delete a clusterrolebinding

In the validate scenarios,
https://github.com/Azure/sg-aks-workshop/tree/master/validate-scenarios

Here the rules validated are for falco rules but there isn't any description provided for audit rules.

In the Falco documentation, it says to enable auditing policy for Master Node Kube API Server but in AKS since the users doesn't have access to Master Node, How does the Falco audit rules comes into action?

an idea would be to document the impact of applications logs generating additional cost and how to control this.
in particular some clusters would benefit from using Capacity commitment in Log analytics
https://azure.microsoft.com/en-us/updates/azure-monitor-log-analytics-new-capacity-based-pricing-option-is-now-available/

Build multi-tenancy requirements into a requirement

Hi,

I am using VS code for running the TF scripts in the sample. When i initalize the below variables i get an error that "expoort commend is not recognizable"

export TF_VAR_prefix=$PREFIX
export TF_VAR_resource_group=$RG
export TF_VAR_location=$LOC
export TF_VAR_client_id=$APPID
export TF_VAR_client_secret=$PASSWORD
export TF_VAR_azure_subnet_id=$(az network vnet subnet show -g $RG --vnet-name $VNET_NAME --name $AKSSUBNET_NAME --query id -o tsv)
export TF_VAR_azure_aag_subnet_id=$(az network vnet subnet show -g $RG --vnet-name $VNET_NAME --name $APPGWSUBNET_NAME --query id -o tsv)
export TF_VAR_azure_subnet_name=$APPGWSUBNET_NAME
export TF_VAR_azure_aag_name=$AGNAME
export TF_VAR_azure_aag_public_ip=$(az network public-ip show -g $RG -n $AGPUBLICIP_NAME --query id -o tsv)
export TF_VAR_azure_vnet_name=$VNET_NAME
export TF_VAR_github_organization=Azure # PLEASE NOTE: This should be your github username if you forked the repository.
export TF_VAR_github_token=
export TF_VAR_aad_server_app_id=<ask_instructor>
export TF_VAR_aad_server_app_secret=<ask_instructor>
export TF_VAR_aad_client_app_id=<ask_instructor>
export TF_VAR_aad_tenant_id=<ask_instructor>**

If i try to set the variables as below then when i am running terraform plan it shows variables are not getting set correctly and command keeps prompting for all variables starting from client_id to the bottom.

$TF_VAR_prefix=$PREFIX
$TF_VAR_resource_group=$RG
$TF_VAR_location=$LOC
$TF_VAR_client_id=$SERVER_APPID
$TF_VAR_client_secret=$PASSWORD
$TF_VAR_azure_subnet_id=$(az network vnet subnet show -g $RG --vnet-name $VNET_NAME --name $AKSSUBNET_NAME --query id -o tsv)
$TF_VAR_azure_aag_subnet_id=$(az network vnet subnet show -g $RG --vnet-name $VNET_NAME --name $APPGWSUBNET_NAME --query id -o tsv)
$TF_VAR_azure_subnet_name=$APPGWSUBNET_NAME
$TF_VAR_azure_aag_name=$AGNAME
$TF_VAR_azure_aag_public_ip=$(az network public-ip show -g $RG -n $AGPUBLICIP_NAME --query id -o tsv)
$TF_VAR_azure_vnet_name=$VNET_NAME
$TF_VAR_github_organization= $GITHUB_ORG_NAME # PLEASE NOTE: This should be your github username if you forked the repository.
$TF_VAR_github_token=$GITHUB_TOKEN
$TF_VAR_aad_server_app_id=$SERVER_APPID
$TF_VAR_aad_server_app_secret=$PASSWORD
$TF_VAR_aad_client_app_id=$CLIENT_APPID
$TF_VAR_aad_tenant_id=$TENANT_ID

Can anyone help how to resolve this issue?

In the Secrets Management Chapter (https://github.com/Azure/sg-aks-workshop/blob/master/cluster-design/SecretManagement.md) we propose 4 options:

• Using the plain Kubernetes secrets model
• Using the plain Kubernetes secrets model but encrypt the data in etcd using Azure KeyVault and the KMS plugin for Azure Key Vault
• Using the plain Kubernetes secrets but storing encrypted values in the Kubernetes secrets which are only encrypted at runtime
• Using Azure KeyVault for storing secrets and certificates and mounting them into a Kubernetes volume using the KeyVault FlexVolume driver.

And indeed there is a fifth and recommended option based on managed identities:

https://docs.microsoft.com/en-us/azure/aks/developer-best-practices-pod-security#use-pod-managed-identities

Do you know if there is any issue with this option? If not, I would like to add it to the Secrets Management Chapter.

Use multiple different attack scenario and show how to alert on those events

• Add Full deployment of all components
• Add workflow for cluster deployment only
• Add CI/CD workflow of our example app

Create Blue/green Cluster deployment scenario

Creat a doc that explains in greater detail the design choices that were made.