Issue

When we added a 2nd Key Vault instance to store the SQL account info it changed the way that the createAppRegistrations.sh script behaves. The script was developed to expect only 1 key vault name and is now confused when it sees 2 vaults.

The multiregional deployment should demonstrate how data is kept in sync between regions.

Azure SQL Database is used to store information about concerts, customers, and the tickets that they've purchased. This service stores this data and the relationships between this data to help customers shop for concerts and see their tickets when they login.

Without synchronizing this data, customers will not be able to see the tickets that they purchased. And, when the code runs in two regions it will randomly seed the data in a different order, so the concert details will not appear the same after a failover.

To address this concern we recommend two Azure SQL Features:

• Active geo-replication to synchronize data between the two regions
• Auto-failover groups to handle the failover of the system from one region to another

Per field feedback, we have many customers that have invested heavily in Azure DevOps pipelines and do not intend to migrate to GitHub Actions anytime soon. We want to meet them where they are for our patterns deployment assets.

Issue

We need to associate the new GH repo with an Azure subscription so that we can test the GH workflows that will deploy resources.

Todo

Run azd pipeline config

Issue

When the bicep templates create the SQL resource it sets the managed identity as the SQL administrator. This is the only valid login for the box because a SQL administrator user is not created. The problem is that when the app runs users do not see data. The root cause of this is that the managed identity account is not actually a valid user until someone logs into the server to create a SQL user.

This expected workflow for managed identity accounts is described by docs.

• Tutorial: Access data with managed identity - Azure App Service
• Tutorial: Use a managed identity to access Azure SQL Database

Todo

We need to update the bicep templates to

1. DONE Set a SQL administrator account as the login (this is a break glass account that should be stored in separate key vault)
2. DONE Create a separate key vault to store the SQL pwd secret - because we don't want the web app to be able to see it
3. DONE Modify existing SQL deployment to use SQL account (key vault reference)
4. DONE Run a bicep deployment script to setup the SQL user for the managed identity
5. DONE Grant the managed identity db_owner access
6. DONE change SQL server to managed identity access only
7. DONE for prod - block public access. for non-prod public access is enabled to support the dev inner-loop.

This grants more permission than the web app should have but avoids refactoring the database creation/seeding that will be handled by a separate work item.

Progress

Referenced the doc to understand how to pass Key Vault secret to another component of Bicep. The implication is that the Sql setup must be refactored to a module.

The principal that is executing the deployment does not grant the deployment access to read the secret.

Also learned that Key Vault must be configured to allow access from ARM.

The password must be more than letters and numbers.

src

Need to understand how to run SQL from a bicep deployment. There seem to be 2 options.

1. PowerShell
2. CLI utility

Found known issue: Cannot reference resources across modules

produces errors like...

Must use explicit params to bypass this issue.

Found that the SQL user account cannot be used to grant managed identity access.

Others have encountered this error, and with the shared goal of devOps automation, and received the following advice from support.

https://stackoverflow.com/a/56150547

I opened a ticket with Azure support and they gave me this solution.

The sql statement needs to be:

 -- type X for AAD Group
create user [myAADGroupName] with sid = <sid>, type = X;

-- type E for AAD User or Service Principal/MSI
create user [myAADUserName] with sid = <sid>, type = E;
The sid needs to be generated from the AAD Principal ObjectID in most cases. However, for Service Principals/MSIs, it needs to come from the AppId. Here's a powershell script to generate the sid value:

param (
    [string]$objectIdOrAppId
)

[guid]$guid = [System.Guid]::Parse($objectIdOrAppId)

foreach ($byte in $guid.ToByteArray())
{
    $byteGuid += [System.String]::Format("{0:X2}", $byte)
}

return "0x" + $byteGuid

Pending the resolution of an open issue with AZD we should update the nightly build to remove the AppConfigSvcPurg step and shell script

Align multiregional deployment with the features that the azd team are developing to support high availability deployments.

Nightly build job failed because it could not find the 2nd resource group to delete

Failed action

Error: deployment failed: failed deploying: failed running az deployment sub create:

Deployment Error Details:
InvalidTemplateDeployment: The template deployment 'relecloudqa7' is not valid according to the validation procedure. The tracking id is '42fb931e-b7d4-4c78-adce-f834ba1c935a'. See inner errors for details.

Run azd provision --debug and found that the inner error is

Inner Errors:
{"message": "Object reference not set to an instance of an object."}

This issue is impacting the nightly deployment - #43

Issue

The security pillar of the Well-architected framework encourages a defense in depth approach which means securing resources in multiple ways. One of the best protections for an Azure resource is limiting network access to resources with Private Endpoints. We should consider changing the way that the Web API app and Azure Functions connect to storage. They could use Azure Private Endpoints to limit network access to approved resources.

Issue

The sample should demonstrate the operational excellence we recommend by tagging each of the deployed resources.

Todo

Review the bicep templates and ensure that each resource is tagged.

Issue

App Services are using Application Insights but the Azure Portal doesn't recognize the connection and still suggests enabling App Insights.

Todo

Use bicep to setup relationship between App Insights and App Service.

• pending - looks like this sample creates the link by describing the relationship as a site extension

Issue

The SQL server password is deterministic and there may be a more secure way to generate this data.

Issue

All of the bicep resources are located in the resources.bicep file. It is becoming cumbersome to maintain and to understand conceptually. It should align with our operational excellence principles by splitting the deployment into discrete modules with names that align to their purpose.

Todo

Separate the deployment into multiple bicep templates to improve maintainability.

Progress

While working on the sql setup issue the team identified that key vault references (where the sql admin password is stored) must be passed via modules. src

Issue

It would be a better demonstration of operational excellence if the web app could detect or "pick up" changes from App Configuration Service or Key Vault without an App Pool restart.

Todo

• Research whether this is a code change or a feature change.
• Modify the code
• Modify the guide to describe this feature and to explain what's happening when simulating faults

Issue

These settings should be enabled

https://docs.microsoft.com/en-us/azure/azure-sql/database/auditing-overview?view=azuresql

Progress

looks like this may be something we can enable via bicep
https://docs.microsoft.com/en-us/azure/templates/microsoft.sql/servers/auditingsettings?pivots=deployment-language-bicep

reviewed the existing dotnet sample - does not demonstrate this feature
https://github.com/microsoft/App-Templates

Inside the ReleCloud App, the search feature always appears to return the same results, no matter what search terms I put in. For example, if I search for, "Foo Fighters," I get a great big list of concerts - not just the "Foo Fighters" concert. Also, if I enter a band that doesn't exist in the list like "Dead Milkmen" it still returns the same list of concerts.

azd env new -e $myEnvironmentName will fail if one has never logged in before. az login rectifies the problem.

Issue

When the Web API app is deployed with private endpoints enabled it uses Azure Private DNS (only) to retrieve IP addresses. Even though Redis is allowing connections the web api has trouble connecting because it cannot resolve the IP address.

Todo

Add the IP address entry to the Azure DNS

Perhaps this is obvious, but one needs the dotnet sdk to be installed in order to run azd provision instruction.

Update Azure Key Vault usage to align with security best practice recommendations - https://docs.microsoft.com/en-us/azure/defender-for-cloud/recommendations-reference#identityandaccess-recommendations

The guide tells you that if you encounter an error running createAppRegistrations.sh, you should change the line endings with VSCode. It would save people time searching the internet if the guide told people how to do this. https://stackoverflow.com/questions/39525417/visual-studio-code-how-to-show-line-endings

Typical enterprise customers build virtual data center in the cloud. These data centers are comprised of hub-and-spoke networks that facilitate security and control over the virtual data center. Enterprise applications, such as the one described here, must exist within the context of this virtual data center. Please provide guidance on how this application pattern can fit into the virtual data center.

Issue

Data in Azure should be stored with encryption at rest. In Azure SQL Server Database this is handled with Transparent Data Encryption (TDE). We should enable this on the database to align with Azure best practices.

Todo

Enable TDE as part of the deployment in bicep.

Progress

Found this is now on by default for all environments. No additional research or configuration needed.

"By default, TDE is enabled for all newly deployed Azure SQL Databases"

Transparent data encryption - Azure SQL Database

Update the repo so we're included in the search list https://github.com/topics/azd-templates

The initial code samples were images. Now that we are in markdown, we should use the code block style:

//sample code here

The project guide contains a security section which describes the use of identity-based authentication. It mentions that Managed Identity will be used to secure access to services like Key Vault and App Configuration. It would be useful to include a description of how this is done; role assignments applied on the target services. It would also be useful to mention that a service may run with a system provided managed identity, or a user-specified managed identity with links to background on managed identity. (Some may prefer to control their application identities)

The Project Guide begins with a section on application reliability. It describes how Polly may be used to achieve application reliability. Please include some content that mentions the Azure SDKs themselves also offer reliability. Otherwise, there is a chance that the reader will be left with the impression that Azure itself is not reliable.

Inside the ReleCloud app, choose a concert and attempt to purchase tickets. Once you complete the purchase workflow, you are re-directed to the Your Tickets screen and it says, "You haven't purchased any tickets yet, why don't you browse around the upcoming concerts?"

We should review the reference sample with other teams and incorporate their recommendations to ensure that we're providing the best possible guidance.

Topics to review:

• SQL Admin configuration and Password Generation
• Azure VNet configuration with respect to hybrid network recommendations

The Project Guide begins with an architeture diagram and then immediately launches into a discussion of reliability. Please set the stage first by describing the solution and its projects; otherwise, it is unclear where this reliability code goes.

Issue

A single region deployment was found to have a composite SLA of 99.56%. We need to create an option for customers to be able to deploy at least three 9's of availability.

Todo

Investigate using azd to deploy a multiregion deployment with Front door to reach a higher SLA as described by docs.

Include a description of estimated cost and Azure prerequisites in the project readme.md. This information is available in the project guide, however, it would help to set expectations (reduce fear) by letting the reader know that they need to provide an Azure Subscription, have sufficient privileges to create resources, and sufficient funds before executing the deployment script.

Issue

The nightly build pipeline failed to deploy SQL server.

{
    "status": "Failed",
    "error": {
        "code": "OperationTimedOut",
        "message": "The operation timed out and automatically rolled back. Please retry the operation."
    }
}

Status: Conflict
Duration: 49 minutes 56 seconds
serviceRequestId: 17fac9f4-21b7-481b-b940-f2139058a986
Tracking ID: fde7b995-efcf-4503-be1b-6a3d77dca48c
Deployment Correlation ID: c04742b6-1c00-4926-9f2a-636af68c9034

After planning discussion, decided to reduce the concept load of phase 1 by extracting the queue-based load-leveling assets into phase 2 and leaving the image computation monolithic for now. It aides in the following ways:

1. Reduces concept count for phase 1 in guidance and code
2. Sets us up for demonstrating the value of observability in phase 2, when we identify a hot spot in the app that needs to be extracted into a separate background process.
3. Sets us up for monolithic decomposition in phase 2, which will be more of a focus in phase 2.

Recommended Workaround

If you experience this issue the recommendation is to delete the Azure Cache for Redis resource from your subscription and then to re-run the azd provision command.

You can also restart your deployment process by changing the value of $myEnvironmentName to a new unique string so that a new resource group will be created.

Issue

This item is created to capture, and evaluate, the issues we're experiencing when deploying Azure Cache for Redis. At this point there have been a handful of errors deploying this resource that we have dismissed as one-off or temporary issues. Capturing them here in one location will allow us to understand the frequency, and severity, of these issues.

1. 8/8/2022 - relecloudqa-rg (Ken Schlobohm Subscription - eastus): Was unable to provision the Azure Cache for Redis. Gave up on the deployment operation after more than an hour of waiting for this issue to resolve to an error or success.
   

Underlying status/error was

{
    "status": "Failed",
    "error": {
        "code": "InternalServerError",
        "message": "Something went wrong.\r\nRequestID=55555555-eeee-4444-9999-000000000000"
    }
}

Resolution
Had to make a new resource group to bypass the issue

# Step 7: Set managed identity only access
Enable-AzSqlServerActiveDirectoryOnlyAuthentication -ServerName $ServerName -ResourceGroupName $ResourceGroupName
#if ($IsProd) {
    # Step 8: Block public access
    Set-AzSqlServer -ServerName $ServerName -PublicNetworkAccess 'disabled' -ResourceGroupName $ResourceGroupName
#}

Issue

The intent is to have SQL server only allow Azure AD connections and, for production only, disable public network access.

These features were previously part of the bicep template and had to be removed as part of the SQL server user setup work item.

A deployment script is used to setup managed identity and it needs the ability to connect as SQL admin from Azure (0.0.0.0).

To address this issue we need another deploymentScript to run after SQL setup has been completed to enable these security features. The challenge will be providing enough credentials to the deploymentScript to be able to perform the changes.

Examining App Insights details for request failures on the "Upcoming Concerts" page shows that the web app cannot talk to Redis this impacts the ability to store session, MSAL tokens, and cache data from SQL Database.

Resources should be consistently named with prefixes

e.g.
https://github.com/Azure/azure-dev/blob/7c79ad9efdcb90a161ac588db4493e33d92e40e3/templates/common/infra/bicep/abbreviations.json

Goal

• A "prod" ready environment uses "prod" SKUs

Services

• DONE - Key vault
  • Non-prod sku - standard
  • Prod sku - standard
    • We're not using keys so we don't get value out of the higher SKU
    • Offers 'Premium tier, which includes hardware security module(HSM)-protected keys' From https://docs.microsoft.com/en-us/azure/key-vault/general/overview
• DONE - App Configuration Service
  • Non-prod sku - "Standard"
  • This is the highest sku available - Pricing - App Configuration | Microsoft Azure
• DONE - App Service (Front end and API)
  • Scale up
    • Non-prod sku - "B1"
    • Prod SKU "P1V2"
  • Scale out
    • Scale out by 1 instance, max of 10 instances, when CPU is greater than 85% for 10min
    • Scale in by 1 instance, min of 1 instance, when CPU is less than 60% for 10min
• DONE - Azure Cache for Redis
  • Non-prod sku - Basic C0
  • Prod sku - Standard C1
• DONE - SQL Database
  • Non-prod sku Standard 10 DTU with 250gB max storage
  • Prod sku Premium 125 DTU with read-replica support, 250gB max storage, geo-redundant backup, and backups
    • Azure SQL Database creates full backups every week, differential backups every 12-24 hours, and transaction log backups every 10 minutes. src
• DONE - Azure Storage
  • Non-prod sku - Storage v2 Standard_LRS
  • Prod - Standard_ZRS - we don't need low latency blobs to improve solution quality and we can't make a transaction if one of the zones is offline with geo redundant storage (we could only read) so geo redundant doesn't offer an advantage over zone redundant. src

Issue

The Nightly build should tear down correctly.

The store was deleted but not purged.

Listing purged app svc shows nothing pending

Todo

Manually trigger the nightly build pipeline twice and validate that it will succeed both times.

Progress

This is due to expected behavior described by the doc

If none of the deploymentScripts resource properties (including the inline script) are changed, the script doesn't execute when you redeploy the Bicep file

The deploymentScript should use the forceUpdateTag to indicate that the script should be re-run everytime a deployment is triggered.

Further research shows that the deploymentScript was unable to delete the resource. Moving this script into the pipeline may have changed the identity that was running this command because that proves successful.

• Setup secrets.json with URI reference to app configuration URL?
• Can't connect to the SQL Database because the provisioned deployment prevents external access

Issue

The production environment should include support for backups to help businesses address concerns of data loss, or corruption.

Todo

Create, or validate, that the production environment includes support for SQL server backups.

Progress

• When we setup multiple environment support we automatically added database backup support.

Prod sku Premium 125 DTU with read-replica support, 250gB max storage, geo-redundant backup, and backups
Azure SQL Database creates full backups every week, differential backups every 12-24 hours, and transaction log backups every 10 minutes. src

The following topics need to be addressed in the documentation/writeup:

• Discussion point: SPA vs. MVC style application. How do we
  position/explain this decision

Issue

The createAppRegistrations.sh script will create app registrations and place the configuration data that it creates into Key Vault and Azure App Config Svc. Running the script is an essential part of the setup process.

Currently, if the app registrations already exist the script will detect the condition and then exit without setting Key Vault and App Config Svc values.

Todo

The script should read from the existing App Registrations and write the necessary config data to App Config Svc and Key Vault. Users should not have to delete these app registrations to continue testing their deployments.

Workaround

The issue can be worked around by deleting the existing app registrations.