Feedback from @AaronCrawfis

It's difficult to scan the list of subscriptions when you're prompted to choose during environment setup. We should alphabetize this list based on the display name, so it's easier for humans.

AB#1126

Feedback from @orizohar

We should improve the docs here https://docs.radapp.dev/getting-started/create-environment/ to explain what kinds of resource that Radius is going to create in your sub.

This is our chance to explain that while we're leveraging K8s right now, it's an implementation detail of Radius.

We can say things like ... "while Radius is an experiment..."

Observed behavior
When e2e tests are triggered by /run-e2e-tests on a PR, they run on the latest main instead of the PR

Desired behavior
Tests should run on the PR branch

OS/rad version

Screenshots

Additional context

Create a tutorial style walkthrough of authoring radius assets and performing deployment of a Dapr microservices application.

The application should contain at least 3 compute resources

The application should use the "Dapr Managed State Store" and "Dapr Managed Pub/Sub" style

The application should have one of the components exposed to the internet for testing

It should be possible to go from zero knowledge of Radius, author the assets yourself and deploy a working application by following this tutorial.

When this is ready we'll share this with internal stakeholders for feedback

We should add support for key vault:

• Ability to deploy a keyvault instance
• Ability to store keys for a service (connection string) in keyvault via depends on
• Ability to access keyvault via depends on

Tried a deployment using the newly created dev subscription and getting this error:-

Details=[{"code":"Conflict","message":"{\r\n "status": "failed",\r\n "error": {\r\n "code": "ResourceDeploymentFailure",\r\n "message": "The resource operation completed with terminal provisioning state 'failed'.",\r\n "details": [\r\n {\r\n "code": "DeploymentScriptOperationFailed",\r\n "message": "The client 'xxxxxx' with object id 'xxxxxx' does not have authorization to perform action 'Microsoft.ContainerInstance/register/action' over scope '/subscriptions/xxxxxxx' or the scope is invalid. If access was recently granted, please refresh your credentials."\r\n }\r\n ]\r\n }\r\n}"}]error encountered: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details."

Tried creating a new service principal but still see this.

We should add a rad --version

This should output the build version (or EDGE) as well as the git SHA it's based on.

Ideally this would be plaintext not JSON, but using kubectl as an example of how detailed this could be (we don't need all of this).

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-19T07:38:38Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}

Absolute madlad @msfussell managed to get kubectl to crash during the runtime setup process.

error encountered: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details." 



Details=[{​​​​​​​​"code":"Conflict","message":"{​​​​​​​​  \"status\": \"failed\",
  \"error\": {​​​​​​​​
    \"code\": \"ResourceDeploymentFailure\",
    \"message\": \"The resource operation completed with terminal provisioning state 'failed'.\",
    \"details\": [
      {​​​​​​​​
        \"code\": \"DeploymentScriptError\",
        \"message\": \"The provided script failed with multiple errors. First error:\\r\\n+ [[ 2 -ne 2 ]]. Please refer to https://aka.ms/DeploymentScriptsTroubleshoot for more deployment script information.\",
        \"details\": [
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"+ [[ 2 -ne 2 ]]\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"+ RESOURCE_GROUP=mfussellradius\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"+ CLUSTER_NAME=radius-aks-aqwwvi7bfe7v2\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"+ az aks get-credentials --name radius-aks-aqwwvi7bfe7v2 --resource-group mfussellradius\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"++ curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"+ KUBECTL_VERSION=v1.20.4\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"+ curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.20.4/bin/linux/amd64/kubectl\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"% Total % Received % Xferd Average Speed Time Time Time Current\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"Dload Upload Total Spent Left Speed\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"100 38.3M 100 38.3M 0 0 65.1M 0 --:--:-- --:--:-- --:--:-- 65.1M\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"+ chmod +x ./kubectl\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"+ ./kubectl version --client\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"unexpected fault address 0x141c960\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"fatal error: fault\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"[signal SIGBUS: bus error code=0x2 addr=0x141c960 pc=0x141c960]\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"goroutine 1 [running, locked to thread]:\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"runtime.throw(0x1acd0cc, 0x5)\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc00071fe00 sp=0xc00071fdd0 pc=0x437492\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"runtime.sigpanic()\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\": \"/usr/local/go/src/runtime/signal_unix.go:739 +0x485 fp=0xc00071fe30 sp=0xc00071fe00 pc=0x44e025\"
          }​​​​​​​​,
          {​​​​​​​​
            \"code\": \"DeploymentScriptError\",
            \"message\":

Allow jobs to be queued if no test clusters are available

When the CLI is going to prompt you to delete, it should include the resource group name in the output.

Current:

➜ rad env delete azure
Using config file: /Users/ryan/.rad/config.yaml
Resource group associated with the environment will be deleted. Continue deleting? [y/n]? y
Deleting resource group rynowak-radius
Waiting for delete to complete...

Proposed:

➜ rad env delete azure
Using config file: /Users/ryan/.rad/config.yaml
Resource group 'rynowak-radius' will be deleted. Continue deleting? [y/n]? y
Deleting resource group rynowak-radius
Waiting for delete to complete...

We need to produce the following assets from a build:

• ARM template for environment setup
• Runtime configuration script for environment setup (installs Dapr into Kubernetes)
• Container image for control plane
• CLI binary

All of these resources need to be hosted somewhere public. YES the container image does as well.

Currently we use a storage account elbowassests in the incubations subscription for this - we should be using a dedicated subscription, and one of the rejected names (elbow) should not appear in the URL 🤣

We should also implement some kind of basic URL-based versioning scheme here. When we hand external users the CLI we want it to use a stable version of all of the assets, not a floating version.

Right now when we have a failure during application deployment, the error message printed will be like:

deploying application...Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details." Details=

[{"code":"BadRequest","message":"{\r\n  \"error\": {\r\n    \"code\": \"DownstreamEndpointError\",\r\n    \"message\": \"The resource provider 'radius' received a non-success response 'BadRequest' from the downstream endpoint for the request 'PUT' on 'Applications/frontend-backend/Deployments/implicit'. Please refer to additional info for details.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"Downstream\",\r\n        \"message\": \"{\\\"error\\\":{\\\"code\\\":\\\"\\\",\\\"message\\\":\\\"multiple errors: error applying workload resource map[] backend: failed to create typed patch object: .spec.ports: element 0: associative list with keys has an element that omits key field \\\\\\\"protocol\\\\\\\",error applying workload resource map[] frontend: failed to create typed patch object: .spec.ports: element 0: associative list with keys has an element that omits key field \\\\\\\"protocol\\\\\\\"\\\",\\\"target\\\":\\\"\\\"}}\"\r\n      }\r\n    ]\r\n  }\r\n}"}]

Yeah, that's right the message is a JSON string, with lots of extra encoding of quotes and newlines. It's really hard to read, and so far when team members have seen these error messages they haven't really tried to read/understand them.

We should unescape these errors and pretty-print them. This will help users report high quality issues.

AB#1128

Create a tutorial style walkthrough of deployment a LOB style web application using cosmosdb mongo and a containerized web front-end, accessible from the internet.

It should be possible to go from zero-knowledge of Radius to deploying a working application.

We'll share this tutorial and have internal stakeholders use the product and give feedback.

AB#7051

This wasn't working when we demoed this together a few days ago.

Need to investigate what's going wrong.

We have integration tests for deployment now, but they have the limitation that don't deploy any changes to the RP. This is problematic because it means we don't have any guarantees about what build of the RP is running when do the tests. It might be the latest build, or it might not 🤷

We should update these tests to build the RP image, and push it somewhere, and then trigger an update on the web site.

Come up with an project icon for Radius.

AB#1215

AB#5802

We need to add management operations to the rad CLI. Currently you can deploy and application and the CLI has no opinion of how you can interact with it, and you have to get out kubectl.

Importantly - we need these operations to be backed by ARM - they should use the ARM protocol and talk to our control-plane for all of this data, not directly to a Kubernetes cluster unless 100% necessary. This allows us to build the same experiences for our data types regardless of whether they are cloud data resources or compute resources.

Whomever does this work should start with a design proposal for how we're going to surface the concepts as ARM resources/operations.

For this checkpoint we need:

Applications:

• List
• Get (show details)
• Delete

Components:

• List
• Get (show details)

Deployments:

• List
• Get (show details)
• Delete

It's ok for now if the amount of detail is small. We don't need to convey the status of containers/pods in this checkpoint. That is covered by a future work item.

We need some guidance about troubleshooting when environment setup failed.

We likely need to just get started here, and work out along the way what that guidance should be 😆

Summary

We have some support for CosmosDB (Mongo) in the control plane - we also want to add the SQL variant as well.

There's some confusion about this because we got the wrong branding for the existing features. It's currently azure.com/CosmosDocumentDB but that's not the right terminology - Cosmos Document DB refers to the product as a whole, not just the Mongo support.

Description

Some notes about what we need to do (at a minimum).

• Correct the branding of CosmosDB Mongo (suggested: azure.com/CosmosDBMongo)
• Define MSSQL as a new binding type in the RP (suggested: microsoft.com/SQL) supporting connection strings
• Add a new component type in the RP for CosmosDB SQL (suggested: azure.com/CosmosDBSQL) supporting the microsoft.com/SQL binding
• Add integration test and how-to example
• Add types to rad-bicep (ask @rynowak to help) - do this once the initial PR is merged

AB#961

We've mostly been testing in isolation but we need to think about the UX for multiple people sharing access to the same environment.

For example would we want init to be able to attach to an existing environment?

Would we want a separate command for that?

This should result in (at a minimum) new content in the docs explaining how this works.

Currently we don't update config if environment deployment fails halfway, but leave the resources created as is. This prevents user from listing and deleting the env resources from rad cli. Example below -

Env deployment failed due to internal error from AKS, which eventually did succeed, but rad cli doesn't recognize it since config wasn't updated.

Validating Subscription...
Deploying Environment...
Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details." Details=[{"code":"Conflict","message":"{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"ReconcileEtcdBackupBlobStorageError\",\r\n        \"message\": \"AKS encountered an internal error while attempting the requested Creating operation. AKS will continuously retry the requested operation until successful or a retry timeout is hit. Check back to see if the operation requires resubmission. Correlation ID: 13f7d19a-3e62-4504-b5ee-67f68f67eaa0, Operation ID: ef6ddb72-3b6d-466d-8dc6-a6c2d3511ac7, Timestamp: 2021-03-16T04:03:14Z.\"\r\n      }\r\n    ]\r\n  }\r\n}"}]
exit status 1

https://chromium.googlesource.com/external/github.com/spf13/cobra/+/refs/heads/master/doc/md_docs.md

e2e-tests workflow should be either scheduled or triggered by a comment on the PR/issue. For the latter, create a workflow to parse the PR/issue comments and create a repository dispatch event to run e2e tests

This is something that broke in the move as part of a rewrite of the service building code

deploying application...Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details." Details=

[{"code":"BadRequest","message":"{
  \"error\": {
    \"code\": \"DownstreamEndpointError\",
    \"message\": \"The resource provider 'radius' received a non-success response 'BadRequest' from the downstream endpoint for the request 'PUT' on 'Applications/frontend-backend/Deployments/implicit'. Please refer to additional info for details.\",
    \"details\": [
      {
        \"code\": \"Downstream\",
        \"message\": \"{\\\"error\\\":{\\\"code\\\":\\\"\\\",\\\"message\\\":\\\"multiple errors: error applying workload resource map[] backend: failed to create typed patch object: .spec.ports: element 0: associative list with keys has an element that omits key field \\\\\\\"protocol\\\\\\\",error applying workload resource map[] frontend: failed to create typed patch object: .spec.ports: element 0: associative list with keys has an element that omits key field \\\\\\\"protocol\\\\\\\"\\\",\\\"target\\\":\\\"\\\"}}\"
      }
    ]
  }
}"}]

Right now rad deploy will output the generated ARM template to disk - we should use pipes or a temp file for this instead to hide it.

Running a rad command and failing to supply an argument results in a kinda lame experience:

radius on  main via 🐹 v1.16
➜ rad env delete
Using config file: /Users/ryan/.rad/config.yaml
accepts 1 arg(s), received 0

I'm supposed to put in an environment name, but if I didn't work on radius I'd have no idea 😆. I think we probably have this issue everywhere we use positional args in the CLI.

https://github.com/Azure/radius/blob/b139bda0d753f6e9cfcab5471f768f5df7cba20b/cmd/cli/cmd/expose.go#L26

Add an integration test for rad env init azure that can run nightly

The rationale for doing this nightly is that it's going to be slow (6-10 minutes) and we don't make changes to this area often. Basically we want to create a test that runs frequently enough so that we can know if this becomes unreliable and needs hardening.

It will be possible to queue on demand through the UI as well.

Static web apps isn't scaling with our requirement for 50+ users. We need to replat to App Service and setup auth to the Teams group.

Current end to end test runs use hardcoded cluster names in the code and share the dev subscription. To make it more reliable and less prone to manual deletes etc, we should use a separate test subscription and let the test run aks list to retrieve a cluster from the pool.

Also, would be good to automate creating the clusters in the test if none exists (will make the tests run longer but still better than letting it fail to manually create the cluster).

This could become the easiest way for people to engage with Radius and try it out.

We'd still want the Radius VS Code extension - we can't really solve that with a dev container 😭

The idea is that we'd have the CI publish an image that's set up with:

• az CLI
• kubectl (for now)
• rad CLI

This way there's no need to install anything other than the VSCode extension - you just open the container and do a tutorial there.

This could scale up if we build more kinds of samples - basically build a repo with the apps in them, and give uses the whole experience including making changes to the code. These could each be their own container.

1 node cluster

smaller more reasonable machine

We need to walk through the process and make sure the docs cover all of the relevant things for WSL.

This needs to include:

• Listing WSL as an option
• How to install CLI on WSL (should be same as other OSes)
• How to install VSCode extension on WSL (requires an extra step)

AB#1190

We should make the VS Code extension pop from our docs, and make it clear that it's highly recommended.

• We don't make the case that you should get the VSCode extension
• Right now the fact that we have an extension at all is buried in the CLI docs. We should make a dedicated page and sequence it after the CLI in instructions.
• We don't provide clear instructions about how to install it (double-clicking a VSIX doesn't work).

Using the CLI it should be possible to:

• store a list of saved environments
• remove environments
  • deleting the environment's resources
  • just remove it from the list without deleting resources
• switch between environments (set the current environment used for other commands)

This exists in a really basic form, but we need to review what's there, add tests where it matters and make sure the structure has the right design.

Add support for resource management of service bus

Add support for other components to access a service bus connection string

Right now the only option for using rad is to build from source. We should publish releases of rad and add a set of install scripts.

It's not possible to run rad init env azure over and over for the same resource group because AKS does not support changing service principals once created. We generate the service principal ID with a uuid right now, so it's always random.

Ultimately this means that when creating an environment fails, you cannot retry.

We should see if this can be fixed, either by generating stable service principal IDs or by detecting existing AKS instances.

We should:

• Centrally maintain a radius environment for automated tests
• Add a test via gh actions that deploys and then deletes an application
• This should run on every PR and commit
• Commits to master should update the version of the control plane image to ensure that's what is tested

Over time we'll add more components and validations to this test to include more surface area. To start with, it's ok if this is very simple (two containers).

We need some good conceptual documentation about the application model's concepts

This should serve as an intro for new users, as well as guidance for how to design schemas

All of the operations we've implemented for Radius RP so far are synchronous.

For a PUT or DELETE on deployments we should implement the asynchronous flow described here: https://github.com/Azure/azure-resource-manager-rpc/blob/master/v1.0/Addendum.md#asynchronous-operations

According to reports, the AZ CLI might install with a filename like az.cmd. We currently look for az.exe which may not be correct.

We should implement fallback logic and/or test this on a clean windows VM and see what gets installed.

The rad CLI will use the version hardcoded in our ARM template when creating an environment. Tagging as bug, because it results in environment setup not working depending on the location you choose when creating a resource group.

This is problematic because:

• The version support is different from region to region
• The version support changes over time

We should update the CLI to query the available versions in a region and select the default.

If we get requests to allow the user to configure the version in the future, then we can address that at that time.

It would be great to use ARM's tag system to track the resources created by the RP. We should standardize on a tag and apply it consistently.

We do this already with kubernetes with the kubernetes.io/managed-by: radius-rp label.

AB#1129

Currently delete environment deletes the resource group the environment belongs to, while it is unlikely that a customer would reuse a resource group across environments, there is nothing that prevents them from using an existing resource group during environment creation. We should revisit if we want to delete based on tags instead of entire resource group or make it a strict 1-1 mapping between resource group and environment.

AB#994

When @AaronCrawfis was trying to set up an environment we ran into an authentication error using the OCTO AD/Azure tenant.

We haven't done testing of this kind, we need to understand is this is a limitation we can solve or not, and either provide a clearer error or fix the issue.