retaildevcrews / helium-csharp Goto Github PK

Description:

For secrets that are retrieved from KeyVault and kept in memory we should use SecureString so if the memory were ever able to be dumped the secrets would not be exposed.

This helper class may be useful in the implementation: https://github.com/jofultz/AdobeClickStreamIngestion/blob/master/src/TestAdobeLiveStream/SecureStringHelper.cs

If EXPERIMENTATION_BUILD=true GitHub secret is set, build and publish a docker object tagged with :experimentation

Description

• It was observed that the build triggered from a PR in this repo passes even though unit tests in the solution are failing

Expected Behavior

• The CI/CD checks should fail when unit tests fail in the solution.

Reproduce

1. Break a unit test in the C# repo/solution and verify
2. Submit a draft PR with the change
3. Observe the CI/CD pipeline pass successfully
4. Observe the details of the build, specifically the test by clicking on the details and choosing "Display full test results".

name: Found Dead Links
about: Dead link
title: 'Dead link(s) found in markdown files in helium-csharp repository'
labels: 'Documentation'
assignees: ''

To find the dead links, please review the GitHub Actions logs for this repository

Nozomi - Helium ( P&G Engagement)

During the ADS session with customer, Helium was identified as an enabler and accelerator for the project by re-using some of its key components and patterns.

We have started contributing back to Helium based on engagement requirements. The major contributions are towards working with Cosmos DB.

Team already submitted PR's and will also contribute further.

Contributor Names:

• Sapinder Singh
• Lakshay Kaushik
• Ritesh Modi

Description:

• Update Ci/CD to allow for integrated Webv testing. Cleaned Up GitHub Actions to allow for enabling ACR of Docker Hub Pushes
• References retaildevcrews/helium#370

Description:

• Add to Bluebell

• References #xx

The solution could create Cosmos Client object during initialization phase and store it in IoC container. The controllers and business objects could later retrieve the same whenever needed by code. The lifetime for such objects could be singleton or scoped.

The following items have been flagged in the C# repo related to the HTTP/400 error messages

• Resolve GetMethodText encapsulation todo for both Actors and Movies
• Investigate cleaner implementation of tests (might not need to mock certain objects to get HttpContext)
• Investigate alternative to ServiceActivator class to get ILogger

Description:

• Currently / redirects to /index.html - update so it returns 200 with swagger content

Update CI-CD to publish a docker image on any source changes to master

Description

• setting --log-level is not always overriding the appsettings.json file

Expected Behavior

• if specified, --log-level should take precedence

Reproduce

dotnet run -- -l error ...

References retaildevcrews/helium#567

Wrap the new 400/404 json functionality in an #ifdef so we can build an experimentation version as well as a stable version.

Description

I believe that CosmosDBService is currently susceptible to SQL injection attacks because SQL strings are being built with simple string concatenation.

Reproduce

If we were to call one of CosmosDBService's methods such as queryActors using a malicious query string parameter such as an actor name of "Joe Smith; DELETE * FROM actors;" then we have an attack vector.

Additional Context

• (If applicable: Add any other context about the problem here; for example: doc changes, screenshots, logs, links, etc)

References retaildevcrews/helium#334

We are getting more healthz target duration errors than we should.

I propose extending the target duration to 250 ms for the single actor / movie read and 400 ms for the other checks

Description

At moderate load, about 2% of the 400 / 404 responses are significantly slower than mean and average. Since there are no external dependencies, the response rate should be very consistent. Need to investigate and also check for other "flyers" on different request types.

After more research, most of the variance is in the 404 errors.

Expected Behavior

• Less variance on response time

Reproduce

• This will generate about 90 rps for 5 minutes
  • webv -s {server} -f e2e.json -r --duration 300 -l 10

Capture the console log at the server and analyze

should test the other implementations as well

name: Found Dead Links
about: Dead link
title: 'Dead link(s) found in markdown files in helium-csharp repository'
labels: 'Documentation'
assignees: ''

To find the dead links, please review the GitHub Actions logs for this repository

Description

• resolve or override stylecop issues

Description

Currently Helium C# API Parameters are not following standard specifications.

Expected Behavior

• The goal is to apply commons parameter validations for actors
  https://github.com/retaildevcrews/helium/blob/api_specs/docs/HeliumParamSpecs.md
• On API calls, the responses for valid and invalid scenarios should reflect rules specified in the specifications.

Problem Statement

• Current logging implementation relies on 'boxing' which wraps value types and stores them on the managed heap.
• Calling logger extension methods such as LogWarning have a higher computational overhead than LoggerMessage in ASP.NET Core.

Proposed Solution

• Leverage LoggerMessage in ASP.NET Core.
• LoggerMessage uses cacheable delegates
• Improve developer experience by writing log messages which are relevant to the method calling them. For example, logger.InvalidGenreParameter(context).
• Performance test and document differences from default main branch.
• Based on performance results, decide whether or not to implement new feature.

Alternative Proposals

• Keep the existing logger implementation.

Additional Context

• Since performance is a key objective for Helium, any small improvements we can observe, document, and implement will result in a better customer experience at scale.

Next Steps

• Team consensus to proceed
• Complete Design Review Template (if applicable)
• Schedule Design Session

Update to latest Azure SDKs

Update to Key Vault v4.1.0

References #66
References #77

This is especially useful when the number of documents are large in Cosmos DB collection. Looping them on demand using continuation token and using stream iterators would improve performance, the memory footprint would be small, cost effective from RU perspective and providing paging benefits to callee.

When running this docker file in a network restricted environment, it is necessary to sometime set the value of the following variables so that the dotnet command can pull packages from nuget

• HTTPS_PROXY
• HTTP_PROXY
• https_proxy
• http_proxy

The process is described here

name: Found Dead Links
about: Dead link
title: 'Dead link(s) found in markdown files in helium-csharp repository'
labels: 'Documentation'
assignees: ''

To find the dead links, please review the GitHub Actions logs for this repository

Problem Statement

• Cosmos DB provided out of box encryption at rest using its own keys. We should be able to bring our own keys and use it for encrypting data at rest.

Proposed Solution

• Add additional documentation about steps to be performed to enabled encryption at rest using custom keys and link it within overall documentation

Alternative Proposals

Additional Context

Next Steps

• Team consensus to proceed
• Complete Design Review Template (if applicable)
• Schedule Design Session

Refactor code to increase test coverage

Description

• the github action Docker Image Build does not run dotnet test
• we changed the PR build and need to reflect those changes in the master build
• need to review the changes in PR build to validate first

• convert dockerfiles to use alpine as the base image

C# User Story

As a web application developer, I want a reference web application that is secure by default while leveraging engineering best practices so that I can easily fork and code and deploy.

Acceptance Criteria

• Provide a step-by-step quick start that delivers a positive developer experience
• Securely build, deploy, and run an Azure App Service (Web App for Containers) application
• Securely build, deploy and run an Azure Kubernetes Service (AKS) application
• Use Managed Identity to securely access resources
• Securely store, access, and maintain secrets in Key Vault
• Securely build and deploy the Docker container from Azure Container Registry (ACR), Azure DevOps, GitHub Actions or manually
• Securely connect to and query Cosmos DB
• Automatically send telemetry and logs to Azure Monitor

Key Features

• Security
  • "Secure by Design"
  • Managed Identity
    • Pod Identity
  • Key Vault
  • Key Rotation
  • Developer Experience
• Cloud Native Platform (containerization)
  • ACR
  • App Services
  • AKS
  • DevOps
  • IaC
  • CI-CD
  • End-to-end Testing
• CosmosDB
  • Document Modeling
  • RU optimization
  • Key Rotation
• Observability
  • Logging
  • Monitoring
  • Alerting
• Developer Experience
  • Secure by Design
  • Fork and Code
  • Local dev or container dev

Definition of Done

• Code changes reviewed & signed off
• Existing documentation is updated (readme, .md's)
• New documentation needed to support the change is created
• Code changes checked into master
• All existing automated tests (unit and/or e2e) pass successfully, new tests added as needed
• Build completes successfully
• Deployment completes successfully
• Smoke test production deployment for minimum of 2 weeks

Engineering Playbook Definition of Done

Description

Give users a Docker environment for setting up Helium. Some users can't meet the current requirements, but they are able to use Docker Desktop.

• A Dockerfile for helium csharp that can be used locally and in codespaces. This should include all the tools needed to step through the Helium setup.

• Build the Dockerfile as part of ci/cd. Build on changes to the file, build and push to Docker Hub when merging to main.

• Build and push the Dockerfile on a recurring schedule for package updates.

• For codespaces, use the pre-built images for faster startup.

• References retaildevcrews/helium#579

Description

The docker build for codespaces is failing because of a missing file. The cause if that the build starts from the parent directory.

The build is triggered like

docker build . -t codespaces-dotnet -f .devcontainer/Dockerfile

But then the dockerfile looks for library-scripts/... which does not exist in the current directory. It is under .devcontainer/

Fix the build.

name: Found Dead Links
about: Dead link
title: 'Dead link(s) found in markdown files in helium-csharp repository'
labels: 'Documentation'
assignees: ''

To find the dead links, please review the GitHub Actions logs for this repository

name: Found Dead Links
about: Dead link
title: 'Dead link(s) found in markdown files in helium-csharp repository'
labels: 'Documentation'
assignees: ''

To find the dead links, please review the GitHub Actions logs for this repository

• Tracks Helium issue 90

Update nuget packages to latest version

• update Codespaces to use ghcr.io/cse-labs images

According to https://tools.ietf.org/id/draft-inadarei-api-health-check-01.html#rfc.section.3, the HTTP status code returned for "warn" should be 2xx/3xx, currently, 503 is returned for the /healthz/ietf endpoint.

• Add a command line param and env var for specific key vault authentication type

• Managed Identity

  • value: MSI
  • default value
  • cmd line: --authtype MSI
  • env var: AUTH_TYPE=MSI

• Azure CLI cached credential

  • value: CLI
  • cmd line: --authtype CLI
  • env var: AUTH_TYPE=CLI

• Visual Studio cached credential

  • value: VS
  • cmd line: --authtype VS
  • env var: AUTH_TYPE=VS

• no auto fallback

• no support for other identity types

• Unused / invalid parameters should be an error with exit(-1) and display Usage()

• add Usage() for no args or -h --help as the first arg.

• Required

  • --kvname - name or full URL of the Azure Key Vault

• Optional

  • -h --help - display command line usage
  • --authtype - authentication type
  • MSI - managed identity (default)
  • CLI - Azure CLI cached credentials
  • VS - Visual Studio cached credentials

Description

• GetGenreAsync should use ReadItemAsync instead of GetItemQueryIterator
• CosmosDetail class has hard coded values

Description:

• /healthz swagger docs still references "Healthy", "Degraded", "Unhealthy"
• /healthz/ietf and /healthz/dotnet have strange values for example value and schema
• Probably worth meeting to discuss expectations all up for swagger docs, maybe combine with retaildevcrews/helium#243?

Update to Azure.Cosmos from Microsoft.Azure.Cosmos

References #66

• Remove "deathYear" from API output when the field does not exist in the database's Actor document (when the actor is still alive), currently, it defaults to 0 when the field does not exist.
• Check for any other regressions that may have been introduced after upgrade to dotnet 3.0

Description

Currently Helium C# API Parameters for Movies are not following standard specifications.

Expected Behavior

• The goal is to apply commons parameter validations for movies
  https://github.com/retaildevcrews/helium/blob/master/docs/ParameterValidation.md

• On API calls, the responses for valid and invalid scenarios should reflect rules specified in the specifications.