As some self-hosted agents are adopting mu_devops we want to reduce the number of dependencies on those agents. For example, eliminate bash support on Windows and Powershell on Linux. This simplifies agent software requirements and reduces the size of software installed and maintained on the agents.

The common language we'd prefer to use is Python. By moving all pipelines to use Python for inline scripting, the templates will be more portable to these agents.

The issue-labeler 2.6 updates enforces HTTP errors. An error is being returned when the action attempts to remove a label that does not exist.

Ideally, a check would be in place to see if a label exists before attempting to remove it to prevent the error.

This task tracks following up on a plan to move off v2.5.

PR #299 updated .sync/workflows/leaf/codeql.yml to first check the new location (as of Mu 2311) for the CodeQL plugin in Mu Basecore BaseTools. If not found, it falls back to the previous location in .pytools.

This is to allow immediate backward compatibility with the latest Mu DevOps release across existing Mu branches. It is recommended that 2302 (and other relevant branches) move to the BaseTools location. When all branches are confirmed to have moved, the fallback to .pytools can be removed in the file.

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Dependencies on the mu_devops repo currently use mu_devops/main.

To improve traceability and stability, they should use a release version of the repo.

Background

Scripts/TagGenerator/TagGenerator.py contains code to automatically tag a commit with a release version tag for ADO use cases where GitHub Actions don't apply.

The script currently specifies a single semantic versioning format that works for Mu repos:

• <Major>.<Minor>.<Patch>

Other consumers have expressed interest in the script but need to use a custom version format. In this particular case:

• [ProductName].[VendorNumber].[FirmwareComponent].[MajorVersion][MinorVersion].[Sku].[Patch]

An original Project Mu versioning goal was to drive consistent versioning through a well-established standard like semantic versioning. Given a deviation is needed due to conflicting requirements for this consumer, the issue proposes changes to handle both formats.

It is still a goal in this proposal to make the versioning information completely visible in mu_devops without depending on ambiguous gaps filled in by code elsewhere. This keeps opinionated consistency in place, makes the code here more useful to a broader set of users, and makes overall maintenance of the code much easier in a single upstream location.

Details

I'd prefer to continue to capture the full context of the version in a mu_devops script. Ideally, different formats would be abstracted behind a class. The class would provide information needed to make common decisions irrespective of the underlying format details. Then we don't need half-baked information here connected with other scripts elsewhere, all of the version information can be maintained behind a single abstracted version API. This also gives the benefit that other Python code using the class wouldn't need to change significantly if the underlying version format changed in the future.

Module level functions like is_breaking_change(), is_security_change(), is_new_feature(), generate_notes(), etc. would become class methods and return consistent information regardless of how they return the format for the underlying version.

In the end, this would mean:

• That the existing format ("semantic versioning format") and new format ("Product Component format") would have their underlying format details readily visible to developers that look at the scripts in mu_devops.
  • Other projects can easily choose the appropriate format without reinventing the wheel.
• Other Python code can use the class API to get various version information regardless of the underlying format details.
• The script can easily and obviously be extended in the future to support new formats without modifying various lines of code here and there in the procedural logic which is error prone and messy.
• Mu Devops still provides ready-to-use versioning schemes that don't need details filled in elsewhere.

This issue is in reference to tracking a change discussed in #293.

The auto merge workflow currently checks for PR updates for 1 minute. In some cases, a bot PR cannot be merged right away. For example, either another bot PR is merged before its CI checks complete or a manual PR is merged. In any case, the bot PR needs to be rebased and the status checks run again.

This task tracks increasing the time these updates can be detected.

Keep builds up to date with the latest image and tools.

The Mu DevOps PR Formatting Validator workflow validates correct formatting of PR descriptions in Project Mu repos.

The PR formatter should be updated to check that these instructions have been followed:

1. Replace this text with an actual descrption:

   <_Include a description of the change and why this change was made._>

2. Remove this line of instructions so the PR description shows cleanly in release notes:

   "For details on how to complete to complete these options and their meaning refer to [CONTRIBUTING.md](https://github.com/microsoft/mu/blob/HEAD/CONTRIBUTING.md)."

3. For each checkbox in the PR description, place an "x" in between [ and ] if true. Example: [x].
   (you can also check items in the GitHub UI)

   • [] Impacts functionality?
     • Functionality - Does the change ultimately impact how firmware functions?
     • Examples: Add a new library, publish a new PPI, update an algorithm, ...
   • [] Impacts security?
     • Security - Does the change have a direct security impact on an application,
       flow, or firmware?
     • Examples: Crypto algorithm change, buffer overflow fix, parameter
       validation improvement, ...
   • [] Breaking change?
     • Breaking change - Will anyone consuming this change experience a break
       in build or boot behavior?
     • Examples: Add a new library class, move a module to a different repo, call
       a function in a new library class in a pre-existing module, ...
   • [] Includes tests?
     • Tests - Does the change include any explicit test code?
     • Examples: Unit tests, integration tests, robot tests, ...
   • [] Includes documentation?
     • Documentation - Does the change contain explicit documentation additions
       outside direct code modifications (and comments)?
     • Examples: Update readme file, add feature readme file, link to documentation
       on an a separate Web page, ...

4. Replace this text as instructed:

   <_Describe the test(s) that were run to verify the changes._>

5. Replace this text with as instructed:

   <_Describe how these changes should be integrated. Use N/A if nothing is required._>

A new Project Mu feature repo was created recently:

https://github.com/microsoft/mu_feature_dfci

This needs to be integrated into Mu DevOps.

Several repos have pipeline files considered different enough, that they have been broken out into their own directories (in the .sync/azure_pipelines/custom directory of this repo).

The files are maintained in mu_devops so they can be patched with the mu_devops version to use during the file sync process but, are otherwise copied directly to the relevant repo.

Note that files in .sync/azure_pipelines/matrix_dependent already are consolidated across all repos that use a Matrix-Build-Job<...>.yml file.

This task is to consider consolidation of the files in "custom" such that the total number of files could be reduced. This could involve content substitution or generation at sync time.

I did originally consider doing more to consolidate these files but I felt the tradeoff in complexity of dynamically forming the file versus content reduction made maintainability worse. Looking at the files 1:1 seemed to make it easier to maintain. I'll revisit this in this task though.

With crate-focused repos like mu_rust_hid beginning to be created, a reusable workflow is needed to release the crate to crates.io.

Also generate release notes on the previous Mu release branch since it is likely to still be picking up changes and needs to version them.

We can generate coverage data today with cargo tarpaulin. This task tracks adding that coverage data to codecov.io so we can get PR comments showing coverage delta.

This item is meant to track the overall task, though the changes may not be entirely in mu_devops.

Requiring PowerShell for contains can increase their size by ~0.25 Gb for fairly limited use. It may be best to use common environment scripting rather than relying on PowerShell.

PRs present options that automatically apply labels to the PR.

Example:

The "Impacts functionality?" box currently applies the impact:non-functional label if checked. The logic needs to be inverted to be correct.

In the issue templates synced from mu_devops to other repos, the submitter can elect to own the issue. Currently, if the submitter does not take ownership the state:needs-ownerlabel is added.

This request tracks additionally adding the submitter as the assignee if they choose to take ownership of the issue.

In 61bd5b5, I originally passed the command and params in a sequence to Popen(). I switched params to be a single string instead of a sequence. I forgot to update the Popen() call to pass the value as a single string instead of the sequence.

The command will still work but parameters will not be passed correctly until only the single string is passed.

@Javagedes will include this in another change he's making.

Currently the submodule release updater is implemented as a composite GitHub Action that wraps a single Python script. Because this currently only targets GitHub repos, it also directly uses the GitHub REST API.

It would be useful to expand support for Azure DevOps (ADO) repos as well. In that case, the Python script could be removed directly from the action.yml file and an endpoint abstraction put in place to redirect API calls either to GitHub or ADO.

The GitHub label impact:breaking-change currently updates the major version component of the repo if present in a change.

Since integration of breaking changes is important, the release notes should also highlight any breaking changes so consumers can quickly see what action is needed.

6e00a3d synced the release drafter leaf workflow to repos. However, a local config file is also needed. So, the config file currently in mu_devops/.github/ReleaseDraft.yml needs to be synced to the .github directories of Mu repos as well.

Ensure there's no regression in Mu repo usage and update the GitHub action in Mu DevOps.

An alternate mechanism is needed to place iasl into Ubunter container images as the dockerfile currently pulls the tarball from acpica.org.

Currently, pip installation as performed by the steps in the SetupPythonPreReqs.yml template in this repo result in a number of DEPRECATION warnings and notices installation may fail in an upcoming pip update.

This should be resolved to prevent unexpected failures in the future.

Example of current output from a release/202208 mu_basecore pipeline:

Collecting edk2-pytool-library~=0.12.1
  Downloading edk2_pytool_library-0.12.1-py3-none-any.whl (453 kB)
     ------------------------------------- 453.5/453.5 kB 14.3 MB/s eta 0:00:00
Collecting edk2-pytool-extensions~=0.20.0
  Downloading edk2_pytool_extensions-0.20.0-py3-none-any.whl (2.5 MB)
     ---------------------------------------- 2.5/2.5 MB 31.5 MB/s eta 0:00:00
Collecting edk2-basetools==0.1.24
  Downloading edk2_basetools-0.1.24-py3-none-any.whl (1.3 MB)
     ---------------------------------------- 1.3/1.3 MB 21.2 MB/s eta 0:00:00
Collecting antlr4-python3-runtime==4.7.1
  Downloading antlr4-python3-runtime-4.7.1.tar.gz (111 kB)
     -------------------------------------- 111.4/111.4 kB 6.7 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting regex
  Downloading regex-2022.10.31-cp311-cp311-win_amd64.whl (267 kB)
     ------------------------------------- 267.7/267.7 kB 17.2 MB/s eta 0:00:00
Collecting pyyaml>=5.3.1
  Downloading PyYAML-6.0-cp311-cp311-win_amd64.whl (143 kB)
     -------------------------------------- 143.2/143.2 kB 4.3 MB/s eta 0:00:00
Collecting pefile>=2019.4.18
  Downloading pefile-2022.5.30.tar.gz (72 kB)
     ---------------------------------------- 72.9/72.9 kB 2.0 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting semantic-version>=2.10.0
  Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
Collecting future
  Downloading future-0.18.2.tar.gz (829 kB)
     ------------------------------------- 829.2/829.2 kB 17.4 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Installing collected packages: edk2-pytool-library, antlr4-python3-runtime, semantic-version, regex, pyyaml, future, edk2-basetools, pefile, edk2-pytool-extensions
  DEPRECATION: antlr4-python3-runtime is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for antlr4-python3-runtime: started
  Running setup.py install for antlr4-python3-runtime: finished with status 'done'
  DEPRECATION: future is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for future: started
  Running setup.py install for future: finished with status 'done'
  DEPRECATION: pefile is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for pefile: started
  Running setup.py install for pefile: finished with status 'done'
Successfully installed antlr4-python3-runtime-4.7.1 edk2-basetools-0.1.24 edk2-pytool-extensions-0.20.0 edk2-pytool-library-0.12.1 future-0.18.2 pefile-2022.5.30 pyyaml-6.0 regex-2022.10.31 semantic-version-2.10.0

[notice] A new release of pip available: 22.3 -> 22.3.1
[notice] To update, run: python.exe -m pip install --upgrade pip

As the file grows, groups of repos are emerging as commonly being bundled together. This request tracks adding named groups to make these collections of repos more descriptive and easier to maintain.

Ensure there's no regression in Mu repo usage and update the GitHub action in Mu DevOps.

Now that file syncing has been added, we prefer file changes to be made in the common file copy in mu_devops rather than in the copies in downstream repos.

This issue tracks the ability notify users (other than the notices currently in the copyright area) that they are modifying such a file and to direct them to mu_devops.

Users that do not have permissions to set labels may want to request maintainer feedback via the state:needs-maintainer-feedback.

This issue tracks adding that to the templates synced from mu_devops.

We need to ensure that changes made in mu_devops pass all consumer pipelines.

Pull requests into mu_devops need to have status checks against all repos that use it as a repository resource to verify the build succeeds with the mu_devops change.

For example:

• Azure universal artifacts with az devops login
• NuGet feeds in same org or different org with service connection using NuGetAuthenticate

Currently we set /subsystem=efi_boot_services_driver for all pure Rust binaries, which works for development, but as we get runtime service and applications, we need to appropriately set the subsystem type. @makubacki for reference.

We have several labels now that indicate an issue or PR is no longer valid. For example, state:wont-fix, state:invalid, and so on.

To help complete the automation flow of identifying and closing items we no longer plan to look at, the ability to auto close such issues and PRs should be added.

See title

Ensure there's no regression in Mu repo usage and update the GitHub action in Mu DevOps.

Link fragment checking support has been requested for markdown files

v0.32.2 implements this feature

(https://github.com/DavidAnson/markdownlint/blob/main/doc/md051.md)

Upgrade to v0.32.2 to enable this feature.

Issue forms currently have a number of places where input is standardized to allow automated decisions to be made based on the input values.

Free text input (not very reliable) should be able to be handled by the reg ex flow already in place. This feature is mainly about adding the ability to exactly match the values in dropdowns and auto assign labels since these are very reliable (and form validated) input values.

The change should allow values in each dropdown to map to a unique label that will be applied.

containers provide a more consistent and controlled build environment. mu_devops should allow consumers to specify a container environment to use and/or provide a default image.

Need to switch mu_devops to use NuGetAuthenticate@1 tasks.

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/nuget-authenticate-v0?view=azure-pipelines

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/nuget-authenticate-v1?view=azure-pipelines

PR #25 installs the wheel and setuptools pip modules in a pipeline step.

This is to ensure all Project Mu repos have these installed as pre-requisites (for the reason mentioned in the PR) before installing other pip modules.

However, the same pre-requisites need to be accounted for when local users are installing pip modules.

Since this is not an immediate issue, all of the pip-requirements.txt files in all of the repos are not updated to have wheel and setuptools to prevent unnecessary dependency thrash across the repos. We're expected to enable a "git subtree like" process soon that could consolidate pip requirements such as these.

This issue is filed as a reminder to include these modules in the consolidated pip requirements when possible. If this becomes more urgent before that is done, the pip-requirements.txt files in repos can be updated.

When a large number of pipeline runs at once (e.g. pytools pip update), the PublishCoverage job in Jobs/PrGate.yml significantly contributes to the overall time for the pipeline to finish.

This makes PRs take much longer to reach a status check finished state and for the overall job queue to drain. Essentially, the pipeline has to queue twice, once for all of the matrix jobs and again for the code coverage publishing job.

This issue tracks optimizing code coverage publishing to reduce the impact on overall pipeline execution time.

For example, a matrix will spawn N jobs. At the end of each matrix job checking if all other matrix jobs (N-1) are complete, perhaps using the ADO REST API, and then attempting to publish code coverage directly from the job, etc.

The build queue can extend quite long when certain automated tasks occur such as dependabot updates. This is especially true if an update causes other automation to trigger. For example, a dependabot update in a repo picks up a pip update, other repos have a submodule dependency on that repo so they trigger when that build completes, and so on.

Right now, the plan is to simply move most automation tasks that can have a high impact on the build system to times when most development does not occur and then reevaluate if any further changes are needed.

Some code coverage results are not reported in the Azure pipeline UI.

For example this build runs MacAddressEmulationDxe tests, but they do not appear in the coverage.xml 

• https://dev.azure.com/projectmu/mu/_build/results?buildId=34335&view=ms.vss-test-web.build-test-results-tab
  Pipelines

There's also been changes to code coverage reporting and result file merge capabilities added to edk2 recently. See:

tianocore/edk2#3349

This task tracks reviewing and updating code coverage for Project Mu repos.

We can use information already available (labels) to improve our ability to track semantic versioning compliance with more frequent and better documented releases.

This feature should include the ability have automated releases where semantic versioning is rolled according to PR impact data and the next release info is always available to see what's happened since the last release. This makes it easy to determine when to release (on minor and major version revs) and do so with minimal effort.

In addition, the release info should be categorized so it's easier to understand what kind of changes are in a release.

Tracks updating .sync/workflows/leaf/codeql.yml to include support for platform builds - repos that build with a "PlatformBuild.py" file in a package rather than a top-level "CiSettings.py" file in a .pytool directory.

At this time, this affects mu_tiano_platforms.