Giter Club home page Giter Club logo

msbuildcache's Introduction

Microsoft.MSBuildCache

Build Status

This project provides plugin implementations for the experimental MSBuild Project Cache, which enables project-level caching within MSBuild.

Important

Currently MSBuildCache assumes that the build is running in a clean repo. Incremental builds, e.g. local developer builds, are not supported. Target scenarios include PR builds and CI builds.

Usage

To enable caching, simply add a <PackageReference> for the desired cache implementation and set various properties to configure it.

For repos which build C# code, add a <PackageReference> to Microsoft.MSBuildCache.SharedCompilation for shared compilation support.

Here is an example if you're using NuGet's Central Package Management and Azure Pipelines:

Directory.Packages.props:

  <PropertyGroup>
    <!-- In Azure pipelines, use Pipeline Caching as the cache storage backend. Otherwise, use the local cache. -->
    <MSBuildCachePackageName Condition="'$(TF_BUILD)' != ''">Microsoft.MSBuildCache.AzurePipelines</MSBuildCachePackageName>
    <MSBuildCachePackageName Condition="'$(MSBuildCachePackageName)' == ''">Microsoft.MSBuildCache.Local</MSBuildCachePackageName>

    <!-- Replace this with the latest version -->
    <MSBuildCachePackageVersion>...</MSBuildCachePackageVersion>
  </PropertyGroup>
  <ItemGroup>
    <GlobalPackageReference Include="$(MSBuildCachePackageName)" Version="$(MSBuildCachePackageVersion)" />
    <GlobalPackageReference Include="Microsoft.MSBuildCache.SharedCompilation" Version="$(MSBuildCachePackageVersion)" />
  </ItemGroup>

For repos using C++, you will need to add the projects to a packages.config and import the props/targets files directly.

Directory.Build.props:

  <PropertyGroup>
    <PackagesConfigFile>packages.config</PackagesConfigFile>
    <PackagesConfigContents>$([System.IO.File]::ReadAllText("$(PackagesConfigFile)"))</PackagesConfigContents>
    <MSBuildCachePackageVersion>$([System.Text.RegularExpressions.Regex]::Match($(PackagesConfigContents), 'Microsoft.MSBuildCache.*?version="(.*?)"').Groups[1].Value)</MSBuildCachePackageVersion>
    <MSBuildCachePackageRoot>$(NugetPackageDirectory)\$(MSBuildCachePackageName).$(MSBuildCachePackageVersion)</MSBuildCachePackageRoot>
  </PropertyGroup>
  <Import Project="$(MSBuildCachePackageRoot)\build\$(MSBuildCachePackageName).props" />

Directory.Build.targets:

  <Import Project="$(MSBuildCachePackageRoot)\build\$(MSBuildCachePackageName).targets" />

Common Settings

These settings are common across all plugins, although different implementations may ignore or handle them differently. Refer to the specific plugin documentation for details.

MSBuild Property Name Setting Type Default value Description
$(MSBuildCacheLogDirectory) string "MSBuildCacheLogs" Base directory to use for logging. If a relative path, it's assumed relative to the repo root.
$(MSBuildCacheCacheUniverse) string "default" The cache universe is used to isolate the cache. This can be used to bust the cache, or to isolate some types of builds from other types.
$(MSBuildCacheMaxConcurrentCacheContentOperations) int 64 The maximum number of cache operations to perform concurrently
$(MSBuildCacheLocalCacheRootPath) string "\MSBuildCache" (in repo's drive root) Base directory to use for the local cache.
$(MSBuildCacheLocalCacheSizeInMegabytes) int 102400 (100 GB) The maximum size in megabytes of the local cache
$(MSBuildCacheIgnoredInputPatterns) Glob[] Files which are part of the repo which should be ignored for cache invalidation
$(MSBuildCacheIgnoredOutputPatterns) Glob[] *.assets.cache; *assemblyreference.cache Files to ignore for output collection into the cache. Note that if output are ignored, the replayed cache entry will not have these files. This should be used for intermediate outputs which are not properly portable
$(MSBuildCacheIdenticalDuplicateOutputPatterns) Glob[] Files to allow duplicate outputs, with identical content, across projects. Projects which produce files at the same path with differing content will still produce an error. Note: this setting should be used sparingly as it impacts performance
$(MSBuildCacheRemoteCacheIsReadOnly) bool false Whether the remote cache is read-only. This can be useful for scenarios where the remote cache should only be read from but not produced to.
$(MSBuildCacheAsyncCachePublishing) bool true Whether files are published asynchronously to the cache as opposed to delaying the completion signal to MSBuild until publishing is complete. Note: Publishing will be awaited before the overall build completes.
$(MSBuildCacheAsyncCacheMaterialization) bool true Whether files are materialized on disk asynchronously from the cache as opposed to delaying the completion signal to MSBuild until publishing is complete. Note: Materialization will be awaited when a depending project requires the files and before the overall build completes.
$(MSBuildCacheAllowFileAccessAfterProjectFinishProcessPatterns) Glob[] \**\vctip.exe Processes to allow file accesses after the project which launched it completes, ie accesses by a detached process. Note: these accesses will not be considered for caching.
$(MSBuildCacheAllowFileAccessAfterProjectFinishFilePatterns) Glob[] Files to allow to be accessed by a process launched by a project after the project completes, ie accesses by a detached process. Note: these accesses will not be considered for caching.
$(MSBuildCacheAllowProcessCloseAfterProjectFinishProcessPatterns) Glob[] \**\mspdbsrv.exe Processes to allow to exit after the project which launched it completes, ie detached processes.
$(MSBuildCacheGlobalPropertiesToIgnore) string[] CurrentSolutionConfigurationContents; ShouldUnsetParentConfigurationAndPlatform; BuildingInsideVisualStudio; BuildingSolutionFile; SolutionDir; SolutionExt; SolutionFileName; SolutionName; SolutionPath; _MSDeployUserAgent, as well as all proeprties related to plugin settings, The list of global properties to exclude from consideration by the cache

When configuring settings which are list types, you should always append to the existing value to avoid overriding the defaults:

<PropertyGroup>
  <MSBuildCacheIdenticalDuplicateOutputPatterns>$(MSBuildCacheIdenticalDuplicateOutputPatterns);**\foo.txt</MSBuildCacheIdenticalDuplicateOutputPatterns>
</PropertyGroup>

Plugins

Microsoft.MSBuildCache.AzurePipelines

NuGet Version NuGet Downloads

This implementation uses Azure Pipeline Caching as the cache storage, which is ideal for repos using Azure Pipelines for their builds.

Please refer to the cache isolation and security pipeline caching provides to understand the security around this cache implementation.

It is expected that this plugin is used within an Azure Pipeline. The SYSTEM_ACCESSTOKEN environment variable will need to be explicitly passed to the task calling MSBuild since it's considered a secret. In classic pipelines this is done by checking "Allow scripts to access the OAuth token" in the job. In yml pipelines, this is done by passing the env var explicitly using the $(System.AccessToken) variable:

- script: MSBuild /graph /restore /reportfileaccesses /bl:$(Build.ArtifactStagingDirectory)\Logs\msbuild.binlog /p:Configuration=Release
  displayName: Build
  env:
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)

For the access token to have scopes relevant to Pipeline Caching, the Cache task must be used in the pipeline. There is work planned to opt-into this explicitly, but using the task is currently the workaround. You can cache a dummy folder if there is no otherwise reasonable use of the task in your build.

- task: Cache@2
  displayName: Enable Pipeline Caching
  inputs:
    key: '"dummy"'
    path: build

Microsoft.MSBuildCache.Local

NuGet Version NuGet Downloads

This implementation uses the local file system for the cache. In particular, it uses BuildXL's LocalCache. This is recommended for locally testing and debugging caching for your repo.

Microsoft.MSBuildCache.AzureBlobStorage

NuGet Version NuGet Downloads

This implementation uses Azure Blob Storage as the cache storage.

Warning

This implementation does not yet have a robust security model. All builds using this will need write access to the storage resource, so for example an external contributor could send a PR which would write/overwrite arbitrary content which could then be used by CI builds. Builds using this plugin must be restricted to trusted team members. Use at your own risk.

The connection string to the blob storage account must be provided in the MSBUILDCACHE_CONNECTIONSTRING environment variable. This connection string needs both read and write access to the resource.

Other Packages

Microsoft.MSBuildCache.SharedCompilation

NuGet Version NuGet Downloads

This package enables accurate file access reporting for the Roslyn Compiler Server. Because the compiler server (vbcscompiler) launches as a detached process, its file accesses are not observed by MSBuild. This package manually reports these files accesses to the plugin.

In the future, this feature will be directly supported by Roslyn, at which point this package will no longer be needed.

Microsoft.MSBuildCache.Common

NuGet Version NuGet Downloads

This package contains much of the shared logic for the plugins, including handling file accesses and cache fingerprinting. For those wanting to use a different cache backend than the ones associated with the available plugins, referencing this package can be a good option to avoid having to duplicate the rest of the logic involved with caching.

Contributing

See CONTRIBUTING.md

License

Microsoft.MSBuildCache is licensed under the MIT license.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

msbuildcache's People

Contributors

dfederm avatar johnterickson avatar microsoftopensource avatar rainersigwald avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.