dotnet / roslyn-tools Goto Github PK

I am interested in using signtool for our own projects. But since we build everything inside docker containers we can only use certificate files and not from a store (as far as I know).
It would be great to be able to specify a certificate file and password or private key file.

These packages won't exist until dotnet/roslyn#23196 is resolved.

Currently the repo toolset package has a version formatted as <Major>.<Minor>.<Patch>-<Suffix><number> (ex.: 1.0.0-alpha9 or 1.0.0-alpha40).

However, this does not follow the Package Versioning guidelines from NuGet (https://docs.microsoft.com/en-us/nuget/reference/package-versioning) and causes older packages to be reported as the 'latest version' (ex: 1.0.0-alpha9 is reported as being newer than 1.0.0-alpha40).

The package version should be updated to either place the postfix number (i.e. 9 or 40) in the slot or place a dot between the Suffix and postfix number (do: 1.0.9-alpha or 1.0.0-alpha.9, the former is probably "better", since that is the format we use elsewhere).

Steps to reproduce:

1. Build a project containing a XAML page
2. Make a change to a XAML page (in a *.xaml file)
3. Build the project again

Expected results:

The project builds.

Actual results:

The project does not build, reporting that it is already up to date.

Resolution steps:

Add the following somewhere in the toolset targets:

<ItemGroup>
  <UpToDateCheckInput Include="@(Page);@(Resource)" />
</ItemGroup>

@Pilchie, to track the work you're doing for @MattGertz, which might already be done. (This is mostly for tracking kudos down the road.)

Use package references instead.
This is blocking VS-less build of this repo.

https://github.com/dotnet/roslyn-tools/blob/master/src/VSIXExpInstaller/VSIXExpInstaller.csproj#L17-L56

None of the repositories that use repotoolset work with live-unit-testing. Talking to @KevinH-MS it appears to be related to how repo-toolset and live-unit-testing interact when manipulating the intermediate output path.

@tmat

Not sure if this is projectsystem problem or repo toolset problem, but we used to show a friendly error if you didn't have the right VS installed and you ran build.cmd, now we show:

E:\project-system2>build
Cannot bind argument to parameter 'Path' because it is null.
System.Management.Automation.ParameterBindingValidationException: Cannot bind argument to parameter 'Path' because it is null.
   at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
   at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at LocateVisualStudio, E:\project-system2\build\Build.ps1: line 96
at Build, E:\project-system2\build\Build.ps1: line 104
at <ScriptBlock>, E:\project-system2\build\Build.ps1: line 155
at <ScriptBlock>, <No file>: line 1

Didn't specify it should build packages but it is

C:\Users\jaredpar\code\roslyn-tools> .\Restore.cmd -msbuildversion "14.0"

Starting restore...
The restore completed successfully.

Skipping build...

Starting package...

Running: nuget.exe pack C:\Users\jaredpar\code\roslyn-tools\src/NuGet\SignTool.nuspec -BasePath "C:\Users\jaredpar\code\roslyn-tools\Artifacts\Debug\bin\SignTool" -OutputDirectory "C:\Users\jaredpar\code\roslyn-tools\Artifacts\Debug\bin\SignTool\NuGet\PreRelease" -prop licenseUrl="http://go.microsoft.com/fwlink/?LinkId=529443" -prop version="0.1.0-beta" -prop authors=Microsoft -prop projectURL="http://msdn.com/roslyn" -prop tags="Roslyn Signing" -prop thirdPartyNoticesPath="C:\Users\jaredpar\code\roslyn-tools\build/NuGetAdditionalFiles\ThirdPartyNotices.rtf" -prop netCompilersPropsPath="C:\Users\jaredpar\code\roslyn-tools\build/NuGetAdditionalFiles\Microsoft.Net.Compilers.props" -prop emptyDirPath="C:\Users\jaredpar\AppData\Local\Temp\2\u1kbmzib.muv" -prop NewtonsoftJsonVersion="9.0.1" -prop SystemCollectionsImmutableVersion="1.2.0" -prop SystemReflectionMetadataVersion="1.4.1-beta-24322-03"
Attempting to build package from 'SignTool.nuspec'.
Successfully created package 'C:\Users\jaredpar\code\roslyn-tools\Artifacts\Debug\bin\SignTool\NuGet\PreRelease\RoslynTools.Microsoft.SignTool.0.1.0-beta.nupkg'.
The package task completed successfully.
C:\Users\jaredpar\code\roslyn-tools>

@tmat suggested that we update repo toolset to have a single UsingDotNetCli property, rather than the current auto-detection plus explicit switches.

Right now we are kicking off new inter-branch merges every midnight Pacific Time, but we'd like to ramp this up to every few minutes so we can flow things immediately when they're merged.

Currently a repository using repo toolset that also specifies /p:QuietRestore=true cannot build if the enlistment path contains a space. Instead, the build fails with an error similar to the following:

MSBUILD : error MSB1008: Only one project can be specified. [C:\Users\tagoo\.nuget\packages\RoslynTools.RepoToolset\1.0.0-beta-62615-02\tools\Build.proj]
C:\Users\tagoo\.nuget\packages\RoslynTools.RepoToolset\1.0.0-beta-62615-02\tools\Build.proj(152,5): error MSB3073: The command ""C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\MS
Build.exe" /nodeReuse:false "C:\Users\tagoo\.nuget\packages\RoslynTools.RepoToolset\1.0.0-beta-62615-02\tools\Tools.proj" /bl:"D:\project system\artifacts\Debug\log\RestoreRepoTools.binlog" /nologo /m /v:quiet
/t:Restore /p:Configuration=Debug /p:CIBuild=False /p:RepoRoot=D:\project system\ /p:VersionsPropsPath=D:\project system\build\Versions.props /p:DotNetPackageVersionPropsPath= /p:BaseIntermediateOutputPath=D:\p
roject system\artifacts\toolset\Common /p:ExcludeRestorePackageImports=true /p:UsingPipeBuildPublishing=false" exited with code 1.

This failure looks to be because repo-toolset is calling Exec and not properly quoting the command line arguments.

Quoting the arguments is fairly trivial. However, there is an additional issue where, even if quoted, the build still fails for any paths that end with \.

The recommended fix would be to trim the trailing \ from any arguments and ensure that it is re-added, where required, in the props files themselves.

The Nerdbank.GitVersioning package defines a GetBuildVersion target: https://github.com/AArnott/Nerdbank.GitVersioning/blob/d398833fcdcb857fe44c3e8287f25b4058921181/src/Nerdbank.GitVersioning.NuGet/build/Nerdbank.GitVersioning.targets#L62

This conflicts with the GetBuildVersion target which repo toolset defines if UsingToolVSSDK is set to true. The result is that if you try to enable the VS SDK in a project, then Nerdbank.GitVersioning will stop working.

PR #132 allows this to be worked around by setting the DefineGetBuildVersionTargetForVsix property to false. However, a better fix would be to rename GetBuildVersion to GetVsixVersion in repo toolset.

This would be a breaking change for consumers of repo toolset- they'd have to update their vsix definitions to use the new target name.

Roslyn-analyzers repo currently has an experimental NuGet package for dataflow based analyzers Microsoft.CodeQuality.Analyzers.Exp.nupkg. We want to ship pre-release version of this package to NuGet to allow dogfooding, but never want to ship a release package (once we are comfortable with the DFA based rules to make it release, these rules will be moved to other existing analyzer packages and we will delete Microsoft.CodeQuality.Analyzers.Exp). It would be good if NuGetRepack can support blocking generation of release packages for such experimental packages - essentially address dotnet/roslyn-analyzers#1622 (comment).

As per the title, IsTestProject is declared late enough that it cannot be used as a condition on the TargetFramework property.

It would be great if this was possible, since some repos are targeting net46 for shipping binaries and net461 for test binaries.

Right now the insertion tool is trying to insert builds kicked off as part of the overall product construction build. It should only be trying to insert builds that actually have artifacts on our drop share.

It's not uncommon we forget to delete merged insertion branches, or forget to merge them entirely. The insertion tool should probably clean up old branches after a few weeks since we accumulate hundreds of them.

We need a dashboard to show the last time F#, Live Unit Testing, Project System, Roslyn, and CLI were inserted compared to their current latest source to ensure we don't have something breaking the flow process.

The proposed artifact layout has the following structure:

artifacts
  $(Configuration)
    obj
       $(MSBuildProjectName)

I do not believe that will work with the new SDK and project.assets.json. The NuGet model uses a single project.assets.json for all configurations. In this layout there doesn't appear to be anywhere to put that file.

Roslyn originally had this model but we had to modify it to support project.assets.json to the following:

Binaries
  Obj
    $(MSBuildProjectName)
      $(Configuration) 

The insertion tool creates pull requests whenever it updates any file on disk. If we're trying to insert a build that is already the inserted build, and this implicitly changes whitespace in a file (because we didn't round-trip it), we end up creating pull requests. This gets confusing because it looks like something is being updated but it isn't.

The fix (I think) is before creating the commit, look at all modified files and unpend any changes that are whitespace only changes.

In dotnet/sdk/src/Tests/Directory.Build.targets, PackAsTool is being set to true. This causes the Pack to use PackTool, instead of Pack.

In RepoToolset, if ExcludeFromSourceBuild is set to true, which it is for these tests, instead of importing the real Pack target, it imports an empty Pack target which basically noops for the project. But, since the SDK test projects switched to using PackTool, it isn’t noop’d. The easy solution is to add PackTool to the list of empty imports, but the other thing that happens when setting PackAsTool to true is that TargetsForTfmSpecificContentInPackage gets PackTool included, which then gets set as a DependsOnTarget for some other targets that get included and then fail because they expect certain artifacts to be present.

RepoToolset is using the $(RepoRoot)build folder location for build stuff. There's discussion in dotnet/arcade#33, where it was decided to change the folder structure to something like

\eng
   \core
   \tools

Arcade cannot make this change until RepoToolset is updated.

I'm trying to use repotoolset to compile the dotnet/arcade repo on macOS, and I get the following error

/Users/namc/.nuget/packages/roslyntools.repotoolset/1.0.0-beta-62614-02/tools/RepoLayout.props(37,5): error MSB4185: The function "GetFolderPath" on type "System.Environment" is not available for execution as an MSBuild property function.

It looks like the problem is here:

Using: dotnet 2.1.300-preview2-008251, macOS 10.13

cc @tmat

The insertion tool right now (I believe) won't respond to updateRevisions.txt files, since it does the commit via libgit2sharp which won't do the commit hooks. This is bad, because it means we won't update revision.txt files that we need to do in some cases, like assembly version changes.

We could fix this in two ways:

1. Shell out to git commit which will run the hooks.
2. Shell out to the hook directly.
3. Reimplement the support in the insertion tool.

When we build a solution in build.proj via

<MSBuild Projects="X.sln" Targets="Build"/>

msbuild generates a metaproj whose Build target builds the contained projects using their default targets, not their Build target.

If the projects do not specify default target the first target in the project is selected instead.

This is an issue when we are building in source build projects that should be excluded from source build, in which case the default targets are not defined anywhere.

The insertion tool's logic is:

1. Find the oldest passing build.
2. Insert that.
3. If it turns out no files changed, then simply don't create a PR and call it a "skip".

This can backfire if the oldest passing build is the build that's already inserted, and the reason it picked that build is because the newer builds were all failing. If that's the case, it should treat that as failure, not as a false "success".

It should be possible to generate an IVT via the following:

  <ItemGroup>
    <InternalsVisibleTo Include="SignTool.UnitTests" />
  </ItemGroup>

Presently it is not.

The word "part" isn't resonating with end users in the error message. Use a different word to describe items contained inside a VSIX.

The toolset uses a <PackageReference> to reference Microsoft.Net.Compilers:

This feature assumes that build extensions included in the package will be used during the build. However, for the .tmp_proj created for WPF projects, this is not the case. The underlying bug is NuGet/Home#5894.

The workaround is to explicitly import the NuGet build files when building the WPF temporary project, as shown in dotnet/roslyn-sdk#94.

  Packaging ...
MSBUILD : error MSB1008: Only one project can be specified. [C:\Users\micodoba\.nuget\packages\roslyntools.repotoolset\1.0.0-beta-62512-02\tools\Build.proj]
  Switch: spaces\msbuild\

  For switch syntax, type "MSBuild /help"
C:\Users\micodoba\.nuget\packages\roslyntools.repotoolset\1.0.0-beta-62512-02\tools\Build.proj(191,5): error MSB3073: The command ""C:\Program Files (x86)\Microsoft Visual Studio\Preview\Enterprise\MSBuild\15.0\bin\MSBuild.exe" /nodeRe
use:false "E:\projects\with spaces\msbuild\MSBuild.sln" /nologo /m /v:minimal /t:Pack /p:NoBuild=true /bl:"E:\projects\with spaces\msbuild\artifacts\Debug\log\Pack.binlog" /p:Configuration=Debug /p:CIBuild=True /p:RepoRoot=E:\projects\
with spaces\msbuild\ /p:VersionsPropsPath=E:\projects\with spaces\msbuild\build\Versions.props /p:DotNetPackageVersionPropsPath= /p:__BuildPhase=SolutionBuild" exited with code 1.

Root path is e:\projects\with spaces\msbuild

Arguments to the failing Exec task

"C:\Program Files (x86)\Microsoft Visual Studio\Preview\Enterprise\MSBuild\15.0\bin\MSBuild.exe" /nodeReuse:false "E:\projects\with spaces\msbuild\MSBuild.sln" /nologo /m /v:minimal /t:Pack /p:NoBuild=true /bl:"E:\projects\with spaces\msbuild\artifacts\Debug\log\Pack.binlog" /p:Configuration=Debug /p:CIBuild=True /p:RepoRoot=E:\projects\with spaces\msbuild\ /p:VersionsPropsPath=E:\projects\with spaces\msbuild\build\Versions.props /p:DotNetPackageVersionPropsPath= /p:__BuildPhase=SolutionBuild

Right now we kick off archiving for inserted builds. But if that job fails, or nobody ever ran the insertion tool (and it was inserted manually), we won't try again. The retention tool already has what's needed to do this, so we should just update it to check too.

The directory has been renamed to eng. See dotnet/arcade#33

Versioning currently defines version properties before analyzing the project. This means that the project can't define a project specific major / minor number. In some repos, like Arcade, version numbers are not consistent across artifacts produced by the repo.

Providing the version suffix as a separate property would allow a project to override the generated version.

DeployExtension is a standard VSSDK property that controls extension deployment. Here is an example usage:
https://github.com/DotNetAnalyzers/StyleCopAnalyzers/blob/d18b15f2342b52800c86a9540f95c1a51b0ddbbb/StyleCop.Analyzers/StyleCop.Analyzers.Vsix/StyleCop.Analyzers.Vsix.csproj#L21-L24

This approach does not work with RepoToolset because the DeployExtension property is overwritten with the value from DeployProjectOutput. This is unintuitive and results in example VS extension projects whose behavior is highly context-dependent and will not work if other users try to take the same approach.

Some of the repositories, such as the sdk/cli (and related) repos, are using the ASP.NET strong name key, rather than the .NET strong name key.

We need to, at a minimum, support selecting which of these two keys are used and support IVT generation for each.

If there are restore errors, build will continue and run the build. It should stop the build if there are restore errors.

The version of build tools currently used is 14.3.2.5407. That is not properly restored by using Restore.cmd.

Get the following error:

Starting toolset restore...
Locating MSBuild install path...
System.DllNotFoundException: Unable to load DLL 'Microsoft.VisualStudio.Setup.Configuration.Native.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)
   at LocateVS.Interop.GetSetupConfiguration(ISetupConfiguration& setupConfiguration)
   at LocateVS.Instance.GetSetupConfiguration()
   at LocateVS.Instance.<EnumerateVisualStudioInstances>d__1.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at LocateVS.Instance.LocateVisualStudioInstance(String vsProductVersion, HashSet`1 requiredPackageIds)
   at LocateVS.Instance.GetInstallPath(String vsProductVersion, String[] requiredPackageIds)
   at CallSite.Target(Closure , CallSite , Type , Object , Object )

The insertion tool creates a branch dev/vslsnap/... which it uses as the source of the pull request. It should also grant permissions to the swat group to be able to force push and delete this branch. This would allow swat members to delete the branches after merging, and would allow changes to be made when necessary.

Doing this will allow us to route all locating of VS through a single call to LocateVS and will additionally allow people to differentiate between errors in the LocateVS or ManagedQuery APIs and errors in locating VS itself.

Existing resx generator has an option to generate public resource class. Our generator should have the same option.

This is needed by VSUnitTesting repo.

The worst part is that some of these changes need to already be in the branch when the tool runs, so it will fail automated insertion unless these changes are added separately.

As far as I can tell, here's a complete list of things that need to be updated when a new CoreFX dependency gets added:

• A new line for the version in src/ProductData/AssemblyVersions.tt
• Two new lines for each dependency in src/ProductData/versions.xml
• New entries in every *.config.tt file below src/appid
• A new binding redirect in src/VSSDK/VSIntegration/ISOShell/Templates/VSShellTemplate/VSShellIso/VSShellStubExe/Stub.exe.config

It's not clear how many of these are actually necessary, but the insertion tool should probably do all of them automatically.

We used to show the actual failed tests inline in the console output so you could quickly see the test that failed, now build.cmd it just tells you the dll:

  [...]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.VisualBasic.Un
  itTests.dll [net461|x86]
XUnit : error : E:\project-system2\artifacts\Debug\TestResults\Microsoft.VisualStudio.ProjectSystem.Managed.TestService
s.UnitTests_net461_x86.html [net461|x86] [E:\project-system2\src\Microsoft.VisualStudio.ProjectSystem.Managed.TestServi
ces.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.csproj]
  Running tests: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.FSharp.VS.UnitTe
  sts.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.FSharp.VS.Unit
  Tests.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.CSharp.VS.Unit
  Tests.dll [net461|x86]
  Running tests: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.FSharp.UnitTests
  .dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.FSharp.UnitTes
  ts.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTe
  sts.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.VS.Uni
  tTests.dll [net461|x86]
  Running tests: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.VisualBasic.VS.U
  nitTests.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.VisualBasic.VS
  .UnitTests.dll [net461|x86]

Build FAILED.

XUnit : error : E:\project-system2\artifacts\Debug\TestResults\Microsoft.VisualStudio.ProjectSystem.Managed.TestService
s.UnitTests_net461_x86.html [net461|x86] [E:\project-system2\src\Microsoft.VisualStudio.ProjectSystem.Managed.TestServi
ces.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.csproj]
    0 Warning(s)

The WorkaroundIncorrectSatelliteDeployment has a comment on it:

Work around bug in Microsoft.NET.Sdk < v2.0 where satellites were deployed on top of each other in root. dotnet/sdk#1360

The SDK bug has been fixed, so this target can probably be removed. The workaround is causing issues when trying to pack a project:

C:\git\dotnet-sdk\artifacts.dotnet\2.1.300-preview3-008616\sdk\2.1.300-preview3-008616\Sdks\NuGet.Build.Tasks.Pack\build\NuGet.Build.Tasks.Pack.targets(198,5): error NU5118: File 'C:\git\dotnet-sdk\artifacts\Debug\bin\Tests\Microsoft.NET.Build.Tests\publish\de\Microsoft.DotNet.Cli.Utils.resources.dll' is not added because the package already contains file 'tools\netcoreapp2.1\any\de\Microsoft.DotNet.Cli.Utils.resources.dll' [C:\git\dotnet-sdk\src\Tests\Microsoft.NET.Build.Tests\Microsoft.NET.Build.Tests.csproj]

I'm trying to compile this on macOS: dotnet/arcade#35

After working around #172, I get this error:

/Users/namc/dev/dotnet/arcade/packages/microsoft.net.compilers/2.7.0-beta2-62330-02/tools/Microsoft.CSharp.Core.targets(84,5): error MSB6006: "csc.exe" exited with code 8

dotnet --info:

.NET Command Line Tools (2.1.100-preview-007366)

Product Information:
 Version:            2.1.100-preview-007366
 Commit SHA-1 hash:  8265c551de

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.13
 OS Platform: Darwin
 RID:         osx.10.12-x64
 Base Path:   /Users/namc/.dotnet/sdk/2.1.100-preview-007366/

Microsoft .NET Core Shared Framework Host

  Version  : 2.1.0-preview2-26131-06
  Build    : b13a0d5c331f374afd35ded57b9a4b4ab128864c

msbuild.binlog.zip

cc @tmat