aspnet / buildtools Goto Github PK
View Code? Open in Web Editor NEWInfrastructure utilities used in the ASP.NET Core build system. (Obsolete: use https://github.com/dotnet/arcade for all new projects)
License: Other
Infrastructure utilities used in the ASP.NET Core build system. (Obsolete: use https://github.com/dotnet/arcade for all new projects)
License: Other
When the nuget verifier runs, the logs contain a lot of information that's not needed. It seems to write 4 lines/package when there are no errors. That's 405 packages x 4 lines/package = 1620 lines of text for the CoreCLR builds just to tell us that everything is okay:
15:21:14][Step 1/1] INFO: Analyzing runtime.rhel.7-x64.Microsoft.NETCore.TestHost (1.0.0-rc2-23914)
[15:21:14][Step 1/1] INFO: No issues found with package runtime.rhel.7-x64.Microsoft.NETCore.TestHost (1.0.0-rc2-23914)
[15:21:14][Step 1/1] INFO: Took 24ms
[15:21:14][Step 1/1]
[15:21:14][Step 1/1] INFO: Analyzing runtime.rhel.7-x64.runtime.native.System (1.0.1-rc2-23914)
[15:21:14][Step 1/1] INFO: No issues found with package runtime.rhel.7-x64.runtime.native.System (1.0.1-rc2-23914)
[15:21:14][Step 1/1] INFO: Took 27ms
[15:21:14][Step 1/1]
[15:21:14][Step 1/1] INFO: Analyzing runtime.rhel.7-x64.runtime.native.System.IO.Compression (1.0.1-rc2-23914)
[15:21:14][Step 1/1] INFO: No issues found with package runtime.rhel.7-x64.runtime.native.System.IO.Compression (1.0.1-rc2-23914)
[15:21:14][Step 1/1] INFO: Took 22ms
[15:21:14][Step 1/1]
[15:21:14][Step 1/1] INFO: Analyzing runtime.rhel.7-x64.runtime.native.System.Net.Http (1.0.1-rc2-23914)
[15:21:14][Step 1/1] INFO: No issues found with package runtime.rhel.7-x64.runtime.native.System.Net.Http (1.0.1-rc2-23914)
[15:21:14][Step 1/1] INFO: Took 22ms
[15:21:14][Step 1/1]
[15:21:14][Step 1/1] INFO: Analyzing runtime.rhel.7-x64.runtime.native.System.Security.Cryptography (1.0.1-rc2-23914)
[15:21:14][Step 1/1] INFO: No issues found with package runtime.rhel.7-x64.runtime.native.System.Security.Cryptography (1.0.1-rc2-23914)
[15:21:14][Step 1/1] INFO: Took 27ms
[15:21:14][Step 1/1]
[15:21:14][Step 1/1] INFO: Analyzing runtime.ubuntu.14.04-x64.Microsoft.NETCore.ConsoleHost (1.0.0-rc2-23914)
[15:21:14][Step 1/1] INFO: No issues found with package runtime.ubuntu.14.04-x64.Microsoft.NETCore.ConsoleHost (1.0.0-rc2-23914)
[15:21:14][Step 1/1] INFO: Took 26ms
[15:21:14][Step 1/1]
[15:21:14][Step 1/1] INFO: Analyzing runtime.ubuntu.14.04-x64.Microsoft.DotNet.CoreHost (0.0.1-beta-00001)
[15:21:14][Step 1/1] INFO: No issues found with package runtime.ubuntu.14.04-x64.Microsoft.DotNet.CoreHost (0.0.1-beta-00001)
[15:21:14][Step 1/1] INFO: Took 35ms
Scrolling thought all that and finding an error is not fun:
http://aspnetci/viewLog.html?buildId=74464&buildTypeId=CoreCLR_CoreCLRRoot&tab=buildLog
Also ,the verifier logs can be as much as 25% of the log file.
For example, Microsoft.EntityFrameworkCore.Tools.DotNet. This causes restore errors and test failures on Coherence
Package Microsoft.EntityFrameworkCore.Tools.DotNet 1.0.0-preview3-21860 is not compatible with net451
cc @javiercn
The CI picks up any messages to the error stream and adds it to the summary.
The verifier has dependencies which make it desktop only. We should figure out if there are alternatives we could use to make it xplat compatible.
We should add it as a secondary check to individual repo check.
DependenciesPackager 1.0.1-rc2-15132 fails with this error on Linux.
fail: DependenciesPackager.Program[0]
System.AggregateException: One or more errors occurred. (Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: chunkLength) ---> System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: chunkLength
at System.Text.StringBuilder.ToString()
at DependenciesPackager.CrossgenContext.CrossgenAssembly(PackageEntry package, LibraryAsset asset, String targetPath)
at DependenciesPackager.CrossgenContext.<>c__DisplayClass19_0.b__0(PackageEntry package)
We use these tools in other repos and consequently changes to this might break older APIs. We should instead have these be per branch so we can work on dev and release independently.
The current powershell script has some dependencies on Powershell 5.0 which our CI doesn't support, so we should make it an exe instead
Just putting all the packages in a dependencies node is not good enough. Packages need to be classified in three categories. 'All' for packages that support netstandardapp1.5 and net451, netstandardapp1.5 for packages that support netstandard or a compatible framework and net451 for packages that are exclusive for net451
To avoid this happening: dotnet/efcore#5867
All product assemblies need this.
When crossgen fails, DependenciesPackager only issues a warning and will still pass. It should fail so we can resolve the problem.
e.g.
warn: DependenciesPackager.Program[0]
Crossgen failed for /buildspace/artifacts/rc1-22720-dev/x64/microsoft.aspnet.webapi.client/5.2.2/lib/portable-wp8+netcore45+net45+wp81+wpa81/System.Net.Http.Formatting.dll. Exit code 33.
Changes introduced in #128
Unhandled Exception: System.IO.IOException: The process cannot access the file 'C:\Users\namc\AppData\Local\Temp\tmp98A3.dll' because it is being used by another process.
at System.IO.Win32FileSystem.DeleteFile(String fullPath)
at System.IO.File.Delete(String path)
at NuGetPackageVerifier.Rules.AssemblyHasAttributeRuleBase.Validate(PackageAnalysisContext context)
at NuGetPackageVerifier.Rules.DefaultCompositeRule.d__1.MoveNext()
at NuGetPackageVerifier.Rules.AdxVerificationCompositeRule.d__1.MoveNext()
at System.Collections.Generic.List1..ctor(IEnumerable
1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
Cuz ya know, DNX
There should be a dnu command to check if the latest versions of dependencies are configured in project.json.
Something like: "dnu check update" which checks all dependencies and updates the project.json or prints the list or new releases inside console if "update" command is not provided.
Generate a list of all packages.
The split packages tool generates 3 list of packages: ship, no ship, and ext as project.json
files. It would be nice if it could generate 1 gigantic file with all the packages from a build...
I think @davidfowl needs the full list.
cc @javiercn
Then immediately enable rule (2) in Coherence-Signed.
We can't yet enable rule (1) because we're still missing some package metadata.
Example:
System.Net.Sockets.4.0.1-beta-23302
System.Net.Sockets.4.1.0-beta-23302
Right now it's 1.0.0, which means that we'd have to manually update it every time, or else it won't restore correctly. Right now the CI will keep building and pushing 1.0.0, but anyone with a package cache will never get the new build.
We should change this to 1.0.1-build-*
or something.
Ideally, all of our packages should be able to be restored without requiring any imports in the project.json file. Imports are required right now due to third party dependencies and obsolete project.json definitions in our packages.
We want to generate a project.json file without imports so that we can run dotnet restore on it and determine which packages are responsible for requiring imports in the project.json file.
Root:
Cause this issue on CI:
[13:49:03] fail: DependenciesPackager.Program[0]
[13:49:03] System.ArgumentException: An item with the same key has already been added. Key: NUGET_XMLDOC_MODE
[13:49:03] at System.ThrowHelper.ThrowAddingDuplicateWithKeyArgumentException(Object key)
[13:49:03] at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
[13:49:03] at DependenciesPackager.ProcessRunner.CreateProcessInfo()
[13:49:03] at DependenciesPackager.ProcessRunner.Run()
[13:49:03] at DependenciesPackager.PackageRestoreContext.RestoreAndRemoveUnecessaryFiles()
[13:49:03] at DependenciesPackager.Program.Execute()
An existing environment shouldn't fail the execution.
Right now it looks like it fails at the first package that is missing or new. It makes it very difficult to work with when a bunch of csv changes have to be made.
Rules like this would be kind of neat for other package authors.
Kind of like a baseline to ensure that your package have good metadata.
Here are my questions:
@ajaybhargavb can the nuget verifier check properties on the assemblies inside? Recently I made a change to embed the commit hash as an AssemblyMetadataAttribute
and it would be nice to check it for the packages that we ship
Ref: aspnet/Universe#256 (comment)
cc' @Eilon
Currently there is no support to hide Info log messages. It would be better to have the verifier just log messages for errors and warnings as part of the build.
We landed on not requiring listing individual packages to be listed in NuGetPackageVerifier.json. However the executable continues to print a warning for packages that aren't present:
Found 2 unprocessed packages. Every package in the repo should be listed in exactly one package set. Running default or all rules on unlisted packages.
Presumably unlisted packages should no longer be treated as a warning.
Right now, the dependencies packager tool is constructing paths using OriginalCase
package ID and version, not lowercase
in two places: hash path, directory name.
This change in NuGet, along with it's downstream affected parties is outlined here:
NuGet/Home#2522
I think the right fix here it to consume NuGet APIs to build these paths, much like what .NET CLI is doing (example). The NuGet type to use is probably VersionFolderPathResolver
.
If this work is not done, the DOTNET_HOSTING_OPTIMIZATION_CACHE
on Linux will no longer work, since the .deps.json
file of .NET Core 1.1.0 apps will have lowercase paths. Windows will keep working since it's a case-insensitive file system, but it should be fixed as well since this is basically just a convenient coincidence. From what I understand, this will not break the Linux app, it will just mean startup time is much slower.
Right now, AssemblyStrongNameRule
is hardcoded to a specific strong name. Right now, other users have to recompile the tool to set their own strong name information.
In general, rules could benefit from configurability feature (perhaps storing the configured vaues in the NuGetPackageVerifier.json
file per rule.
dnu restore gets lost recursing down node_modules directories I think
PS [dev] C:\User\Dev\ext\Universe\MusicStore>.\build.cmd
Attempting to gather dependencies information for package 'KoreBuild.0.2.1-rc1-15542' with respect to project 'packages', targeting 'Any,Version=v0.0'
Attempting to resolve dependencies for package 'KoreBuild.0.2.1-rc1-15542' with DependencyBehavior 'Lowest'
Resolving actions to install package 'KoreBuild.0.2.1-rc1-15542'
Resolved actions to install package 'KoreBuild.0.2.1-rc1-15542'
Adding package 'KoreBuild.0.2.1-rc1-15542' to folder 'packages'
Added package 'KoreBuild.0.2.1-rc1-15542' to folder 'packages'
Successfully installed 'KoreBuild 0.2.1-rc1-15542' to packages
Attempting to gather dependencies information for package 'Sake.0.2.2' with respect to project 'packages', targeting 'Any,Version=v0.0'
Attempting to resolve dependencies for package 'Sake.0.2.2' with DependencyBehavior 'Lowest'
Resolving actions to install package 'Sake.0.2.2'
Resolved actions to install package 'Sake.0.2.2'
Adding package 'Sake.0.2.2' to folder 'packages'
Added package 'Sake.0.2.2' to folder 'packages'
Successfully installed 'Sake 0.2.2' to packages
Determining latest version
'dnx-clr-win-x86.1.0.0-rc1-15844' is already installed in c:\Users\User\.dnx\runtimes\dnx-clr-win-x86.1.0.0-rc1-15844.
Adding c:\Users\User\.dnx\runtimes\dnx-clr-win-x86.1.0.0-rc1-15844\bin to process PATH
Adding c:\Users\User\.dnx\runtimes\dnx-clr-win-x86.1.0.0-rc1-15844\bin to user PATH
Updating alias 'default' to 'dnx-clr-win-x86.1.0.0-rc1-15844'
'dnx-coreclr-win-x86.1.0.0-rc1-15844' is already installed in c:\Users\User\.dnx\runtimes\dnx-coreclr-win-x86.1.0.0-rc1-15844.
Adding c:\Users\User\.dnx\runtimes\dnx-coreclr-win-x86.1.0.0-rc1-15844\bin to process PATH
'dnx-clr-win-x64.1.0.0-rc1-15844' is already installed in c:\Users\User\.dnx\runtimes\dnx-clr-win-x64.1.0.0-rc1-15844.
Adding c:\Users\User\.dnx\runtimes\dnx-clr-win-x64.1.0.0-rc1-15844\bin to process PATH
'dnx-coreclr-win-x64.1.0.0-rc1-15844' is already installed in c:\Users\User\.dnx\runtimes\dnx-coreclr-win-x64.1.0.0-rc1-15844.
Adding c:\Users\User\.dnx\runtimes\dnx-coreclr-win-x64.1.0.0-rc1-15844\bin to process PATH
Adding c:\Users\User\.dnx\runtimes\dnx-clr-win-x86.1.0.0-rc1-15844\bin to process PATH
info: Build v2: False
info: Exec
info: program: cmd
info: commandline: /C dnu restore --parallel
info: workingdir: C:\Dam\Dev\ext\Universe\MusicStore
Microsoft .NET Development Utility Clr-x86-1.0.0-rc1-15844
----------
System.IO.PathTooLongException: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
at System.IO.Path.GetDirectoryName(String path)
at System.IO.FileSystemEnumerableIterator`1..ctor(String path, String originalUserPath, String searchPattern, SearchOption searchOption, SearchResultHandler`1 resultHandler, Boolean checkHost)
at System.IO.Directory.EnumerateDirectories(String path)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.CollectProjectFiles(String directory, List`1 results)
at Microsoft.Dnx.Tooling.RestoreProjectsCollector.Find(String restoreDirectory, IEnumerable`1& projectJsonFiles)
at Microsoft.Dnx.Tooling.RestoreCommand.<Execute>d__68.MoveNext()
----------
Restore failed
The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
fail: Exit code 1 from cmd
fail: Stack trace:
at Viewf4002438f8ae4f6f8b7eb68e18e9a133.<RenderViewLevel0>b__9f() in c:\Users\User\AppData\Local\Temp\218894c96a1e4b568904f1a85e0e8e47-1.cs:line 8837
at Sake.Engine.Builder.BuilderBase`1.CallTarget(String name)
at Sake.Engine.Builder.BuilderBase`1.StartingTarget(String name)
at Viewf4002438f8ae4f6f8b7eb68e18e9a133.StartingTarget(String name) in c:\Users\User\AppData\Local\Temp\218894c96a1e4b568904f1a85e0e8e47-1.cs:line 1937
at Viewf4002438f8ae4f6f8b7eb68e18e9a133.<RenderViewLevel0>b__5e() in c:\Users\User\AppData\Local\Temp\218894c96a1e4b568904f1a85e0e8e47-1.cs:line 3112
at Sake.Engine.Builder.BuilderBase`1.CallTarget(String name)
at Sake.Engine.Builder.BuilderBase`1.StartingTarget(String name)
at Viewf4002438f8ae4f6f8b7eb68e18e9a133.StartingTarget(String name) in c:\Users\User\AppData\Local\Temp\218894c96a1e4b568904f1a85e0e8e47-1.cs:line 1937
at Viewf4002438f8ae4f6f8b7eb68e18e9a133.<RenderViewLevel0>b__5f() in c:\Users\User\AppData\Local\Temp\218894c96a1e4b568904f1a85e0e8e47-1.cs:line 3129
at Sake.Engine.Builder.BuilderBase`1.CallTarget(String name)
at Sake.Engine.Builder.BuilderBase`1.StartingTarget(String name)
at Viewf4002438f8ae4f6f8b7eb68e18e9a133.StartingTarget(String name) in c:\Users\User\AppData\Local\Temp\218894c96a1e4b568904f1a85e0e8e47-1.cs:line 1937
at Viewf4002438f8ae4f6f8b7eb68e18e9a133.<RenderViewLevel0>b__60() in c:\Users\User\AppData\Local\Temp\218894c96a1e4b568904f1a85e0e8e47-1.cs:line 3146
at Sake.Engine.Builder.BuilderBase`1.CallTarget(String name)
at Sake.Engine.Builder.BuilderBase`1.StartingTarget(String name)
at Viewf4002438f8ae4f6f8b7eb68e18e9a133.StartingTarget(String name) in c:\Users\User\AppData\Local\Temp\218894c96a1e4b568904f1a85e0e8e47-1.cs:line 1937
at Viewf4002438f8ae4f6f8b7eb68e18e9a133.<RenderViewLevel0>b__61() in c:\Users\User\AppData\Local\Temp\218894c96a1e4b568904f1a85e0e8e47-1.cs:line 3163
at Sake.Engine.Builder.BuilderBase`1.CallTarget(String name)
at Viewf4002438f8ae4f6f8b7eb68e18e9a133.<RenderViewLevel0>b__59() in c:\Users\User\AppData\Local\Temp\218894c96a1e4b568904f1a85e0e8e47-1.cs:line 3019
at Sake.Engine.Builder.BuilderBase`1.CallTarget(String name)
at Sake.Engine.SakeEngine.Execute(Options options)
at Sake.Engine.SakeEngine.Execute(String[] args)
at Sake.Program.Main(String[] args)
We can inspect the project.json of projects and parse the commonly used types to verify their existence.
These are being deprecated, we should switch to using NuGet APIs directly or similar from the CLI
The dependency graph of apps targeting netcoreapp1.1 will be slightly different from those using netcoreapp1.0. Build tools should account for this new TFM.
I found two instances where this might apply, and we should investigate if netcoreapp1.1 makes a difference:
SplitPackages only check compatibility with netcoreapp1.0.
DependencyPackager only crossgen's for netcoreapp1.0.
After this commit: 5c8657c
The cache.project.json contains nothing but Microsoft.NETCore.App.
The new build script doesn't include dnx anymore so the package verifier will not work.
Just so that we make fewer HTTP requests when running verification:
Microsoft.AspNetCore
Microsoft.AspNetCore.AzureAppServicesIntegration
Microsoft.AspNetCore.DataProtection.AzureStorage
Microsoft.AspNetCore.DataProtection.Redis
Microsoft.AspNetCore.Localization.Routing
Microsoft.AspNetCore.Mvc.Razor.ViewCompilation.Design
Microsoft.AspNetCore.Mvc.Razor.ViewCompilation.Tools
Microsoft.AspNetCore.Mvc.RazorPages
Microsoft.AspNetCore.Razor.Evolution
Microsoft.AspNetCore.ResponseCaching
Microsoft.AspNetCore.ResponseCaching.Abstractions
Microsoft.AspNetCore.ResponseCompression
Microsoft.AspNetCore.Rewrite
Microsoft.AspNetCore.SignalR
Microsoft.AspNetCore.SignalR.Redis
Microsoft.AspNetCore.Sockets
Microsoft.AspNetCore.WebSockets
Microsoft.EntityFrameworkCore.Tools.DotNet
Microsoft.Extensions.Configuration.AzureKeyVault
Microsoft.Extensions.Logging.AzureAppServices
Microsoft.VisualStudio.Web.BrowserLink
Add a rule to check for bad references to prevent issues like these:
It's really easy to mess something up when adding projects, and probably better to put this verification in the inner loop since it's pretty fast.
Suggested by @javiercn in PR #95
I would suggest that if you are performing this overhaul of the tool, you try and follow an alternative approach for doing crossgen that consists on passing the semi colon separated list of paths to the tool instead of copying everything to a folder and running crossgen.
This will have a noticeable improvement in performance for the tool and if the size of the command line becomes a problem, you can write a response file in the following format
/nologo
/App_Paths path1;path2;....
And then call crossgen with crossgen.exe /in PathToin.dll /out /PathToOuthDll @response.rsp
This will avoid copying all the files to an intermediate folder which takes like 20% of the tools time if not more.
What I'm suggesting is, get all the dependencies from the project context, build a rsp file with /App_Paths < and pass that to the crossgen input with @myresponsefile.rsp instead of copying the assemblies to a staging folder
SplitPackages current generates a project.json file. Downstream tools use this to run a restore.
We will need to update the tool to generate an xml file that uses PackageReference
's instead.
There are two different rulesets. Coherence-signed adds additional rules, which means rule failures aren't discoverable by running build --for-ci verify
on local runs.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.