Giter Club home page Giter Club logo

source-build's Introduction

.NET Source-Build

Please use GitHub discussions to see announcements, ask questions, make suggestions, and share information with other members of the source-build community.

This repo is the starting point for building .NET from source. It contains documentation, tools, and is used for issue tracking.

Prerequisites

Building .NET 8.0+

.NET 8.0 and newer will be built from the dotnet/dotnet repo. Clone the dotnet/dotnet repo and check out the tag for the desired release. Then, follow the instructions in dotnet/dotnet's README to build .NET from source.

Building .NET 7.0 and .NET 6.0

.NET 6.0 and 7.0 are built from source using the dotnet/installer repo. Clone the dotnet/installer repo and check out the tag for the desired release. Then, follow the instructions in dotnet/installer's README to build .NET from source. Please see the support section below to see which feature branches are currently supported.

The source-build repository doesn't currently support Windows. See source-build#1190.

Source-build goals

There are two primary goals of the source-build effort:

  1. Increase .NET adoption by focusing on making .NET available everywhere

    If .NET was available everywhere - including Linux distributions and package managers like Homebrew - as a first class citizen, it would make .NET a more attractive option for developers who might otherwise look at other languages or runtimes. Users would be more likely to start using and keep using .NET if .NET is available on their development and release platforms of choice.

    To achieve this, we try to make it easier for community and partner maintainers to build and release .NET for their platforms. We need to make sure source-build satisfies the official packaging rules of commonly used Linux distributions, such as Fedora and Debian. Many Linux distributions have similar rules. These rules tend to have three main principles:

    • Limited or no network access
    • Consistent reproducibility
    • Source code for everything

  2. Make maintenance of the .NET product easier

    The current way of making changes to .NET during a servicing release is to make changes to individual product repositories and then adjust the dependency versions to flow the changes to the next set of repositories. This is repeated until all the repositories are updated. If there's an issue discovered late in the release cycle, the fixes and the dependency updates need to be re-done quickly, which becomes difficult. It's also difficult to verify that the issue is fixed in the final product. It would be much easier to make and test product-wide changes if we could make atomic changes to one repository and be able to build the whole product based on that source code at once.

    In addition, getting source-build fully functional would provide everyone with a place to try changes that would otherwise require a lot of coordination between multiple repositories - such as landing features that require changes to both the runtime and the SDK.

    For more details about this Unified Build, see this overview.

Source-build can help achieve both these goals by making it easier for everyone to build and release the entire .NET product end-to-end.

What does the source-build infrastructure do?

Source-build solves common challenges that most developers encounter when trying to build the whole .NET SDK from source.

  • .NET is composed of many repositories that need to be built at a specific combination of commits.
  • Each repository's build output needs to flow into the next repository's build.
  • By default, most .NET repositories download prebuilt binary dependencies from online sources. These are forbidden by typical Linux distribution rules, and interfere with build output flow.
  • Some of the binary build dependencies are under a proprietary license, making it difficult to build everything without taking accidental dependencies on non-Open Source code. Many of our community members and partners want to build an Open Source .NET only.
  • Flowing the build output automatically also means we can make and test changes that require coordination across a number of repositories.
  • Nearly all .NET repositories require the .NET SDK to build. This is a circular dependency, which presents a bootstrapping problem, whether that's bringing up new architectures (eg, riscv), or support new operating systems (eg, FreeBSD).
  • Microsoft controls the SDKs used by dotnet-install.sh scripts and Microsoft.* and runtime.* nuget packages at nuget.org; source-build makes it possible for the community to bring up the platforms without Microsoft having to accept/bless things.

Comprehensive Guidelines

.NET in Linux Distributions

Distro Package Feed Maintainer
Alpine Community @ayakael
Arch Linux Community
Arch User Repo		 @alucryd
CentOS Stream CentOS Stream Mirror @omajid
Fedora Default @omajid
Homebrew Formula @asbjornu
Red Hat Enterprise Linux Default @omajid
Ubuntu Default
Personal Package Archives		 @mirespace

Support

.NET Source-Build is supported on the oldest available .NET SDK feature update for each major release, and on Linux only. For example, if .NET 6.0.1xx, 6.0.2xx, 7.0.1xx, and 7.0.2xx feature updates are available from dotnet.microsoft.com, Source-Build will support 6.0.1xx and 7.0.1xx.

For the latest information about Source-Build support for new .NET versions, please check our GitHub Discussions page for announcements.

License

This repo is licensed under the MIT license.

source-build's People

Contributors

adaggarwal avatar ali-turan7 avatar am11 avatar brettfo avatar chcosta avatar crummel avatar dagood avatar dotnet-bot avatar dotnet-maestro-bot avatar dotnet-maestro[bot] avatar dseefeld avatar eerhardt avatar ellahathaway avatar ellismg avatar janvorli avatar jaredpar avatar karajas avatar lbussell avatar michaelsimons avatar mmitche avatar mthalman avatar nguerrera avatar nikolamilosavljevic avatar oleksandr-didyk avatar omajid avatar premun avatar safern avatar tmat avatar tmds avatar weshaggard avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

crummel karajas tmds jaredpar weshaggard zeroinfinite dagood eerhardt am11 dleeapho dseefeld gitter-badger slamj1 nguerrera janvorli mmitche dagood-bot nkolev92 codito omajid tmat brettfo sunandabalu aslicerh dotnet-maestro-bot jddurand dotnetrt luki122 mlorbetske sdmaclea ashrafxcoder mraipsec-mra peterhuene 333fred shawnro joperezr safern amir94fs bruceforstall wfurt qmlnet eroen criteo-forks molongwudi chcosta regisbsb cybersys wjk nikolamilosavljevic mcineral williamteoh1976 jashook dominofire jcagme waqashaneef adaggarwal scott-wilder tharukia lin-deng twanano phipat0079 maibroadcast azurementor wilvk anilrath51 terrajobst rheoz lvyitian connectionmaster abhitejjohn nohwnd bprakash92 dystudio theaoqi gsvm-stage chadnedzlek chasend0990 tomaszek2902 antonlapounov aurelsarac michaelsimons ladipro seekingmeaning persianyagami90xs trevorgrayson devbox10 astralchan shushanhf jonsequitur lbussell zhamppx97 mr-tao jonfortescue sopra-steria-norge-cs hdost isabella232 skyn9ne open-dotnet ayakael adapa158

source-build's Issues

Build CoreFX on Linux

Everything except for Microsoft.VisualBasic can be built from source on *nix via build.sh. The VisualBasic issues are due to a bug in MSBuild.

Build Roslyn on Linux

cibuild.sh will actually build a fair amount of the product, but it also runs tests and does not produce nuget packages.

The build is also not an "official" build so the assembly versions are all wrong.

Working with submodules

While getting started in this repo I forget to update/initialize the submodules. I suspect that might be a common issue for folks so I was thinking we should consider adding a git submodule update --init to the beginning of the build scripts but given I'm pretty new to submodules I don't know if that will start to cause issue if there are changes to the modules or if there are patches applied to them. Anyone have enough experience to advise for or against always doing an update/init from the build scripts?

Build NuGet.Client on Linux

The existing build.sh actually just builds and runs tests. It also doesn't produce an nuget packages and for some projects (which cross compile for Desktop and netstandard) requires reference assembly goop that is not present without Mono.

cli including store packages [2.0]

@omajid and I asked some questions on this PR: dotnet/cli#6460 which were unanswered, so I am creating this issue.

In the meanwhile I got some more info as I hit an issue when publishing an mvc application: https://github.com/dotnet/cli/issues/6516.

I think the PR is to populate the store with typical ASP.NET packages. Those packages will no longer be included on publish.

For RHEL, we don't want to ship binary blobs we are not building. So it would be great if we can build the CLI and not include the packages. Instead, we can include the manifest file. Then the user can use the manifest to install those packages in his store.

Does this make sense? Can we make this happen?

CC @JunTaoLuo @ellismg @bleroy @davidfowl @eerhardt @livarcocc @aslicerh

Do not use NI/R2R images in prebuilt tools

Prebuilt tools (cli, roslyn's toolchain, etc) shouldn't use NI/R2R images. Instead, they should be IL only and after boot strapping coreclr, we can crossgen them as needed.

This is needed because R2R images are not portable across either architectures or OSes (e.g. Windows, Linux, OSX or other *NIXes).

Obviously, NI images are even more fragile.

NuGet restore broken

Ran build.sh against master today and got the following error:

/Users/jaredpar/code/source-build/packages/microsoft.dotnet.buildtools/2.0.0-prerelease-01721-01/lib/tool-runtime/project.csproj : error NU1003: PackageTargetFallback and AssetTargetFallback cannot be used together. Remove PackageTargetFallback(deprecated) references from the project environment.

This happens during the init-tools.sh script and blocks most repo operations.

Build AppInsights-dotnet on Linux

The build system looks complication but at least it is MSBuild based. Until this is built we can either pull it from a binary package or patch CLI to not depend on it.

BuildTools version

It looks like there are no recent tags/releases of this package which is used by source-build. BuildToolsVersion.txt is referring to a myget package.

Roadmap for implementing support for OSes

In some projects I am working on an ability to use this build system would be very helpful. To enable any planing around this I would appreciate if team would share feature implementation roadmap indicating approximate times when support for .NET supported OSes will be available in this repo. This could potentially save some duplicate work on my side.

Statically define variables for a repository build

When we do the work to generate build.props (renamed to phases.props), we should also generate a switch statement so that we can statically define the organization, branch, etc... info rather than having to pass it directly when calling a repository build. ie, I still think we should only be passing "repositoryName" and maybe "pathtoproj", everything else should be statically defined...

https://github.com/dotnet/source-build/blob/master/build.proj#L26

Instead of what we have today...

https://github.com/dotnet/source-build/blob/master/phases.props

... we would have something more like...

  <ItemGroup>
    <Phases Include="0;1;2;3;4;5;6" />
  </ItemGroup>

  <ItemGroup>
    <Repository Include="standard" >
      <BuildPhase>0</BuildPhase>
      <PathToRepo>$(PathToStandard)</PathToRepo>
    </Repository>
    <Repository Include="libuv" >
      <BuildPhase>0</BuildPhase>
      <PathToRepo>$(PathToLibuv)</PathToRepo>
      <IsJenkinsBuild>true</IsJenkinsBuild>
    </Repository>
  </ItemGroup>

  <Choose>
    <When Condition="'$(RepositoryName)' == 'standard'">
      <PropertyGroup>
        <Organization>dotnet</Organization>
        <Branch>master</Branch>
      </PropertyGroup>
    </When>
    <When Condition="'$(RepositoryName)' == libuv">
      <PropertyGroup>
        <Organization>ellismg</Organization>
        <Branch>master</Branch>
      </PropertyGroup>
    </When>
  </Choose>

/cc @ellismg @karajas @bryanAR

Patches fail to apply on Fedora 25

Some patches fail to apply due to different line endings:

  patch -p1 --ignore-whitespace -i /home/tmds/repos/source-build/patches/nuget-client/0001-Revert-38949ce93081016167dfd8e37e4e897bedf47d65-DOS-.patch
  (Stripping trailing CRs from patch; use --binary to disable.)
  patching file build/common.project.props
  Hunk #1 FAILED at 14 (different line endings).
  1 out of 1 hunk FAILED -- saving rejects to file build/common.project.props.rej
  (Stripping trailing CRs from patch; use --binary to disable.)
  patching file src/NuGet.Core/NuGet.Build.Tasks.Pack/NuGet.Build.Tasks.Pack.csproj
  Hunk #1 FAILED at 5 (different line endings).
  Hunk #2 FAILED at 51 (different line endings).
  2 out of 2 hunks FAILED -- saving rejects to file src/NuGet.Core/NuGet.Build.Tasks.Pack/NuGet.Build.Tasks.Pack.csproj.rej

This is not failing in CI since it is using a different patch command

source-build/targets/repository.proj

Lines 45 to 46 in 14d33dc

<PatchCommand Condition="'$(IsJenkinsBuild)' == 'true'">git apply --ignore-whitespace --whitespace=nowarn</PatchCommand>
<PatchCommand Condition="'$(IsJenkinsBuild)' != 'true'">patch -p1 --ignore-whitespace -i</PatchCommand>

CI should use the same command as non-CI builds.
Can we use the git apply command always (since that is working)? Or should we use patch also in CI?

CC @crummel @omajid

Exclude NuGet.Client tests from tarball

There are a lot of NuGet.Client tests that get brought in via the nuget-client submodule. We should try to exclude these from the output source tarball. In addition to general economy, some of these have odd licenses that don't jibe with the source-build license. Example:

src/nuget-client/submodules/NuGet.Services.Metadata/tests/AAD/NuGet.Test.WebSite/Scripts/jquery-1.10.2.intellisense.js src/nuget-client/submodules/NuGet.Services.Metadata/tests/AAD/NuGet.Test.WebSite/Scripts/jquery.validate-vsdoc.js src/nuget-client/submodules/NuGet.Services.Metadata/tests/AAD/NuGet.Test.WebSite/Scripts/jquery.validate.unobtrusive.js src/nuget-client/submodules/NuGet.Services.Metadata/tests/AAD/NuGet.Test.WebSite/Scripts/jquery.validate.unobtrusive.min.js src/nuget-client/submodules/NuGet.Services.Metadata/tests/AAD/NuGet.Test.WebSite/Scripts/modernizr-2.6.2.js src/nuget-client/submodules/NuGet.Services.Metadata/tests/PublishTestDriverWebSite/Scripts/jquery.validate-vsdoc.js src/nuget-client/submodules/NuGet.Services.Metadata/tests/PublishTestDriverWebSite/Scripts/jquery.validate.unobtrusive.js src/nuget-client/submodules/NuGet.Services.Metadata/tests/PublishTestDriverWebSite/Scripts/jquery.validate.unobtrusive.min.js src/nuget-client/test/EndToEnd/ProjectTemplates/JScriptVisualBasicLightSwitchProjectTemplate.zip/jquery-1.9.1.js src/nuget-client/test/EndToEnd/ProjectTemplates/JScriptVisualBasicLightSwitchProjectTemplate.zip/jquery-1.9.1.min.js src/nuget-client/test/EndToEnd/ProjectTemplates/JScriptVisualBasicLightSwitchProjectTemplate.zip/jquery.mobile-1.3.0.css src/nuget-client/test/EndToEnd/ProjectTemplates/JScriptVisualBasicLightSwitchProjectTemplate.zip/jquery.mobile-1.3.0.js src/nuget-client/test/EndToEnd/ProjectTemplates/JScriptVisualBasicLightSwitchProjectTemplate.zip/jquery.mobile-1.3.0.min.css src/nuget-client/test/EndToEnd/ProjectTemplates/JScriptVisualBasicLightSwitchProjectTemplate.zip/jquery.mobile-1.3.0.min.js src/nuget-client/test/EndToEnd/ProjectTemplates/JScriptVisualBasicLightSwitchProjectTemplate.zip/jquery.mobile.structure-1.3.0.css src/nuget-client/test/EndToEnd/ProjectTemplates/JScriptVisualBasicLightSwitchProjectTemplate.zip/jquery.mobile.structure-1.3.0.min.css src/nuget-client/test/EndToEnd/ProjectTemplates/JScriptVisualBasicLightSwitchProjectTemplate.zip/jquery.mobile.theme-1.3.0.css src/nuget-client/test/EndToEnd/ProjectTemplates/JScriptVisualBasicLightSwitchProjectTemplate.zip/jquery.mobile.theme-1.3.0.min.css

/cc @omajid

Build fails on Fedora 26

Build (./build.sh) fails when applying a patch:

"/home/omajid/devel/dotnet/source-build/build.proj" (default target) (1) ->
"/home/omajid/devel/dotnet/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/omajid/devel/dotnet/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:3) ->
"/home/omajid/devel/dotnet/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:6) ->
(Build target) ->
  /home/omajid/devel/dotnet/source-build/src/corefx/Tools/depProj.targets(83,5): error : Error no assets were resolved from NuGet packages. [/home/omajid/devel/dotnet/so
urce-build/src/corefx/external/ilasm/ilasm.depproj] [/home/omajid/devel/dotnet/source-build/targets/repository.proj]
  /home/omajid/devel/dotnet/source-build/src/corefx/Tools/depProj.targets(83,5): error : Error no assets were resolved from NuGet packages. [/home/omajid/devel/dotnet/so
urce-build/src/corefx/external/ilasm/ilasm.depproj] [/home/omajid/devel/dotnet/source-build/targets/repository.proj]
  /home/omajid/devel/dotnet/source-build/targets/repository.proj(75,5): error MSB3073: The command "/home/omajid/devel/dotnet/source-build/src/corefx/build.sh -Release -
buildArch=x64 -portable=false -- /p:ILLinkTrimAssembly=false" exited with code 1.

"/home/omajid/devel/dotnet/source-build/build.proj" (default target) (1) ->
"/home/omajid/devel/dotnet/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/omajid/devel/dotnet/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:10) ->
"/home/omajid/devel/dotnet/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:12) ->
(ApplyPatches target) ->
  /home/omajid/devel/dotnet/source-build/targets/repository.proj(49,5): error MSB3073: The command "patch -p1 --binary --ignore-whitespace -i /home/omajid/devel/dotnet/$ource-build/patches/nuget-client/0001-Revert-38949ce93081016167dfd8e37e4e897bedf47d65-DOS-.patch" exited with code 1.


"/home/omajid/devel/dotnet/source-build/build.proj" (default target) (1) ->
"/home/omajid/devel/dotnet/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/omajid/devel/dotnet/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:38) ->
(Build target) ->
  /home/omajid/devel/dotnet/source-build/Tools/dotnetcli/sdk/2.0.0-preview3-006845/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(141,5)$ error : The file '/home/omajid/devel/dotnet/source-build/src/xliff-tasks/bin/Release/XliffTasks/netcoreapp1.1/XliffTasks.dll' to be packed was not found on disk. [/hom$/omajid/devel/dotnet/source-build/src/xliff-tasks/src/XliffTasks/XliffTasks.csproj] [/home/omajid/devel/dotnet/source-build/targets/repository.proj]
  /home/omajid/devel/dotnet/source-build/Tools/dotnetcli/sdk/2.0.0-preview3-006845/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(141,5)$ error : The file '/home/omajid/devel/dotnet/source-build/src/xliff-tasks/bin/Release/XliffTasks/netcoreapp1.1/XliffTasks.dll' to be packed was not found on disk. [/hom$/omajid/devel/dotnet/source-build/src/xliff-tasks/src/XliffTasks/XliffTasks.csproj] [/home/omajid/devel/dotnet/source-build/targets/repository.proj]
  /home/omajid/devel/dotnet/source-build/targets/xliff-tasks.targets(8,5): error MSB3073: The command "/home/omajid/devel/dotnet/source-build/Tools/dotnetcli/dotnet bui$d /home/omajid/devel/dotnet/source-build/src/xliff-tasks/build/build.proj /t:Pack /v:normal /flp:Verbosity=Diag;LogFile=/home/omajid/devel/dotnet/source-build/src/xliff$tasks/msbuild.log" exited with code -1. [/home/omajid/devel/dotnet/source-build/targets/repository.proj]

This is on Fedora 26:

$ cat /etc/os-release 
NAME=Fedora
VERSION="26 (Workstation Edition)"
ID=fedora
VERSION_ID=26
PRETTY_NAME="Fedora 26 (Workstation Edition)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:26"
HOME_URL="https://fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=26
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=26
PRIVACY_POLICY_URL=https://fedoraproject.org/wiki/Legal:PrivacyPolicy
VARIANT="Workstation Edition"
VARIANT_ID=workstation
$ uname -a
Linux trull 4.11.11-300.fc26.x86_64 #1 SMP Mon Jul 17 16:32:11 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Reverting #103 did not help.

"Rebuild" fails on patches

Related to #113, it looks like in general this repo expects a clean environment.

Steps to reproduce:

  • ./build.sh
  • let it do it's thing, cancel if you're impatient
  • ./build.sh again

Expected result:

  • build starts over, completes successfully

Observed result:

  • ApplyPatches step fails

Not sure what's better, a git reset --hard of the submodules that get patched (as ./clean-submodules.sh does), or just continuing on if the patch fails (|| true) -- I'd suggest the latter, as then things that have already been built don't need to be rebuilt, and users can always explicitly call ./clean-submodules.sh themselves.

Console output:

...
Done building target "BuildReferenceAssemblies" in project "build.proj".
Target "Build" in project "/home/vtsv/source-build/build.proj" (entry point):
Using "Message" task from assembly "Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
Task "Message"
  Build Environment: arm Release Linux ubuntu.16.04-x64
Done executing task "Message".
Using "MSBuild" task from assembly "Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
Task "MSBuild"
  Additional Properties for project "/home/vtsv/source-build/targets/repository.proj":
    RepositoryName=core-setup
    PathToRepo=
    RepositoryOrganization=dotnet
    RepositoryBranch=release/2.0.0
Property reassignment: $(DefaultRepository)="core-setup" (previous value: "cli") at /home/vtsv/source-build/dir.props (71,5)
Property reassignment: $(BuildArguments)="-ConfigurationGroup=Release -PortableBuild=false -strip-symbols -SkipTests=true  -TargetArchitecture=arm -DistroRid=linux-arm -DisableCrossgen=true -SkipTests=true -CrossBuild=true" (previous value: "-ConfigurationGroup=Release -PortableBuild=false -strip-symbols -SkipTests=true ") at /home/vtsv/source-build/targets/core-setup.props (6,5)
Property reassignment: $(BuildCommand)="ROOTFS_DIR=/home/vtsv/source-build/cross/rootfs/arm /home/vtsv/source-build/src/core-setup/build.sh -ConfigurationGroup=Release -PortableBuild=false -strip-symbols -SkipTests=true  -TargetArchitecture=arm -DistroRid=linux-arm -DisableCrossgen=true -SkipTests=true -CrossBuild=true -- /p:BuildDebPackage=false" (previous value: "/home/vtsv/source-build/src/core-setup/build.sh -ConfigurationGroup=Release -PortableBuild=false -strip-symbols -SkipTests=true  -TargetArchitecture=arm -DistroRid=linux-arm -DisableCrossgen=true -SkipTests=true -CrossBuild=true -- /p:BuildDebPackage=false") at /home/vtsv/source-build/targets/core-setup.props (8,5)
Project "/home/vtsv/source-build/build.proj" (1) is building "/home/vtsv/source-build/targets/repository.proj" (2) on node 1 (BuildRepositoryAndDependencies target(s)).
Building with tools version "15.0".
Target "SetNuGetPackagesEnvironment" skipped, due to false condition; ('$(ArchiveDownloadedPackages)' == 'true') was evaluated as ('' == 'true').
Target "BuildRepositoryReferences" in project "/home/vtsv/source-build/targets/repository.proj" (target "BuildRepositoryAndDependencies" depends on it):
Task "MSBuild"
  Additional Properties for project "repository.proj":
    RepositoryName=standard
    PathToRepo=
    RepositoryOrganization=dotnet
    RepositoryBranch=release/2.0.0
Property reassignment: $(DefaultRepository)="core-setup" (previous value: "cli") at /home/vtsv/source-build/dir.props (71,5)
Project "/home/vtsv/source-build/targets/repository.proj" (2) is building "/home/vtsv/source-build/targets/repository.proj" (2:2) on node 1 (BuildRepositoryAndDependencies target(s)).
Building with tools version "15.0".
Target "SetNuGetPackagesEnvironment" skipped, due to false condition; ('$(ArchiveDownloadedPackages)' == 'true') was evaluated as ('' == 'true').
Target "BuildRepositoryReferences" skipped, due to false condition; ('@(RepositoryReference)' != '' and '$(BuildDependentRepositories)' != 'false') was evaluated as ('' != '' and '' != 'false').
Target "ApplyPatches" in project "/home/vtsv/source-build/targets/repository.proj" (target "Build" depends on it):
Task "Exec"
  Environment Variables passed to tool:
    DOTNET_RUNTIME_ID=ubuntu.16.04-x64
    DOTNET_TOOL_DIR=/home/vtsv/source-build/Tools/dotnetcli/
    BUILD_TOOLS_TOOL_DIR=/home/vtsv/source-build/Tools/
    BUILDTOOLS_SKIP_CROSSGEN=1
    LatestCommit=d318b764a689cfe285d8484bda918646905664e7
    OfficialBuildId=20170719-03
    VersionSeedDate=20170719-03
    OverridePackageSource=/home/vtsv/source-build/bin/obj/arm/Release/source-built/
  patch -p1 --ignore-whitespace -i /home/vtsv/source-build/patches/standard/0001-Respect-NUGET_PACKAGES-if-it-is-set.patch
  patching file dir.props
  Reversed (or previously applied) patch detected!  Assume -R? [n]
  Apply anyway? [n]
  Skipping patch.
  2 out of 2 hunks ignored -- saving rejects to file dir.props.rej
/home/vtsv/source-build/targets/repository.proj(49,5): error MSB3073: The command "patch -p1 --ignore-whitespace -i /home/vtsv/source-build/patches/standard/0001-Respect-NUGET_PACKAGES-if-it-is-set.patch" exited with code 1.
Done executing task "Exec" -- FAILED.
Done building target "ApplyPatches" in project "repository.proj" -- FAILED.
Done Building Project "/home/vtsv/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target(s)) -- FAILED.
BuildRepositoryReferences:
  Additional Properties for project "repository.proj":
    RepositoryName=coreclr
    PathToRepo=
    RepositoryOrganization=dotnet
    RepositoryBranch=release/2.0.0
Property reassignment: $(DefaultRepository)="core-setup" (previous value: "cli") at /home/vtsv/source-build/dir.props (71,5)
Property reassignment: $(BuildArguments)="arm Release skiptests -PortableBuild=false  skipnuget cross -skiprestore stripSymbols" (previous value: "arm Release skiptests -PortableBuild=false ") at /home/vtsv/source-build/targets/coreclr.props (6,5)
Property reassignment: $(BuildCommand)="ROOTFS_DIR=/home/vtsv/source-build/cross/rootfs/arm /home/vtsv/source-build/src/coreclr/build.sh arm Release skiptests -PortableBuild=false  skipnuget cross -skiprestore stripSymbols" (previous value: "/home/vtsv/source-build/src/coreclr/build.sh arm Release skiptests -PortableBuild=false  skipnuget cross -skiprestore stripSymbols") at /home/vtsv/source-build/targets/coreclr.props (8,5)
Project "/home/vtsv/source-build/targets/repository.proj" (2) is building "/home/vtsv/source-build/targets/repository.proj" (2:3) on node 1 (BuildRepositoryAndDependencies target(s)).
Building with tools version "15.0".
Target "SetNuGetPackagesEnvironment" skipped, due to false condition; ('$(ArchiveDownloadedPackages)' == 'true') was evaluated as ('' == 'true').
Target "BuildRepositoryReferences" skipped, due to false condition; ('@(RepositoryReference)' != '' and '$(BuildDependentRepositories)' != 'false') was evaluated as ('' != '' and '' != 'false').
Target "ApplyPatches" in project "/home/vtsv/source-build/targets/repository.proj" (target "Build" depends on it):
Task "Exec"
  Environment Variables passed to tool:
    DOTNET_RUNTIME_ID=ubuntu.16.04-x64
    DOTNET_TOOL_DIR=/home/vtsv/source-build/Tools/dotnetcli/
    BUILD_TOOLS_TOOL_DIR=/home/vtsv/source-build/Tools/
    BUILDTOOLS_SKIP_CROSSGEN=1
    LatestCommit=36f70fa4be4bd37d4d76d06dd2cc433ff46351bd
    OfficialBuildId=20170719-03
    VersionSeedDate=20170719-03
  patch -p1 --ignore-whitespace -i /home/vtsv/source-build/patches/coreclr/0001-Respect-NUGET_PACKAGES-if-it-is-set.patch
  patching file build.sh
  Reversed (or previously applied) patch detected!  Assume -R? [n]
  Apply anyway? [n]
  Skipping patch.
  1 out of 1 hunk ignored -- saving rejects to file build.sh.rej
  patching file dir.props
  Reversed (or previously applied) patch detected!  Assume -R? [n]
  Apply anyway? [n]
  Skipping patch.
  1 out of 1 hunk ignored -- saving rejects to file dir.props.rej
/home/vtsv/source-build/targets/repository.proj(49,5): error MSB3073: The command "patch -p1 --ignore-whitespace -i /home/vtsv/source-build/patches/coreclr/0001-Respect-NUGET_PACKAGES-if-it-is-set.patch" exited with code 1.
Done executing task "Exec" -- FAILED.
Done building target "ApplyPatches" in project "repository.proj" -- FAILED.
Done Building Project "/home/vtsv/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target(s)) -- FAILED.
BuildRepositoryReferences:
  Additional Properties for project "repository.proj":
    RepositoryName=corefx
    PathToRepo=
    RepositoryOrganization=dotnet
    RepositoryBranch=release/2.0.0
Property reassignment: $(DefaultRepository)="core-setup" (previous value: "cli") at /home/vtsv/source-build/dir.props (71,5)
Property reassignment: $(BuildCommand)="ROOTFS_DIR=/home/vtsv/source-build/cross/rootfs/arm /home/vtsv/source-build/src/corefx/build.sh -Release -buildArch=arm -portable=false -- /p:ILLinkTrimAssembly=false" (previous value: "/home/vtsv/source-build/src/corefx/build.sh -Release -buildArch=arm -portable=false -- /p:ILLinkTrimAssembly=false") at /home/vtsv/source-build/targets/corefx.props (6,5)
Project "/home/vtsv/source-build/targets/repository.proj" (2) is building "/home/vtsv/source-build/targets/repository.proj" (2:4) on node 1 (BuildRepositoryAndDependencies target(s)).
Building with tools version "15.0".
Target "SetNuGetPackagesEnvironment" skipped, due to false condition; ('$(ArchiveDownloadedPackages)' == 'true') was evaluated as ('' == 'true').
Target "BuildRepositoryReferences" in project "/home/vtsv/source-build/targets/repository.proj" (target "BuildRepositoryAndDependencies" depends on it):
Task "MSBuild"
  Additional Properties for project "repository.proj":
    RepositoryName=standard
    PathToRepo=
    RepositoryOrganization=dotnet
    RepositoryBranch=release/2.0.0
Project "/home/vtsv/source-build/targets/repository.proj" (2:4) is building "/home/vtsv/source-build/targets/repository.proj" (2:5) on node 1 (BuildRepositoryAndDependencies target(s)).
Building with tools version "15.0".
Target "SetNuGetPackagesEnvironment" skipped, due to false condition; ('$(ArchiveDownloadedPackages)' == 'true') was evaluated as ('' == 'true').
Target "BuildRepositoryAndDependencies" skipped. Previously built unsuccessfully.
Done Building Project "/home/vtsv/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target(s)) -- FAILED.
BuildRepositoryReferences:
  Additional Properties for project "repository.proj":
    RepositoryName=coreclr
    PathToRepo=
    RepositoryOrganization=dotnet
    RepositoryBranch=release/2.0.0
Project "/home/vtsv/source-build/targets/repository.proj" (2:4) is building "/home/vtsv/source-build/targets/repository.proj" (2:6) on node 1 (BuildRepositoryAndDependencies target(s)).
Building with tools version "15.0".
Target "SetNuGetPackagesEnvironment" skipped, due to false condition; ('$(ArchiveDownloadedPackages)' == 'true') was evaluated as ('' == 'true').
Target "BuildRepositoryAndDependencies" skipped. Previously built unsuccessfully.
Done Building Project "/home/vtsv/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target(s)) -- FAILED.
Done executing task "MSBuild" -- FAILED.
Done building target "BuildRepositoryReferences" in project "repository.proj" -- FAILED.
Done Building Project "/home/vtsv/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target(s)) -- FAILED.
BuildRepositoryReferences:
  Additional Properties for project "repository.proj":
    RepositoryName=newtonsoft-json
    PathToRepo=
    RepositoryOrganization=JamesNK
    RepositoryBranch=master
Property reassignment: $(DefaultRepository)="core-setup" (previous value: "cli") at /home/vtsv/source-build/dir.props (71,5)
Project "/home/vtsv/source-build/targets/repository.proj" (2) is building "/home/vtsv/source-build/targets/repository.proj" (2:7) on node 1 (BuildRepositoryAndDependencies target(s)).
Building with tools version "15.0".
Target "SetNuGetPackagesEnvironment" skipped, due to false condition; ('$(ArchiveDownloadedPackages)' == 'true') was evaluated as ('' == 'true').
Target "BuildRepositoryReferences" skipped, due to false condition; ('@(RepositoryReference)' != '' and '$(BuildDependentRepositories)' != 'false') was evaluated as ('' != '' and '' != 'false').
Target "ApplyPatches" in project "/home/vtsv/source-build/targets/repository.proj" (target "Build" depends on it):
Task "Exec"
  Environment Variables passed to tool:
    DOTNET_RUNTIME_ID=ubuntu.16.04-x64
    DOTNET_TOOL_DIR=/home/vtsv/source-build/Tools/dotnetcli/
    BUILD_TOOLS_TOOL_DIR=/home/vtsv/source-build/Tools/
    BUILDTOOLS_SKIP_CROSSGEN=1
  patch -p1 --ignore-whitespace -i /home/vtsv/source-build/patches/newtonsoft-json/0001-Run-dotnet-migrate-to-generate-Newtonsoft.Json.Dotne.patch
  The next patch would create the file Src/Newtonsoft.Json/Newtonsoft.Json.Dotnet.csproj,
  which already exists!  Assume -R? [n]
  Apply anyway? [n]
  Skipping patch.
  1 out of 1 hunk ignored
  The next patch would delete the file Src/Newtonsoft.Json/Newtonsoft.Json.Dotnet.xproj,
  which does not exist!  Assume -R? [n]
  Apply anyway? [n]
  Skipping patch.
  1 out of 1 hunk ignored
  The next patch would delete the file Src/Newtonsoft.Json/project.json,
  which does not exist!  Assume -R? [n]
  Apply anyway? [n]
  Skipping patch.
  1 out of 1 hunk ignored
/home/vtsv/source-build/targets/repository.proj(49,5): error MSB3073: The command "patch -p1 --ignore-whitespace -i /home/vtsv/source-build/patches/newtonsoft-json/0001-Run-dotnet-migrate-to-generate-Newtonsoft.Json.Dotne.patch" exited with code 1.
Done executing task "Exec" -- FAILED.
Done building target "ApplyPatches" in project "repository.proj" -- FAILED.
Done Building Project "/home/vtsv/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target(s)) -- FAILED.
Done executing task "MSBuild" -- FAILED.
Done building target "BuildRepositoryReferences" in project "repository.proj" -- FAILED.
Done Building Project "/home/vtsv/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target(s)) -- FAILED.
Done executing task "MSBuild" -- FAILED.
Done building target "Build" in project "build.proj" -- FAILED.
Done Building Project "/home/vtsv/source-build/build.proj" (default targets) -- FAILED.

Build FAILED.

"/home/vtsv/source-build/build.proj" (default target) (1) ->
(BuildReferenceAssemblies target) ->
  /home/vtsv/source-build/Tools/dotnetcli/sdk/2.0.0-preview3-006845/Microsoft.Common.CurrentVersion.targets(1987,5): warning MSB3277: Found conflicts between different versions of the same dependent assembly that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed. [/home/vtsv/source-build/src/reference-assemblies/System.Security.Cryptography.Cng/netstandard1.6/4.2.0.0/System.Security.Cryptography.Cng.csproj] [/home/vtsv/source-build/build.proj]
  /home/vtsv/source-build/Tools/dotnetcli/sdk/2.0.0-preview3-006845/Microsoft.Common.CurrentVersion.targets(1987,5): warning MSB3277: Found conflicts between different versions of the same dependent assembly that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed. [/home/vtsv/source-build/src/reference-assemblies/System.Security.Cryptography.OpenSsl/netstandard1.6/4.0.0.0/System.Security.Cryptography.OpenSsl.csproj] [/home/vtsv/source-build/build.proj]
  /home/vtsv/source-build/Tools/dotnetcli/sdk/2.0.0-preview3-006845/Microsoft.Common.CurrentVersion.targets(1987,5): warning MSB3277: Found conflicts between different versions of the same dependent assembly that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed. [/home/vtsv/source-build/src/reference-assemblies/System.Security.Cryptography.Cng/netstandard1.6/4.2.0.0/System.Security.Cryptography.Cng.csproj] [/home/vtsv/source-build/build.proj]
  /home/vtsv/source-build/Tools/dotnetcli/sdk/2.0.0-preview3-006845/Microsoft.Common.CurrentVersion.targets(1987,5): warning MSB3277: Found conflicts between different versions of the same dependent assembly that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed. [/home/vtsv/source-build/src/reference-assemblies/System.Security.Cryptography.OpenSsl/netstandard1.6/4.0.0.0/System.Security.Cryptography.OpenSsl.csproj] [/home/vtsv/source-build/build.proj]


"/home/vtsv/source-build/build.proj" (default target) (1) ->
"/home/vtsv/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/vtsv/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:2) ->
(ApplyPatches target) ->
  /home/vtsv/source-build/targets/repository.proj(49,5): error MSB3073: The command "patch -p1 --ignore-whitespace -i /home/vtsv/source-build/patches/standard/0001-Respect-NUGET_PACKAGES-if-it-is-set.patch" exited with code 1.


"/home/vtsv/source-build/build.proj" (default target) (1) ->
"/home/vtsv/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/vtsv/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:3) ->
  /home/vtsv/source-build/targets/repository.proj(49,5): error MSB3073: The command "patch -p1 --ignore-whitespace -i /home/vtsv/source-build/patches/coreclr/0001-Respect-NUGET_PACKAGES-if-it-is-set.patch" exited with code 1.


"/home/vtsv/source-build/build.proj" (default target) (1) ->
"/home/vtsv/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/vtsv/source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:7) ->
  /home/vtsv/source-build/targets/repository.proj(49,5): error MSB3073: The command "patch -p1 --ignore-whitespace -i /home/vtsv/source-build/patches/newtonsoft-json/0001-Run-dotnet-migrate-to-generate-Newtonsoft.Json.Dotne.patch" exited with code 1.

    4 Warning(s)
    3 Error(s)

Core-Setup sometimes fails to build

I encountered an error building core-setup from a clean source-build repo. The error was exactly the error described in https://github.com/dotnet/core-setup/issues/1461.

Initially I tried to delete the obvious things like "artifacts", ".dotnet_stage0", all "project.lock.json" files, all "bin" and "obj" directories. My build still failed. After finding the linked core-setup issue, I ran "git clean -xdf" and my build passed.

"git clean -xdf" may be a workable solution when building the core-setup repo, but in a source-build world where git is not available, we will need some other solution. Additionally, a flakey build late in the build phases is going to be extremely troublesome.

/cc @ellismg

Align phases and dependencies props files

There are some redundant concepts between phases.props and dependencies.props. we should look into aligning these two files.

#54

chcosta commented 2 hours ago
Thanks for the pointer @dagood . Do you think there's any way we can connect https://github.com/dotnet/source-build/blob/master/phases.props and dependencies.props and remove the need to make updates in two locations? If aligning them means we lose flexibility with respect to the versioning stuff, then we probably don't want to do it, but it would be great if this kind of repo related stuff was just captured in one location.

dagood commented 2 hours ago
I think that would be reasonable, and if a target is required to make that happen, it could be added as a dependency of everything in update-submodules.targets.

Repository is empty

I see from some other issues (and having received private tarballs) that at least a partial implementation of source-build already exists. It would be fantastic if it was all available here so anyone could check out the latest bits and build them.

And, in the spirit of open source, contribute improvements :)

Errors while building on 16.04 Xenial Xerus

This is what I get with a fresh clone (first time I attempt a build as well) with a call to ./build.sh

"/home/sergiusens/dotnet-source-build/build.proj" (default target) (1) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:3) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:5) ->
(Build target) -> 
  EXEC : error : Could not read profile: Unsupported format version [/home/sergiusens/dotnet-source-build/targets/repository.proj]
  /home/sergiusens/dotnet-source-build/targets/repository.proj(72,5): error MSB3073: The command "/home/sergiusens/dotnet-source-build/src/coreclr/build.sh x64 Release skiptests -PortableBuild=false " exited with code 1.


"/home/sergiusens/dotnet-source-build/build.proj" (default target) (1) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:15) ->
  /home/sergiusens/dotnet-source-build/targets/repository.proj(72,5): error MSB3073: The command "/home/sergiusens/dotnet-source-build/src/websdk/build-core.sh" exited with code 126.

    553 Warning(s)
    3 Error(s)

I was successful in the past in just building coreclr by calling ./build.sh on its own from the repo though.

linux numa support [2.0]

coreclr has now support for numa systems

This support is compiled in when the build detects the numa.h header.
When compiled with numa support, libnuma is a native dependency of the runtime.

Will the Microsoft .net core 2.0 (preview) Linux builds ship with numa support?

CC @ellismg @bleroy @omajid @aslicerh

Cant clone this repo

git clone --recursive [email protected]:dotnet/source-build.git fails with:

error: Server does not allow request for unadvertised object f1f3f0820a573b96b2faaf5b7e6be9a036e4c7aa
Fetched in submodule path 'src/nuget-client/submodules/FileSystem', but it did not contain f1f3f0820a573b96b2faaf5b7e6be9a036e4c7aa. Direct fetching of that commit failed.
Submodule path 'src/roslyn': checked out '0a5202322a8aa23eb2454ff5f05ecdb258d64956'
Submodule path 'src/sdk': checked out '275430b3ee34f7a62cce891e3cb5c70830432a75'
Submodule path 'src/standard': checked out '81ffa9f5381f510c2afa17a113e4c4d77144b84e'
Submodule path 'src/templating': checked out '9560b6305ce327d1f13b6705490be37a8450a9ac'
Submodule path 'src/vstest': checked out 'c1b5c363c643c689448825f1e4637697ac056b01'
Submodule path 'src/websdk': checked out 'ccfba21231ab03e9a8aa73ebfeb658c72648e104'
Submodule path 'src/xliff-tasks': checked out '27d43b762aa6dac3a0a6ba48fe55000942d75c1c'
Failed to recurse into submodule path 'src/nuget-client'

Build JSON.NET on Linux

We'll need to build the netstandard1.0 version of JSON.NET on Linux, since that's what will be consumed by CLI. We can ignore building all the other flavors for now.

The build is driven by powershell but for the component we want, dotnet should be sufficient.

We'll need to public sign as well, the key is not included by default.

Error cloning nuget-client

I got this error trying to clone source-build today:

$ git clone --recursive [email protected]:dotnet/source-build
Cloning into 'source-build'...
remote: Counting objects: 802, done.
remote: Compressing objects: 100% (75/75), done.
remote: Total 802 (delta 42), reused 0 (delta 0), pack-reused 725
Receiving objects: 100% (802/802), 195.34 KiB | 0 bytes/s, done.
Resolving deltas: 100% (474/474), done.
Checking connectivity... done.
Submodule 'src/cli' (https://www.github.com/dotnet/cli) registered for path 'src/cli'
Submodule 'src/core-setup' (https://github.com/dotnet/core-setup) registered for path 'src/core-setup'
Submodule 'src/coreclr' (https://github.com/dotnet/coreclr) registered for path 'src/coreclr'
Submodule 'src/corefx' (https://github.com/dotnet/corefx) registered for path 'src/corefx'
Submodule 'src/libuv' (https://github.com/ellismg/libuv) registered for path 'src/libuv'
Submodule 'src/netcorecli-fsc' (https://www.github.com/dotnet/netcorecli-fsc) registered for path 'src/netcorecli-fsc'
Submodule 'src/newtonsoft-json' (https://github.com/JamesNK/Newtonsoft.Json) registered for path 'src/newtonsoft-json'
Submodule 'src/nuget-client' (https://github.com/NuGet/NuGet.Client) registered for path 'src/nuget-client'
Submodule 'src/roslyn' (https://github.com/dotnet/roslyn) registered for path 'src/roslyn'
Submodule 'src/sdk' (https://github.com/karajas/sdk) registered for path 'src/sdk'
Submodule 'src/standard' (https://github.com/dotnet/standard) registered for path 'src/standard'
Submodule 'src/templating' (https://github.com/karajas/templating) registered for path 'src/templating'
Submodule 'src/vstest' (https://github.com/karajas/vstest) registered for path 'src/vstest'
Cloning into '/home/omajid/devel/source-build/src/cli'...
Cloning into '/home/omajid/devel/source-build/src/core-setup'...
Cloning into '/home/omajid/devel/source-build/src/coreclr'...
Cloning into '/home/omajid/devel/source-build/src/corefx'...
Cloning into '/home/omajid/devel/source-build/src/libuv'...
Cloning into '/home/omajid/devel/source-build/src/netcorecli-fsc'...
Cloning into '/home/omajid/devel/source-build/src/newtonsoft-json'...
Cloning into '/home/omajid/devel/source-build/src/nuget-client'...
Cloning into '/home/omajid/devel/source-build/src/roslyn'...
Cloning into '/home/omajid/devel/source-build/src/sdk'...
Cloning into '/home/omajid/devel/source-build/src/standard'...
Cloning into '/home/omajid/devel/source-build/src/templating'...
Cloning into '/home/omajid/devel/source-build/src/vstest'...
Submodule path 'src/cli': checked out '0f092f92c231852f2e03d45a074dfc44f228fe17'
Submodule path 'src/core-setup': checked out 'a4601383a08bd1056075206364d3d6073a2acf22'
Submodule path 'src/coreclr': checked out '97d60cb7ef503ae698b784804530dbe59fdb575b'
Submodule path 'src/corefx': checked out '03f14d7d9b7bc9f628716e38f13770c7f817056a'
Submodule path 'src/libuv': checked out 'a7dd81dbc591182304db72e3a08cf86671773542'
Submodule 'src/libuv' (https://github.com/libuv/libuv.git) registered for path 'src/libuv/src/libuv'
Cloning into '/home/omajid/devel/source-build/src/libuv/src/libuv'...
Submodule path 'src/libuv/src/libuv': checked out 'c8a373c729b4c9392e0e14fc53cd6b67b3051ab9'
Submodule path 'src/netcorecli-fsc': checked out '93bb3e0e7ca25c3564a4be43acade333b7a984a4'
Submodule path 'src/newtonsoft-json': checked out 'bed740f76ecb1224219b55c0d61e5b3122af6792'
error: no such remote ref 913c2b1c11b04716a10885e88d8a49d977c75f30
Fetched in submodule path 'src/nuget-client', but it did not contain 913c2b1c11b04716a10885e88d8a49d977c75f30. Direct fetching of that commit failed.

Any ideas? I can see the commit at https://github.com/NuGet/NuGet.Client/tree/913c2b1c11b04716a10885e88d8a49d977c75f30 but can't clone it.

Don't invoke MSBuild from within the CLI

Today, we invoke MSBuild.dll directly from within the CLI. We do this (vs dotnet msbuild) because dotnet msbuild pushes some additional stuff into your environment. This causes downstream issues in repositories that don't expect these things to be set to the values CLI uses, since they use different build toolsets.

@piotrpMSFT and @eerhardt have asked that we don't do this, as they want the freedom to move stuff within the CLI around and don't want to be a general purpose MSBuild distribution mechanism. Their concern is if folks see another "dotnet" project doing this they will think it's a good and right thing to do.

So instead, we need to start restoring and publish MSBuild, and then invoke that MSBuild.dll (we can do this after building our tasks but before jumping into build.proj.

Come up with a plan for building/packing net45x assemblies on Linux

Nuget.Client, JSON.NET and others want to build nupkg's that contain assets for netstandard and net45X.

Building net45X components on *NIX require targeting packs (which may come in from Mono?). We don't want to depend on mono so for now we don't build these assets and hack up downstream projects to not require them, but we should do this the right way.

Build Symreader on Linux

MSBuild based, so it should be pretty easy. We can reuse the toolset we use for Roslyn because these projects are very similar.

Will also need nupkgs produced as well. Today there's a special project from that but it uses nuget.exe, so that will need to change.

Build broken on Windows 10 platform

Build is broken on Windows platform.

After cloning repo for commit 807953d

running .\build.ps1 throws:

PowerShell version: 5.1.15063.502
Free physical memory: 11573 MB out of 16300 MB

Loading personal and system profiles took 1746ms.
PS C:\> E:
PS E:\> cd src\ms\dotnet\src-bld
PS E:\src\ms\dotnet\src-bld> .\build.ps1
Get-Content : Cannot find path 'E:\src\ms\dotnet\src-bld\.cliversion' because it does not exist.
At E:\src\ms\dotnet\src-bld\build.ps1:5 char:15
+ $SdkVersion = Get-Content (Join-Path $ScriptRoot ".cliversion")
+               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (E:\src\ms\dotnet\src-bld\.cliversion:String) [Get-Content], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetContentCommand

& : The term 'E:\src\ms\dotnet\src-bld\bootstrap\bootstrap.ps1' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path
was included, verify that the path is correct and try again.
At E:\src\ms\dotnet\src-bld\build.ps1:7 char:3
+ & "$ScriptRoot\bootstrap\bootstrap.ps1"
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (E:\src\ms\dotne...p\bootstrap.ps1:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

& : The term 'E:\src\ms\dotnet\src-bld\Tools\dotnetcli\dotnet' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path
was included, verify that the path is correct and try again.
At E:\src\ms\dotnet\src-bld\build.ps1:8 char:3
+ & "$ScriptRoot\Tools\dotnetcli\dotnet" restore "$ScriptRoot\tasks\Mic ...
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (E:\src\ms\dotne...otnetcli\dotnet:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

& : The term 'E:\src\ms\dotnet\src-bld\Tools\dotnetcli\dotnet' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path
was included, verify that the path is correct and try again.
At E:\src\ms\dotnet\src-bld\build.ps1:9 char:3
+ & "$ScriptRoot\Tools\dotnetcli\dotnet" build "$ScriptRoot\tasks\Micro ...
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (E:\src\ms\dotne...otnetcli\dotnet:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

& : The term 'E:\src\ms\dotnet\src-bld\Tools\dotnetcli\dotnet' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path
was included, verify that the path is correct and try again.
At E:\src\ms\dotnet\src-bld\build.ps1:10 char:3
+ & "$ScriptRoot\Tools\dotnetcli\dotnet" "$ScriptRoot\Tools\dotnetcli\s ...
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (E:\src\ms\dotne...otnetcli\dotnet:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

& : The term 'E:\src\ms\dotnet\src-bld\Tools\dotnetcli\dotnet' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path
was included, verify that the path is correct and try again.
At E:\src\ms\dotnet\src-bld\build.ps1:11 char:3
+ & "$ScriptRoot\Tools\dotnetcli\dotnet" "$ScriptRoot\Tools\dotnetcli\s ...
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (E:\src\ms\dotne...otnetcli\dotnet:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Attempt to initialize Tools via init-tools.cmd fails with the following error:

Running init-tools 
Installing 'https://dotnetcli.azureedge.net/dotnet/Sdk/2.0.0-preview3-006845/dotnet-dev-win-x64.2.0.0-preview3-006845.zip' to 'e:\src\ms\dotnet\src-bld\Tools\dotnetcli\dotnet-dev-win-x64.2.0.0-preview3-006845.zip' 
Exception calling "DownloadFile" with "2" argument(s): "The remote server returned an error: (404) Not Found."
At line:1 char:48
+ ...  do { try { (New-Object Net.WebClient).DownloadFile('https://dotnetcl ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : WebException

Make source-build sustainable

  • We can produce community source builds with a clean checklist of steps to 'sync' with repos.

  • CI passes consistently

  • A branching strategy is in place

  • Community can engage productively - can sync, build source-build

Enable iterating on an individual repo build

Today we traverse over the repo's in the root build.proj which is fine but we don't have any way to iterator or rebuild just one repo if something goes wrong. We need to provide a easy way to build/rebuild just a given repo without needing to build the world, because we don't build incrementally very well.

net4x assets are not built and they are needed by VS code

There are net4x-specific assets in the closed build that are not produced in the source build. For standard dotnet usage, these assets are never used, but they are needed if msbuild is running on Mono or .NET Framework and pointed at the targets in the .NET Core SDK folder. VS Code's C# support does builds using Mono this way and things are broken without the net46 assets. VS for Mac has the same requirement. (Edit: So does Visual Studio.)

See dotnet/vscode-csharp#1744

We need to teach source build how to build these assets. In the closed build of dotnet/sdk, the assets are built on Windows and packaged on all platforms.

Major complication: targeting net4x using .NET Core msbuild on non-Windows is not yet officially supported. We can workaround it in various ways, or better yet implement it as a true supported feature, but it will require access to full framework reference assemblies during the build. And that probably means we need to build full framework reference assemblies from source...

Another path forward would be to make all of the task dlls (nuget too, not just dotnet/sdk) target .NETStandard and stop multi-targeting them to net4x and netcoreapp. However, it is currently next to impossible to make .NETStandard msbuild tasks. See dotnet/msbuild#1309 for some of the issues with it.

cc @DustinCampbell @Petermarcu @shawnro @dleeapho @karajas @leecow

source-build first experience

I tried the source-build repo today and want to share my experience.

From looking at the build.sh script the build is in two fases:

  • bootstrap
  • build

The bootstrap is where the sdks needed for the build are gathered or put together. It looks like both 1.1 and 2.0 are needed. The options to get these runtimes are:

  • no arguments: they will be downloaded (your system needs to be capable of running .net core already)
  • point to a --netcore_sdk_11_path and --netcore_sdk_20_path which will be patched with native binaries built during the bootstrap (cfr rover tool)
  • point to an existing runtime in the filesystem via --msbuild_cli

There are two additional parameters: shared_framework_path, managed_coreclr_path. I don't understand what they are for.

I am going for the second option. I downloaded the fedora 23 1.1 sdk from the official download page and the linux-x64 2.0 daily from the cli repo readme.

To make sure the source-build doesn't go off and download some cli for my system, I change /etc/os-release:

ID=banana

Time to kick of the build:

./build.sh -s11 ~/Downloads/net_11 -s20 ~/Downloads/net_20 -f ~/Downloads/net_20

The shared_framework_path (aka -f) is a required parameter, so I put it to the net_20 folder too. Not sure if that's ok.

The step "Copy native CoreFx binaries into shared framework path" fails. Seems like someone changed the casing on the Native folder.

-  COREFXBINDIR=$COREFXREPOROOT/bin/Linux.x64.Release/Native
+  COREFXBINDIR=$COREFXREPOROOT/bin/Linux.x64.Release/native

building some more...

Build fails with:

DOTNET_TOOL_DIR=/home/tmds/Downloads/net_11.patch __TOOLRUNTIME_DIR=/home/tmds/repos/source-build/tools_for_submodules __PUBLISH_RID=ubuntu.14.04-x64 /home/tmds/repos/source-build/bootstrap/init-tools.sh
cat: /home/tmds/repos/source-build/bootstrap/DotnetCLIVersion.txt: No such file or directory
Restoring BuildTools version 1.0.27-prerelease-01209-01...
Failed to initialize CoreCLR, HRESULT: 0x80131500
ERROR: Could not restore build tools correctly. See '/home/tmds/repos/source-build/bootstrap/init-tools.log' for more details.1
Initializing BuildTools...
/home/tmds/repos/source-build/bootstrap/init-tools.sh: line 148: /home/tmds/repos/source-build/bootstrap/packages/Microsoft.DotNet.BuildTools/1.0.27-prerelease-01209-01/lib/init-tools.sh: No such file or directory
ERROR: An error occured when trying to initialize the tools. Please check '/home/tmds/repos/source-build/bootstrap/init-tools.log' for more details.1

CC @ellismg @omajid @nmilosev

Changing tasks/Microsoft.DotNet.SourceBuild.Tasks/vendor/ breaks roslyn

I was trying to understand how new RIDs might be supported by source-build. So I made this change:

diff --git a/tasks/Microsoft.DotNet.SourceBuild.Tasks/vendor/Microsoft.DotNet.PlatformAbstractions/Native/PlatformApis.cs b/tasks/Microsoft.DotNet.SourceBuild.Tasks/vendor/Microsoft.DotNet.PlatformAbstractions/Native/PlatformApis.cs
index 2786534..36ceaac 100644
--- a/tasks/Microsoft.DotNet.SourceBuild.Tasks/vendor/Microsoft.DotNet.PlatformAbstractions/Native/PlatformApis.cs
+++ b/tasks/Microsoft.DotNet.SourceBuild.Tasks/vendor/Microsoft.DotNet.PlatformAbstractions/Native/PlatformApis.cs
@@ -112,10 +112,27 @@ namespace Microsoft.DotNet.PlatformAbstractions.Native
                         result.VersionId = line.Substring(11).Trim('"', '\'');
                     }
                 }
+                if (string.Equals(result.Id, "rhel", StringComparison.Ordinal))
+                {
+                   result = NormalizeRhelDistroInfo(result);
+                }
                 return result;
             }
             return null;
         }
+ 
+        // RHEL minor versions are backwards compatible. Normalize all 7.x
+        // versions to '7'
+        private static DistroInfo NormalizeRhelDistroInfo(DistroInfo distro)
+        {
+            var result = new DistroInfo(); 
+            result.Id = distro.Id;
+            if (distro.VersionId.StartsWith("7."))
+            {
+                result.VersionId = "7";
+            }
+            return result;
+        }
 
         // I could probably have just done one method signature and put the #if inside the body but the implementations
         // are just completely different so I wanted to make that clear by putting the whole thing inside the #if.

Then building using ./build.sh on RHEL 7.2 (actually 7.3 with /etc/os-release claiming 7.2), causes the build to work until roslyn. Then it hits roslyn's Makefile:

$(BINARIES_PATH)/dotnet-cli/dotnet publish -o $(TOOLSET_PATH)

Which fails without a --runtime flag.

"Broken pipe" after executing first build command

Using the Fedora dockerfile on OSX 10.11.6, I get this error which fails the repository build:

Project "/sb/build.proj" (1) is building "/sb/targets/repository.proj" (3) on node 1 (Build target(s)).
Build:
  Executing /sb/src/standard/> /sb/src/standard/build.sh -Release -- /p:BuildVersionFile=/sb/bin/obj/BuildVersion.props
  /sb/src/standard/build.sh -Release -- /p:BuildVersionFile=/sb/bin/obj/BuildVersion.props
cat : write error : Broken pipe [/sb/targets/repository.proj]

I've also hit it in different environments. I haven't run enough builds to see if it's transient or not.

Has anyone else seen this? @ellismg

Don't generate the rootfs every time when cross-compiling

Steps to reproduce:

  1. sudo ./build.sh /p:Platform=arm
  2. let it build all the way
  3. sudo ./build.sh /p:Platform=arm again

Expected result:

  • Not much should happen, since it just got build and nothing changed

Observed result:

  • The rootfs is regenerated right off the bat, which takes a really long time and downloads an entire OS

We should add a condition to only generate the rootfs if it doesn't already exist.

Update source-build for 1.1.x release

@crummel commented on Tue Aug 22 2017

  • Go back to previously shipped tarball commit and make sure everything still builds (no submodules disappeared, etc)
  • Update submodules to new 1.1 branches for CoreCLR, CoreFX, core-setup, CLI
  • Update submodules that these depend on
  • Update patches as needed
  • Do we need to include 1.0.x builds in this? (#1560)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.