[perf] Dotnet restore unable to find 8.0.0-rc.* packages about performance HOT 23 CLOSED

We have been seeing this in wasm too (dotnet/runtime#81222). It's because of

Somehow the version resolved for that wildcard is just slightly newer than the one available in the feed (dotnet8 in this case). (edit: It shouldn't happen often.) Maybe we need to open a nuget issue.

from performance.

My best guess is builds happening in the middle of a publish? Is it possible that NuGet finds a newer version for one package, but that package's dependencies haven't yet been published.

from performance.

Those look like transitive dependencies so I guess it could happen during publish of a runtime build, I wonder if we could list them explicitly?

from performance.

I chatted with @DrewScoggins about this earlier.

Publish cannot be transactional, either on nuget.org or AzDO. It's an open ask for Nuget.org (mainly about staging packages for a release). In theory, you could parse the dependency graph and ensure that you publish all dependencies early enough, but it still wouldn't guarantee a coherent set of packages if you have a dependency on two leaves.

This is a pretty niche use case overall and it doesn't seem like a good idea to optimize for it, at least for daily builds. Either you alter the perf repo workflow to pick a coherent set based on live Maestro data, or you pin on a coherent version.

from performance.

@zivkan Agreed that you can parse the dependency graph, but it still won't guarantee a coherent set of dependencies. You can still end up with dependencies from two builds.

IMO the use of * versioning on live builds is not really a great practice.

from performance.

Thanks. I noticed that it has started occurring a couple of days ago. We didn't get perf coverage for the last week.

@radical Is the https://github.com/dotnet/arcade suitable place for submitting the issue?

from performance.

Weird, this will coincide with the host version, not whatever version it is picking up (sdk?)

from performance.

It shouldn't happen often.

Details: earlier we were seeing this only in blazor scenarios, which is where I had added a similar version string.
But now, we are seeing this everywhere because #3120 added <ExtensionsVersion>8.0.*-*</ExtensionsVersion> for MicroBenchmarks.csproj.

And with that change all the wasm perf runs are also broken.
cc @cincuranet

Edit: sometimes the wasm perf runs do pass, so it isn't deterministic.

from performance.

dotnet/runtime#81222 is the issue I mentioned with blazor scenarios.

from performance.

@dotnet/nuget-team is this a regression in nuget.client?

from performance.

Could somebody from @dotnet/nuget-team please take a look at this? The issue seems to be around at least since ~Jan26 (dotnet/runtime#81222). Using a version string with wildcards causes the restore to race sometimes causing:

src/benchmarks/micro/MicroBenchmarks.csproj :   
error NU1102: Unable to find package System.Text.Json with version (>= 8.0.0-rc.1.23375.7)
error NU1102:   - Found 1290 version(s) in dotnet8 [ Nearest version: 8.0.0-rc.1.23375.4 ]

The version it is complaining about 8.0.0-rc.1.23375.7 is slightly newer than the latest version (8.0.0-rc.1.23375.4) it reports from the dotnet8 feed.

from performance.

@ericstj any idea what is going wrong here?

from performance.

Hi folks. If you think there's a NuGet issue, please file an issue here: https://github.com/NuGet/Home/issues/new/choose. Gathering a binlog would probably be the best first step along with the feeds defined in NuGet.config.

from performance.

We don't collect binlog for the restore step. I will open a PR to obtain that.

from performance.

Update: #3167 - to get the binlog. Once, I have a binlog from a failing run, I'll open an issue for nuget.

from performance.

Listing them all out could result in a downgrade instead. My recollection of * versions are that they have these rough edges. Can we make sure publish is transactional? @mmitche

from performance.

Listing them all out could result in a downgrade instead. My recollection of * versions are that they have these rough edges. Can we make sure publish is transactional? @mmitche

There is an open discussion about that somewhere and so far the answer has been no

from performance.

Those look like transitive dependencies so I guess it could happen during publish of a runtime build, I wonder if we could list them explicitly?

#3168

from performance.

dotnet/sdk#23820

from performance.

Having a look over this issue, it looks to me the current belief is that a package that was pushed has a dependency on a package version not yet pushed.

If it's useful, I can help someone from dnceng use the NuGet Client SDK to read nupkgs that are about to be pushed and find their dependencies. This way we can build a graph of package dependencies, and determine an order so that each package will only be pushed when its dependency is also pushed (is this a feature that nuget should have built-in, when pushing a bunch of packages?). However, this will obviously only work if all the packages are pushed from a single CI task.

I did something similar, but different, recently in NuGet.Client, to ensure the VS Extensibility packages we publish only use versions of the VS SDK that exists on nuget.org: https://github.com/NuGet/NuGet.Client/blob/dev/tools-local/ensure-nupkg-dependencies-on-source/Program.cs

from performance.

This is a pretty niche use case overall and it doesn't seem like a good idea to optimize for it, at least for daily builds. Either you alter the perf repo workflow to pick a coherent set based on live Maestro data, or you pin on a coherent version.

I'm not sure I agree that is a niche case that the tooling for push/publish does not support a mechanism that allows restore for the same tool to succeed.

from performance.

@zivkan Agreed that you can parse the dependency graph, but it still won't guarantee a coherent set of dependencies. You can still end up with dependencies from two builds.

IMO the use of * versioning on live builds is not really a great practice.

agreed, and given how the current code in performance works I'm even less sure that I was before about what code is being used where when analyzing the benchmark reports. If dotnet/performance is testing a runtime commit, the code used to compile the benchmark should be the system/extension assemblies from the commit not the latest ones in the feed. Is that the case?

from performance.

This is a pretty niche use case overall and it doesn't seem like a good idea to optimize for it, at least for daily builds. Either you alter the perf repo workflow to pick a coherent set based on live Maestro data, or you pin on a coherent version.

I'm not sure I agree that is a niche case that the tooling for push/publish does not support a mechanism that allows restore for the same tool to succeed.

a real solution for this looks like a nuget staging area, because there's no way to ensure that a coherent set gets pushed simultaneously from the client's standpoint. This has been discussed, but there are no concrete plans.

from performance.