Improve startup perf about cswin32 HOT 15 CLOSED

Consider MRT as a file format for the docs file as it is binary and includes an index as well.

from cswin32.

Hello,
as a Vanara user I decided to give a try after reading that this will be the new recommended approach for pinvoking.
On an new .Net 5 class library, compiling with a single Kernel32.* has taken a couple of minutes. Once finished, VS (16.8.4) became completely unresponsive. An extension (Codist) that was working fine, suddenly started to freeze VS. Thinking that it was some sort of incompatibility, I uninstalled it, but VS remained unresponsive as was before.
After enabling CsWin32, the VS memory usage suddenly increased from 500 Mb to 1.5 Gb and more. Intellisense is very slow.
Is this library designed to run on a very powerful system?
I want to think that this is happening because I need to configure something that I must have missed out because as is, it's not just poor performance, it makes VS unusable.
If I am not wrong, this seems to be a generator that ends up with the same precompiled dotnet/pinvoke assemblies.
What are the advantages over Vanara (all handle types are safe and include xml doc) or dotnet/pinvoke except saving a couple of Mb?
I must miss for sure something.

var op = OpenProcess(uint dwDesiredAccess, .., ..)
I was hoping to find ProcessAccessFlags at least in some overload. I tried to look on Microsoft.Windows.Sdk.Constants but such namespace does not exists in Microsoft.Windows.Sdk.
These was the very few things I was able to explore with R#. After disabling R#, the native intellisense was even worse. It does not popup except in rarely times (and when this happens after several minutes of waiting).

using Microsoft.Windows.Sdk; after a while and for unknown reason becomes unknown to VS.
As soon as I switch in a project without CsWin32 everything comes back to normal, included the memory usage.
I downloaded the Nuget package advertised in the home package.
If this is how it works, then I am afraid I'll just have to go away, but if it isn't and is supposed to not give me all of these problems, then I must understand if there is at least one reason for which I should leave Vanara and start with CsWin32 because actually I can't find none.

from cswin32.

Thanks for the feedback, @netcorefan1.

compiling with a single Kernel32.* has taken a couple of minutes

That certainly performs quite well on my system, which is running VS 2019 Update 10 (it worked on Update 9 too). It is also a pretty powerful system. That said, I would never expect the length of time or the unresponsiveness you are reporting and suspect you may have an extension that is adversely affecting you. Maybe try disabling all 3rd party extensions, restarting VS, and trying again. Those extensions that you cannot simply disable using the Extension Manager (e.g. Resharper?) you may need to uninstall for this experiment.
Finally, we don't expect a module.* syntax will be appropriate for most folks, who undoubtedly only require a small subset of the APIs from that module. So before you try any of this, see how Kernel32.CreateFile or whatever specific APIs you need works for you.

If I am not wrong, this seems to be a generator that ends up with the same precompiled dotnet/pinvoke assemblies.

It is not the same. We have plans to replace the code in dotnet/pinvoke with the code projected by CsWin32 though.

What are the advantages over Vanara (all handle types are safe and include xml doc) or dotnet/pinvoke except saving a couple of Mb?

Maybe not much. If you're ok with the payload size and the extra dependencies, that's fine. You're welcome to continue using those if they suite you better.
We see this projection solution as more scalable and it has the potential to shape itself to meet the app's needs better over time.
dotnet/pinvoke include xml docs for most APIs as well, as does this projection.

I was hoping to find ProcessAccessFlags at least in some overload.

The metadata that this projection is based on is still very new and we haven't defined all the enums yet. We expect that to improve soon.

I tried to look on Microsoft.Windows.Sdk.Constants but such namespace does not exists in Microsoft.Windows.Sdk.

Constants is not a namespace. It is a static class in the Microsoft.Windows.Sdk namespace.

After disabling R#, the native intellisense was even worse. It does not popup except in rarely times (and when this happens after several minutes of waiting).

Be aware that R# may disable many built-in C# features in order to replace them. The last I heard, removing R# does not re-enable them, which may make Microsoft's own C# lang svc look broken. You should re-enable those features by hand.

from cswin32.

Sorry for my delay and many thanks for your suggestions. Within a month I will have to format and it will be the perfect chance to retry again on a freshly installed VS on a fresh OS. I am sure there is something wrong on my side and I will be able to fix it. If not I will let you know.

compiling with a single Kernel32.* has taken a couple of minutes

Even after I will got it to work, I don't think that the compile time will change, but I don't care because, if remember well it's one time only compilation which need to be recompiled only if we add new entries. However, if you could find a way to optimize the subsequent compilations would be really nice.

Maybe not much. If you're ok with the payload size and the extra dependencies, that's fine. You're welcome to continue using > those if they suite you better.
We see this projection solution as more scalable and it has the potential to shape itself to meet the app's needs better over time.
dotnet/pinvoke include xml docs for most APIs as well, as does this projection.

It's very interesting the ability to not depend on third party libraries and ability to compile only what we really need instead of including several assemblies and even if they are separated, it's still a great amount of stuff (it take several minutes to open the Vanara pinvoke definitions).
If I understood well, CsWin32 will allow us to save on memory consumption, I am right? This for sure will be a great benefit along with the possibility to only include what is used.

I tried to look on Microsoft.Windows.Sdk.Constants but such namespace does not exists in Microsoft.Windows.Sdk.

Constants is not a namespace. It is a static class in the Microsoft.Windows.Sdk namespace.

Yes, since Constants is a static class it should appear when imported through using static Microsoft.Windows.Sdk.Constants;
May be it was my VS completely dead if I have not been able to import it.

With Vanara I like to have everything available in Intellisense because I can find what I need for by searching there. Would be possible to do the same with CsWin32 ? The full refactoring is something that I always do when a project is finished and means recompiling with all unused imports removed. Would be possible with CsWin32?

I think it's time for me to take in serious consideration the switch. May be at beginning I felt a bit discouraged, but once I will fix the issues on a fresh OS, I feel that I will be satisfied from the new changes!
Thanks for your work

from cswin32.

but I don't care because, if remember well it's one time only compilation which need to be recompiled only if we add new entries.

I don't think that is correct. The compiler may both re-invoke the source generator and re-compile the output many times while you work with your code in the IDE. And AFAIK the command-line build does not cache the output of the source generator anywhere, so it gets regenerated on every compilation too.

If I understood well, CsWin32 will allow us to save on memory consumption, I am right?

If you only have it generate the APIs that you require, then yes that will make for overall fewer or smaller assemblies to be loaded in memory, assuming you don't just generate Kernel32.* into your assembly with no need for all those APIs.

With Vanara I like to have everything available in Intellisense because I can find what I need for by searching there. Would be possible to do the same with CsWin32 ?

Yes, that already works.

from cswin32.

Having used a variety of different methods of generating C# syntax in various tools, I am a fan of building strings directly (e.g. IndentedTextWriter) rather than modeling syntax. The generation code becomes twenty times more maintainable. I'm mentioning this because it seems like it could benefit performance. A member of the Roslyn team has made the same case, build strings not syntax for source generators.

from cswin32.

I appreciate the tip, but I don't think that's going to happen. There are too many conditions and permutations on the emitted code to ever want to deal with strings, IMO. Having an OM allows not only for "add modifier" operations that doesn't require me to parse my own string in order to find where to insert the modifier, but it leaves open some very interesting incremental generation opportunities going forward.
But we'll see... I'm not morally opposed to the idea. It just turns out that emitting the syntax is not where the perf goes. It's YAML parsing and the C# syntax rewriting, which #303 substantially improved and ultimately we can remove completely if we need to improve perf further.
At this point, the only remaining perf problem I imagine actually impacts customers is the YAML parsing which on cold startup takes 5-6 seconds on my computer, which I hope to tackle this week.

from cswin32.

Sounds good. I do want to address this point:

Having an OM allows not only for "add modifier" operations that doesn't require me to parse my own string in order to find where to insert the modifier

I don't believe this benefit is unique to having an object model. When building a string, I do operate in a forward-only manner in a single pass. This organizes the building logic in a specific way that never involves parsing my own string. I've found the organization which this forces to be simple and maintainable.

from cswin32.

@AArnott what prevents the generator from actually using a zip file with docs stored in a xml file split in folders named after the win32 dll they are from and the file names being the name of each structure/enum/function they are for then use an json file (index.json) to let the generator know where to find said docs to use and basically it would take the contents of said xml file and spit it directly into the compilation (the only parts that it would need to add to that text per line would be the correct indenting and the /// part then)?

This is because System.IO.Compression zip apis are built into the BCL and the generator already depends on System.Text.Json and it would just need to add a dependency on the XML stuff which is also part of the BCL (and netstandard2.0) and therefore should be considered safe territory to use.

from cswin32.

The docs will eventually be produced in the win32metadata repo, represented in a language-agnostic way so API docs can be projected for any language that supports such a thing.
One file per API would probably cause too much I/O and be a perf drain. A simple binary file with an index at the top, as proposed near the top of this issue, I expect would provide the greatest performance.

from cswin32.

Another option is to embed the docs into each winmd file somehow generated by the metadata.

from cswin32.

We embed the docs in the CsWin32 projection today. But we're going to move to another model where they ship as a separate package and organized so that eventually we can even offer API docs localized to the repo's preferred (human) language.
This precludes the possibility of embedding in the metadata, which is transcends human languages.

from cswin32.

MRT appears to only work on Windows, and only on .NET 5.0+. That makes .pri an unacceptable format since we need to be able to parse it on any OS.

from cswin32.

MRT appears to only work on Windows, and only on .NET 5.0+. That makes .pri an unacceptable format since we need to be able to parse it on any OS.

aren’t pri files just xml files? if so What stops you from falling back to XDocument to parse it properly since it cannot be used in netstandard2.0 and on any OS?

from cswin32.

No. .pri is a binary file format.

from cswin32.