[Bug]: Exception in locate_tools with specific environments about edk2-pytool-library HOT 8 CLOSED

@jorticus-msft I will look into this. Thank you!

from edk2-pytool-library.

@jorticus-msft I am not able to replicate the issue via your example. As shown in your example, you are using the Helper function self.Helper.PackageFmpImageAuth(), which eventually calls FindToolInWinSdk("signtool.exe")which is whereQueryVCVariables()is called. As you suggested, runningFindToolInWinSdk()` many times will eventually cause the error you linked.

As a test, I ran FindToolInWinSdk("singtool.exe") 1000 times in a loop and the number lines returned from vcvarsall.bat.
I verified the returned rows and total length of vcvarsall.bat remained unchanged and that os.environ entries and total length remained unchanged. Due to this, I believe environment pollution is coming from somewhere else (unless I am misunderstanding your issue).

Thanks!
Joey

from edk2-pytool-library.

@Javagedes You're right, it's not QueryVcVariables that is causing the issue. The environ grows between calls to this function, and by the end of the build the environ is so large that QueryVcVariables fails to run.

The main culprit is %PATH% which starts at 3113 characters and grows to 6799 characters. EDK2 tooling adds ~40 paths to the system PATH. The limit for Windows is apparently 8191 chars (when executing .bat files [source], which doesn't quite explain this, but if we assume Python uses wide chars and the windows limitation is actually 8191 bytes, then it's expected that I would run into this issue here.

I am able to resolve the issue by reducing my PATH length. My fault for having such a large PATH I guess... 🙂 I'll leave it up to you if you think this is something that should be fixed in the tooling or not. Maybe a warning would be useful?

from edk2-pytool-library.

Hello, yes on the edk2-pytool-extensions side (stuart commands), we manipulate the os.environ through our environment descriptor files and external dependencies; 40 paths seem a bit much and it may be necessary to evaluate scope usage to reduce the amount of in-scope, but unused, descriptors / external dependencies as each will edit the os.environ in some way.

I should be able to catch the exception when the exception message is The input line is too long. and throw a better worded error about the PATH length. I'll work on that and tag you in the PR if I can get it working!

from edk2-pytool-library.

@jorticus-msft The reason the limit you mentioned does not line up perfectly is because this 8192 limitation is for all arguments passed, which includes all environment variables, not just the PATH. Therefore once the total length of all of the environment variables reach 8192, we throw the exception you mentioned above.

There is actually a second layer to this issue that I'm working through now. That is the fact that when an individual environment variable grows over 8191, that variable is ignored. Because of this, if the PATH grows over 8191, PATH is ignored. When this happens, we no longer throw the above error because PATH is not passed to the .bat command, but we fail to find what we are looking for (depending on if it's found in the PATH variable). I'm working to handle warning for both.

from edk2-pytool-library.

PR to example on the logging message: #320

from edk2-pytool-library.

Thank you! The warning works for me:

ERROR - .bat cmd can not handle args greater than 8191 chars; your total ENV var len exceeds 8191. Reduce the total length of your ENV variables to resolve this (Typically your PATH is too long).
CRITICAL - Build Process Exception
ERROR - Traceback (most recent call last):
...snip stacktrace...
Exception: .bat cmd can not handle args greater than 8191 chars; your total ENV var len exceeds 8191. Reduce the total length of your ENV variables to resolve this (Typically your PATH is too long).

from edk2-pytool-library.

Great thank you for verifying. I will get this in and it will be available in the .15.0 release!

from edk2-pytool-library.