Giter Club home page Giter Club logo

edk2-pytool-library's Introduction

Tianocore Edk2 PyTool Library (edk2toollib)

pypi codecov ci docs

This is a Tianocore maintained project consisting of a python library supporting UEFI firmware development. This package's intent is to provide an easy way to organize and share python code to facilitate reuse across environments, tools, and scripts. Inclusion of this package and dependency management is best managed using Pip/Pypi.

This is a supplemental package and is not required to be used for edk2 builds.

Current Status

codecov ci

The code coverage and CI badges represent unit test status and the code coverage of those unit tests. We require 100% unit test success (Hence the pass / fail) and that code coverage percentage does not lower.

Supported Versions

Host Type Toolchain Status
Windows-Latest Python 3.10 ci
Windows-Latest Python 3.11 ci
Windows-Latest Python 3.12 ci
Ubuntu-Latest Python 3.10 ci
Ubuntu-Latest Python 3.11 ci
Ubuntu-Latest Python 3.12 ci
MacOS-Latest Python 3.10 coming_soon
MacOS-Latest Python 3.11 coming_soon
MacOS-Latest Python 3.12 coming_soon

Current Release

pypi

All release information is now tracked with Github tags, releases and milestones.

Content

The package contains classes and modules that can be used as the building blocks of tools that are relevant to UEFI firmware developers. These modules should attempt to provide generic support and avoid tightly coupling with specific use cases. It is expected these modules do not provide direct interaction with the user (through command line interfaces) but instead are intended to be wrapped in other scripts/tools which contains the specific usage and interface.

Examples:

  • File parsers for edk2 specific file types. These parse the file and provide an object for interacting with the content.
  • UEFI specific services for encoding/decoding binary structures.
  • UEFI defined values and interfaces for usage in python
  • Python wrappers for other system cli tools ( signtool, catalog file generation, inf file generation, etc)
  • Python utilities to provide consistent logging, command invocation, path resolution, etc

License

All content in this repository is licensed under BSD-2-Clause Plus Patent License.

PyPI - License

Usage

NOTE: It is strongly recommended that you use python virtual environments. Virtual environments avoid changing the global python workspace and causing conflicting dependencies. Virtual environments are lightweight and easy to use. Learn more

  • To install run pip install --upgrade edk2-pytool-library

  • To use in your python code

    from edk2toollib.<module> import <class>

History

This library and functionality was ported from Project Mu. For history and documentation prior to this see the original Project Mu project https://github.com/microsoft/mu_pip_python_library

Contribution Process

This project welcomes all types of contributions. For issues, bugs, and questions it is best to open a github issue.

Code Contributions

For code contributions this project leverages github pull requests. See github tutorials, help, and documentation for complete descriptions. For best success please follow the below process.

  1. Contributor opens an issue describing problem or new desired functionality
  2. Contributor forks repository in github
  3. Contributor creates branch for work in their fork
  4. Contributor makes code changes, writes relevant unit tests, authors documentation and release notes as necessary.
  5. Contributor runs tests locally
  6. Contributor submits PR to master branch of tianocore/edk2-pytool-library
    1. PR reviewers will provide feedback on change. If any modifications are required, contributor will make changes and push updates.
    2. PR automation will run and validate tests pass
    3. If all comments resolved, maintainers approved, and tests pass the PR will be squash merged and closed by the maintainers.

Maintainers

See the github team for more details.

Documentation

docs

Users and Consumers

Documentation for the most recent release of edk2-pytool-library is hosted on tianocore.org/edk2-pytool-library. Raw documentation is located in the docs/ folder and is split into two separate categories. The first is located at docs/user/ and is documentation and API references for those that are using this package in their own project. Users can generate a local copy of the documentation by executing the following command from the root of the project:

pip install --upgrade -e .[docs]
mkdocs serve

Contributors

Contributor documentation is located at docs/contributor/ and contains instructions for:

  • Setting up a development and testing environment
  • How edk2pytools is versioned and published
  • Contributing to the edk2-pytool-extensions repository

edk2-pytool-library's People

Contributors

apop5 avatar bkarstens avatar brandonaut avatar corthon avatar dependabot[bot] avatar edwinzmsft avatar flickdm avatar javagedes avatar jgarver avatar joschock avatar kenlautner avatar kuqin12 avatar makubacki avatar matthewfcarlson avatar mdkinney avatar mikeytdisco avatar mknutsen avatar out0xb2 avatar singweemsftgithub avatar spbrogan avatar srilathasridharan avatar zurcher avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

edk2-pytool-library's Issues

Benign bug

For regex groups, enclosed by parentheses, | is needed to separate expression 1 or expression 2 (ex. (a|b)). For bracketed expressions, this is not needed (ex. [ab] or [^ab]). These patterns therefore include | as a possible path separator. Seeing as this character will not show up in paths, this is benign.

seps_group = '[' + '|'.join(seps) + ']'
nonsep = r'[^{}]'.format('|'.join(seps))

Improve error message when Edk2ToolsBuild.py fails due to missing TOOL_CHAIN_TAG and -t option

When I call Edk2ToolsBuild.py without the -t option or pass a missing tool chain tag, the error message (exception) could be better...

Not really an extension issue, but related to building BaseTools the Pytool way.

On a system with multiple versions of Visual Studio and Build Tools, when I did not specify "-t":

(venv_edk2-pytools) C:\git\edk2-platforms-pytools\EDK2\BaseTools>python Edk2ToolsBuild.py
SECTION - Init SDE
SECTION - Loading Plugins
SECTION - Start Invocable Tool
Traceback (most recent call last):
  File "Edk2ToolsBuild.py", line 163, in <module>
    main()
  File "Edk2ToolsBuild.py", line 159, in main
    Edk2ToolsBuild().Invoke()
  File "c:\git\edk2-pytool-extensions\edk2toolext\base_abstract_invocable.py", line 141, in Invoke
    retcode = self.Go()
  File "Edk2ToolsBuild.py", line 120, in Go
    interesting_keys, 'x86', vs_version=self.tool_chain_tag.lower())
  File "C:\git\venv_edk2-pytools\lib\site-packages\edk2toollib\windows\locate_tools.py", line 175, in QueryVcVariables
    popen = subprocess.Popen('"%s" %s & set' % (vcvarsall_path, arch), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "C:\Python\Python37\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "C:\Python\Python37\lib\subprocess.py", line 1178, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

Example when specifying an incorrect -t option

(venv_pytools_dev) C:\git\edk2_ovmf\BaseTools>python Edk2ToolsBuild.py -t VS2015x86
SECTION - Init SDE
SECTION - Loading Plugins
SECTION - Start Invocable Tool
Traceback (most recent call last):
  File "Edk2ToolsBuild.py", line 163, in <module>
    main()
  File "Edk2ToolsBuild.py", line 159, in main
    Edk2ToolsBuild().Invoke()
  File "c:\git\edk2-pytool-extensions\edk2toolext\base_abstract_invocable.py", line 141, in Invoke
    retcode = self.Go()
  File "Edk2ToolsBuild.py", line 120, in Go
    interesting_keys, 'x86', vs_version=self.tool_chain_tag.lower())
  File "c:\git\edk2-pytool-library\edk2toollib\windows\locate_tools.py", line 172, in QueryVcVariables
    raise ValueError("Bad VC")
ValueError: Bad VC

Base Parser does not support quoted arguments or IN operators correctly

The Base Parser does not support a DSC file conditional statement such as the following:

!if $(CRYPTO_SERVICES) IN "ALL NONE MIN_PEI MIN_DXE_MIN_SMM"

One of the arguments is a quoted string that needs to be parsed as a single argument.

The IN operator tests for membership. The Base Parser only looks for 'in' and
does not support 'IN', 'In', or 'iN'. Also, when the 'in' operator is present, the
argument to check against membership is converted to an integer. It should
be converted to a list or set to check for membership.

utility_functions.py reader() raises when presented with output that doesn't decode with utf8

If RunCmd() is used to execute a utility that produces non-UTF8 output, reader() raises UnicodeDecodeError:

File "C:\Development\Devices\Redacted.py", line 105, in Redacted
ret = RunCmd(StitchCmd, params, workingdir=workingDir)
File "C:\Development\VirtualEnv\Development\lib\site-packages\edk2toollib\utility_functions.py", line 186, in RunCmd
thread.join()
File "C:\Development\VirtualEnv\Development\lib\site-packages\edk2toollib\utility_functions.py", line 69, in join
raise self.exc
File "C:\Development\VirtualEnv\Development\lib\site-packages\edk2toollib\utility_functions.py", line 62, in run
self.ret = self._target(*self._args, **self._kwargs)
File "C:\Development\VirtualEnv\Development\lib\site-packages\edk2toollib\utility_functions.py", line 89, in reader
s = stream.readline().decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xae in position 154: invalid start byte

Since it is hard to force all possible things that might be run with RunCmd() to be sane in their output, I think reader needs to pass (errors='ignore') when attempting to decode output.

stuart_ci_build does not support execution from paths other than the workspace root

stuart_ci_build is working when I execute it from the workspace root, but fails when I run it from a subdirectory.

For example, the following works from the Mu_Plus repo workspace root:
mu_plus>stuart_ci_build -c .pytool\CISettings.py -p FirmwarePolicyPkg

But when I change to a subdirectory:
mu_plus\FirmwarePolicyPkg>stuart_ci_build -c ..\.pytool\CISettings.py -p FirmwarePolicyPkg

it fails with

ERROR - --->Test Failed: Compiler Plugin DEBUG returned 1
PROGRESS - --Running FirmwarePolicyPkg: Compiler Plugin RELEASE --
CRITICAL - EXCEPTION: [Errno 2] No such file or directory: 'FirmwarePolicyPkg/FirmwarePolicyPkg.dsc'
CRITICAL - Traceback (most recent call last):
   File "c:\git\venv_1911\lib\site-packages\edk2toolext\invocables\edk2_ci_build.py", line 226, in Go
    tc, plugin_output_stream)
   File "C:\git\FwPolicy\fw_policy\MU_BASECORE\.pytool\Plugin\CompilerPlugin\CompilerPlugin.py", line 80, in RunBuildPlugin
    dp.ParseFile(AP_Path)
   File "c:\git\venv_1911\lib\site-packages\edk2toollib\uefi\edk2\parsers\dsc_parser.py", line 259, in ParseFile
    f = open(os.path.join(filepath), "r")
 FileNotFoundError: [Errno 2] No such file or directory: 'FirmwarePolicyPkg/FirmwarePolicyPkg.dsc'

ERROR - --->Test Failed: Compiler Plugin RELEASE returned 1

Bug: String encoding error in non-English environment.

Hello Developers,
When I use the lastest version of edk2-pytool-library to build Project mu, I meet the exception:

CRITICAL - Build Process Exception
ERROR - Traceback (most recent call last):
  File "d:\uefiworkspace\venv\lib\site-packages\edk2toolext\environment\uefi_build.py", line 152, in Go
    ret = self.Build()
  File "d:\uefiworkspace\venv\lib\site-packages\edk2toolext\environment\uefi_build.py", line 268, in Build
    ret = RunCmd("build", params)
  File "d:\uefiworkspace\venv\lib\site-packages\edk2toollib\utility_functions.py", line 187, in RunCmd
    thread.join()
  File "d:\uefiworkspace\venv\lib\site-packages\edk2toollib\utility_functions.py", line 70, in join
    raise self.exc
  File "d:\uefiworkspace\venv\lib\site-packages\edk2toollib\utility_functions.py", line 63, in run
    self.ret = self._target(*self._args, **self._kwargs)
  File "d:\uefiworkspace\venv\lib\site-packages\edk2toollib\utility_functions.py", line 90, in reader
    s = stream.readline().decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 130: invalid start byte

I don't know about Python programming, but based on online search data, I think it might be caused by a string encoding format error.
My system language is Chinese, and the related compilation software is also installed in the Chinese environment. So I think this is the source of this error.
Expect this error to be fixed so that my work can continue.If you need more information, I can provide more logging and testing environments.

cat_generator.py/inf_generator.py don't support VS2015

The code logic in cat_generator.py/inf_generator.py under edk2toollib/windows/capsule don't support VS2015, and only VS2017 or later. This is a back compatibility issue, the code use these two files(previous version) doesn't work now.

Btw, 3 binaries(vswhere.exe, signtool.exe and inf2cat.exe) are touched at least. The first comes from VS2017, 2nd comes from Windows 10 Software Development Kit (SDK) which will be installed if Desktop development with C++ is selected during VS2017 installation. But the 3rd one comes from WDK which should be installed separately.
For the content on web link https://microsoft.github.io/mu/CodeDevelopment/prerequisites/#visual-studio-2017, the WDK may be missed.

After installation, both WDK and SDK will be installed in C:\Program Files (x86)\Windows Kits\10\bin.
Signtool.exe is located in C:\Program Files (x86)\Windows Kits\10\bin\10.0.17763.0\x86 and C:\Program Files (x86)\Windows Kits\10\bin\10.0.17763.0\x64
Inf2cat.exe is located in C:\Program Files (x86)\Windows Kits\10\bin\10.0.17763.0\x86 - Question1 - is it correct behavior or I miss something ? no inf2cat.exe under x64.
Question2 - Should SDK and WDK version be consistent (in the same10.X.X.X ) and/or in the top/highest version?

DSC parser does not handle conditionals gMinPlatformPkgTokenSpaceGuid.PcdBootStage

See https://github.com/tianocore/edk2-platforms/blob/master/Platform/Intel/KabylakeOpenBoardPkg/KabylakeRvp3/OpenBoardPkgPcd.dsc

DEBUG - Parse Active Platform DSC file: Platform\Intel\KabylakeOpenBoardPkg/KabylakeRvp3/OpenBoardPkg.dsc DEBUG - Getting all build keys for build type DEBUG ERROR - <class 'edk2toollib.uefi.edk2.parsers.dsc_parser.DscParser'>: Unknown value: gMinPlatformPkgTokenSpaceGuid.PcdBootStage <class 'str'> CRITICAL - Build Process Exception ERROR - Traceback (most recent call last): File "c:\git\edk2-pytool-extensions\edk2toolext\environment\uefi_build.py", line 109, in Go ret = self.SetEnv() File "c:\git\edk2-pytool-extensions\edk2toolext\environment\uefi_build.py", line 378, in SetEnv ret = self.ParseDscFile() File "c:\git\edk2-pytool-extensions\edk2toolext\environment\uefi_build.py", line 492, in ParseDscFile dscp.ParseFile(dsc_file_path) File "c:\git\edk2-pytool-library\edk2toollib\uefi\edk2\parsers\dsc_parser.py", line 262, in ParseFile self.__ProcessDefines(file_lines) File "c:\git\edk2-pytool-library\edk2toollib\uefi\edk2\parsers\dsc_parser.py", line 247, in __ProcessDefines self.__ProcessDefines(add) File "c:\git\edk2-pytool-library\edk2toollib\uefi\edk2\parsers\dsc_parser.py", line 246, in __ProcessDefines (line, add) = self.__ParseDefineLine(l) File "c:\git\edk2-pytool-library\edk2toollib\uefi\edk2\parsers\dsc_parser.py", line 167, in __ParseDefineLine if(self.ProcessConditional(line_resolved)): File "c:\git\edk2-pytool-library\edk2toollib\uefi\edk2\parsers\base_parser.py", line 310, in ProcessConditional con = self.ComputeResult(tokens[1].strip(), tokens[2].strip(), tokens[3].strip()) File "c:\git\edk2-pytool-library\edk2toollib\uefi\edk2\parsers\base_parser.py", line 160, in ComputeResult raise ValueError("Unknown value") ValueError: Unknown value

Parsing Cache

It was asked that the parsing objects cache the results across steps. So basically, it would query if it had parsed a given object before and return the results if it had. We'll need to invalidate the hash if the environment has changed.

Improve error reporting when build tools do not include C++

For scenarios where both BuildTools and Visual Studio have been installed, library cannot always find vcvarsall.bat

venv_pytools_dev\lib\site-packages\edk2toollib\bin\vswhere.exe -latest -nologo -all -property installationPath -products *
returns
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools

But vcvarsall.bat is not under the BuildTools path, it is found:

dir /s "c:\Program Files (x86)"\vcvarsall.bat
 Volume in drive C is Local Disk
 Volume Serial Number is F000-B000

 Directory of c:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Auxiliary\Build

09/27/19  01:14 PM             9,889 vcvarsall.bat
               1 File(s)          9,889 bytes

 Directory of c:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build

10/24/19  05:24 PM             9,859 vcvarsall.bat
               1 File(s)          9,859 bytes

 Directory of c:\Program Files (x86)\Microsoft Visual Studio 14.0\VC

02/19/18  08:41 PM             3,600 vcvarsall.bat
               1 File(s)          3,600 bytes

How did I get here?

  1. Install VS2017 & VS2019 Build Tools
  2. Later install both Visual Studio 2017 Enterprise and 2019 Enterprise
  3. Attempt to install VS2015 Build Tools, but it does not really install
  4. Modify VS2017 Build Tools to include the VC2015 toolset

The actual, ugly error message is:

DEBUG - Calling 'C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Auxiliary\Build\vcvarsall.bat x86'
CRITICAL - Build Process Exception
ERROR - Traceback (most recent call last):
File "c:\git\edk2-pytool-extensions\edk2toolext\environment\uefi_build.py", line 127, in Go
ret = self.PreBuild()
File "c:\git\edk2-pytool-extensions\edk2toolext\environment\uefi_build.py", line 296, in PreBuild
rc = Descriptor.Obj.do_pre_build(self)
File "C:\git\edk2_py_platforms\Silicon\Intel\Tools\IntelSiliconTools.py", line 37, in do_pre_build
vs_vars = locate_tools.QueryVcVariables(interesting_keys, "x86")
File "c:\git\venv_pytools_dev\lib\site-packages\edk2toollib\windows\locate_tools.py", line 175, in QueryVcVariables
popen = subprocess.Popen('"%s" %s & set' % (vcvarsall_path, arch), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "C:\Program Files\Python37\lib\subprocess.py", line 775, in __init__ restore_signals, start_new_session)
File "C:\Program Files\Python37\lib\subprocess.py", line 1178, in _execute_child                                                                                                                                                                                                           startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

Parser isn't working correctly for MACRO replacement

It generates several errors like this. It looks like it replaces something like $(MSFT_FAMILY_PKG) to 0 since it can't find it

ERROR - Override processing error FILE_NOT_FOUND in file 0/Override/_____SamplePkg/Library/PeiPolicyInitLib/PeiPolicyInitLib.inf.

Remove markdown log file support

Is your feature request related to a problem? Please describe.
Remove the markdown logger. This logger isn't being used and just wastes time in the logging path.

Describe the solution you'd like
Remove it

DEC Parser does not support structured PCD declarations with braces {}

The DEC parser looks for exactly 4 elements in a PCD declaration statement.

There is an optional 5th element that is an open brace '{' to provide a set of
additional lines for a structured PCD. Relax the DEC parser to support up to
5 elements and ignore the DEC file lines between the '{' and '}'

Does not support "!if $(macro)" when macro is not defined

Try to build OvmfPkg.

The FDFs specify:

!if $(E1000_ENABLE)

Results in:

DEBUG - Parse Active Platform DSC file: C:\git\edk2_py_ovmf\OvmfPkg/OvmfPkgIa32X64.dsc
DEBUG - Getting all build keys for build type DEBUG
DEBUG - Parse Active Flash Definition (FDF) file
DEBUG - Getting all build keys for build type DEBUG
ERROR - !if conditionals need to be formatted correctly (spaces between each token)
CRITICAL - Build Process Exception
ERROR - Traceback (most recent call last):
File "c:\git\venv_pytools\lib\site-packages\edk2toolext\environment\uefi_build.py", line 109, in Go
  ret = self.SetEnv()
File "c:\git\venv_pytools\lib\site-packages\edk2toolext\environment\uefi_build.py", line 384, in SetEnv 
  ret = self.ParseFdfFile()
File "c:\git\venv_pytools\lib\site-packages\edk2toolext\environment\uefi_build.py", line 523, in ParseFdfFile 
   fdf_parser.ParseFile(pa)
File "c:\git\venv_pytools\lib\site-packages\edk2toollib\uefi\edk2\parsers\dsc_parser.py", line 262, in ParseFile 
   self.__ProcessDefines(file_lines)
File "c:\git\venv_pytools\lib\site-packages\edk2toollib\uefi\edk2\parsers\dsc_parser.py", line 246, in __ProcessDefines 
   (line, add) = self.__ParseDefineLine(l) 
File "c:\git\venv_pytools\lib\site-packages\edk2toollib\uefi\edk2\parsers\dsc_parser.py", line 167, in __ParseDefineLine  
   if(self.ProcessConditional(line_resolved)):  
File "c:\git\venv_pytools\lib\site-packages\edk2toollib\uefi\edk2\parsers\base_parser.py", line 271, in ProcessConditional  
   raise Exception("Invalid conditional", text)
Exception: ('Invalid conditional', '!if $(E1000_ENABLE)')

The expected usage is that this define is not present. If someone wants to add E1000 NIC support, the README advises:
'
Add "-D E1000_ENABLE" to your build command
'

The spec says, "When used with the !if or !elseif statements or in an expression used in a value field, a macro that has not been defined has a value of 0".

RunCmd() fails when invoked program writes a large amount to stdout (ex. greater than 770K)

This issue was discovered as part of a hang observed while running a Project Mu LicenseCheck.

The LicenseCheck plug-in calls RunCmd as follows:

return_buffer = StringIO()
params = "diff --unified=0 origin/master HEAD"
RunCmd("git", params, outstream=return_buffer)

RunCmd was found to hang if the diff generates a large amount of text (in my case, greater than 770K).

The following article suggests why this might be the case:
https://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/

When Multiple settings files are found, pick the one that has the deepest leaf

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
John Schock had a great solution that we'd like to move to

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

New version of Flake8 tool finds new violations

./BasicDevTests.py:51:17: E741 ambiguous variable name 'l'
./edk2toollib/uefi/edk2/path_utilities_test.py:581:14: F541 f-string is missing placeholders
./edk2toollib/uefi/edk2/build_objects/dsc_translator.py:27:13: E741 ambiguous variable name 'l'
./edk2toollib/uefi/edk2/parsers/dsc_parser.py:247:17: E741 ambiguous variable name 'l'
./edk2toollib/uefi/edk2/parsers/base_parser.py:117:13: E741 ambiguous variable name 'l'
./edk2toollib/uefi/edk2/parsers/base_parser.py:601:28: E741 ambiguous variable name 'l'
./edk2toollib/uefi/edk2/parsers/base_parser.py:615:25: E741 ambiguous variable name 'l'
./edk2toollib/uefi/edk2/parsers/base_parser.py:710:28: E741 ambiguous variable name 'l'
./edk2toollib/uefi/edk2/parsers/base_parser.py:721:31: E741 ambiguous variable name 'l'
./edk2toollib/uefi/edk2/parsers/dsc_parser_test.py:62:22: F541 f-string is missing placeholders
./edk2toollib/windows/locate_tools.py:61:27: F541 f-string is missing placeholders

Spelling error Unknown word (Mtftp)

According to a new updated spellchecker mtftp is not a word. This is blocking PRs and is most likely caused by a recent change in node/cspell and catching more words.

d:\a\1\s\edk2toollib\uefi\edk2\build_objects\dsc_translator_test.py:429:34 - Unknown word (Mtftp)
d:\a\1\s\edk2toollib\uefi\edk2\build_objects\dsc_translator_test.py:429:44 - Unknown word (Mtftp)

Stuart_setup on Ubuntu box

Hi,

I tried the setup as listed here:
https://github.com/tianocore/edk2/blob/master/EmulatorPkg/PlatformCI/ReadMe.md
on Ubuntu 18.04 LTS.

Sumanth@sumanth-VirtualBox:~/edk2$ stuart_setup -c EmulatorPkg/PlatformCI/PlatformBuild.py TOOL_CHAIN_TAG=GCC5
SECTION - Init SDE
SECTION - Loading Plugins
SECTION - Start Invocable Tool
PROGRESS - ## Syncing Git repositories: CryptoPkg/Library/OpensslLib/openssl ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 UnitTestFrameworkPkg/Library/CmockaLib/cmocka MdeModulePkg/Universal/RegularExpressionDxe/oniguruma MdeModulePkg/Library/BrotliCustomDecompressLib/brotli BaseTools/Source/C/BrotliCompress/brotli...
PROGRESS - Done.

PROGRESS - ## Checking Git repository: CryptoPkg/Library/OpensslLib/openssl...
PROGRESS - Done.

PROGRESS - ## Checking Git repository: ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3...
PROGRESS - Done.

PROGRESS - ## Checking Git repository: UnitTestFrameworkPkg/Library/CmockaLib/cmocka...
PROGRESS - Done.

PROGRESS - ## Checking Git repository: MdeModulePkg/Universal/RegularExpressionDxe/oniguruma...
PROGRESS - Done.

PROGRESS - ## Checking Git repository: MdeModulePkg/Library/BrotliCustomDecompressLib/brotli...
PROGRESS - Done.

PROGRESS - ## Checking Git repository: BaseTools/Source/C/BrotliCompress/brotli...
PROGRESS - Done.

SECTION - Summary
PROGRESS - Success

stuart_update -c EmulatorPkg/PlatformCI/PlatformBuild.py TOOL_CHAIN_TAG=GCC5**
SECTION - Init SDE
SECTION - Loading Plugins
SECTION - Start Invocable Tool
SECTION - Initial update of environment
UpdatingWARNING - [SDE] Failed to fetch NugetDependecy: [email protected]: [Nuget] We failed to install this version 2.14.02 of mu_nasm
. Done
SECTION - Second pass update of environment
UpdatingWARNING - [SDE] Failed to fetch NugetDependecy: [email protected]: [Nuget] We failed to install this version 2.14.02 of mu_nasm
. Done
ERROR - We were unable to successfully update 1 dependencies in environment
SECTION - Summary
ERROR - Error

How do i solve this issue on Ubuntu 18.04 LTS.

Regards,
Sumanth

dk2toollib.uefi.edk2.path_utilities Edk2Path.GetContainingPackage can have an endless loop on Windows due to case insensitivity

Since windows is not case sensitive an error has been seen when a caller creates an Edk2Path object with a workspace path drive letter of uppercase and then calls GetContainingPackage with an absolute path with drive letter of lower case.

Python 3.8 RealPath caused drive letter to be uppercase
Python 3.8 AbsPath had drive letter lowercase

GetContainingPackage runs in an endless loop in this case because the exit criteria is never met.
https://github.com/tianocore/edk2-pytool-library/blob/master/edk2toollib/uefi/edk2/path_utilities.py#L134

Review Path Utilities for other case issues, loop conditionals, and assumptions.
Add unit tests for there parameters.

stuart_setup -c ./CISettings.py doesn't work on FreeBSD

[bcran@photon ~/workspace/edk2-staging]$ stuart_setup -c ./CISettings.py 
Traceback (most recent call last):
  File "/home/bcran/.local/bin/stuart_setup", line 10, in <module>
    sys.exit(main())
  File "/home/bcran/.local/lib/python3.6/site-packages/edk2toolext/invocables/edk2_setup.py", line 196, in main
    Edk2PlatformSetup().Invoke()
  File "/home/bcran/.local/lib/python3.6/site-packages/edk2toolext/base_abstract_invocable.py", line 106, in Invoke
    self.ConfigureLogging()
  File "/home/bcran/.local/lib/python3.6/site-packages/edk2toolext/base_abstract_invocable.py", line 82, in ConfigureLogging
    edk2_logging.setup_console_logging(self.GetLoggingLevel("con"))
  File "/home/bcran/.local/lib/python3.6/site-packages/edk2toolext/edk2_logging.py", line 168, in setup_console_logging
    coloredHandler = ansi_handler.ColoredStreamHandler()
  File "/home/bcran/.local/lib/python3.6/site-packages/edk2toollib/log/ansi_handler.py", line 232, in __init__
    self.on_windows = GetHostInfo().os == "Windows"
  File "/home/bcran/.local/lib/python3.6/site-packages/edk2toollib/utility_functions.py", line 117, in GetHostInfo
    raise EnvironmentError("Host info could not be parsed: {0}".format(str(host_info)))
OSError: Host info could not be parsed: uname_result(system='FreeBSD', node='photon.int.bluestop.org', release='13.0-CURRENT', version='FreeBSD 13.0-CURRENT r352050 GENERIC-NODEBUG', machine='amd64', processor='amd64')

BaseParser fails to process conditionals correctly in two cases found in edk2 emulatorpkg dsc

Issue A

Conditional looks like !if "XCODE5" not in $(TOOL_CHAIN_TAG) assume TOOL_CHAIN_TAG = VS2019. But it doesn't actually seem to matter what the value of TOOL_CHAIN_TAG is.

File "c:\src_2020_Q1\pytool\edk2-pytool-library\edk2toollib\uefi\edk2\parsers\base_parser.py", line 477, in EvaluateConditional
raise RuntimeError(f"We didn't find an operator to execute in {expression}: {text}")
RuntimeError: We didn't find an operator to execute in ['"XCODE5"', False]: "XCODE5" not in VS2019

Issue B

Conditional looks like !if "IA32" in $(ARCH) || "X64" in $(ARCH)
If ARCH == X64 this works fine.
if ARCH == IA32 this fails to parse

File "c:\src_2020_Q1\pytool\edk2-pytool-library\edk2toollib\uefi\edk2\parsers\base_parser.py", line 145, in ComputeResult
return ivalue in ivalue2
TypeError: 'in ' requires string as left operand, not bool

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.