Giter Club home page Giter Club logo

msys2-contrib / cpython-mingw Goto Github PK

View Code? Open in Web Editor NEW

This project forked from python/cpython

36.0 7.0 12.0 550.48 MB

A friendly fork of CPython which adds support for Mingw-w64 + clang/gcc. See https://github.com/msys2-contrib/cpython-mingw/wiki for details

Home Page: https://github.com/msys2-contrib/cpython-mingw/wiki

License: Other

Makefile 0.07% Python 65.51% C 32.21% Batchfile 0.15% CSS 0.01% HTML 0.39% Assembly 0.10% C++ 0.71% DTrace 0.01% Shell 0.12% Roff 0.08% PLSQL 0.04% PowerShell 0.04% Objective-C 0.05% Common Lisp 0.05% M4 0.46% Rich Text Format 0.01% XSLT 0.01% VBScript 0.01% JavaScript 0.01%

cpython-mingw's Introduction

This is Python version 3.11.9

CPython build status on GitHub Actions CPython build status on Azure DevOps Python Discourse chat

Copyright © 2001-2023 Python Software Foundation. All rights reserved.

See the end of this file for further copyright and license information.

For more complete instructions on contributing to CPython development, see the Developer Guide.

Installable Python kits, and information about using Python, are available at python.org.

On Unix, Linux, BSD, macOS, and Cygwin:

./configure
make
make test
sudo make install

This will install Python as python3.

You can pass many options to the configure script; run ./configure --help to find out more. On macOS case-insensitive file systems and on Cygwin, the executable is called python.exe; elsewhere it's just python.

Building a complete Python installation requires the use of various additional third-party libraries, depending on your build platform and configure options. Not all standard library modules are buildable or useable on all platforms. Refer to the Install dependencies section of the Developer Guide for current detailed information on dependencies for various Linux distributions and macOS.

On macOS, there are additional configure and build options related to macOS framework and universal builds. Refer to Mac/README.rst.

On Windows, see PCbuild/readme.txt.

If you wish, you can create a subdirectory and invoke configure from there. For example:

mkdir debug
cd debug
../configure --with-pydebug
make
make test

(This will fail if you also built at the top-level directory. You should do a make clean at the top-level first.)

To get an optimized build of Python, configure --enable-optimizations before you run make. This sets the default make targets up to enable Profile Guided Optimization (PGO) and may be used to auto-enable Link Time Optimization (LTO) on some platforms. For more details, see the sections below.

PGO takes advantage of recent versions of the GCC or Clang compilers. If used, either via configure --enable-optimizations or by manually running make profile-opt regardless of configure flags, the optimized build process will perform the following steps:

The entire Python directory is cleaned of temporary files that may have resulted from a previous compilation.

An instrumented version of the interpreter is built, using suitable compiler flags for each flavor. Note that this is just an intermediary step. The binary resulting from this step is not good for real-life workloads as it has profiling instructions embedded inside.

After the instrumented interpreter is built, the Makefile will run a training workload. This is necessary in order to profile the interpreter's execution. Note also that any output, both stdout and stderr, that may appear at this step is suppressed.

The final step is to build the actual interpreter, using the information collected from the instrumented one. The end result will be a Python binary that is optimized; suitable for distribution or production installation.

Enabled via configure's --with-lto flag. LTO takes advantage of the ability of recent compiler toolchains to optimize across the otherwise arbitrary .o file boundary when building final executables or shared libraries for additional performance gains.

We have a comprehensive overview of the changes in the What's New in Python 3.11 document. For a more detailed change log, read Misc/NEWS, but a full accounting of changes can only be gleaned from the commit history.

If you want to install multiple versions of Python, see the section below entitled "Installing multiple versions".

Documentation for Python 3.11 is online, updated daily.

It can also be downloaded in many formats for faster access. The documentation is downloadable in HTML, PDF, and reStructuredText formats; the latter version is primarily for documentation authors, translators, and people with special formatting requirements.

For information about building Python's documentation, refer to Doc/README.rst.

To test the interpreter, type make test in the top-level directory. The test set produces some output. You can generally ignore the messages about skipped tests due to optional features which can't be imported. If a message is printed about a failed test or a traceback or core dump is produced, something is wrong.

By default, tests are prevented from overusing resources like disk space and memory. To enable these tests, run make testall.

If any tests fail, you can re-run the failing test(s) in verbose mode. For example, if test_os and test_gdb failed, you can run:

make test TESTOPTS="-v test_os test_gdb"

If the failure persists and appears to be a problem with Python rather than your environment, you can file a bug report and include relevant output from that command to show the issue.

See Running & Writing Tests for more on running tests.

On Unix and Mac systems if you intend to install multiple versions of Python using the same installation prefix (--prefix argument to the configure script) you must take care that your primary python executable is not overwritten by the installation of a different version. All files and directories installed using make altinstall contain the major and minor version and can thus live side-by-side. make install also creates ${prefix}/bin/python3 which refers to ${prefix}/bin/python3.X. If you intend to install multiple versions using the same prefix you must decide which version (if any) is your "primary" version. Install that version using make install. Install all other versions using make altinstall.

For example, if you want to install Python 2.7, 3.6, and 3.11 with 3.11 being the primary version, you would execute make install in your 3.11 build directory and make altinstall in the others.

See PEP 664 for Python 3.11 release details.

Copyright © 2001-2023 Python Software Foundation. All rights reserved.

Copyright © 2000 BeOpen.com. All rights reserved.

Copyright © 1995-2001 Corporation for National Research Initiatives. All rights reserved.

Copyright © 1991-1995 Stichting Mathematisch Centrum. All rights reserved.

See the LICENSE for information on the history of this software, terms & conditions for usage, and a DISCLAIMER OF ALL WARRANTIES.

This Python distribution contains no GNU General Public License (GPL) code, so it may be used in proprietary projects. There are interfaces to some GNU code but these are entirely optional.

All trademarks referenced herein are property of their respective holders.

cpython-mingw's People

Contributors

1st1 avatar akuchling avatar benjaminp avatar berkerpeksag avatar birkenfeld avatar bitdancer avatar brettcannon avatar ezio-melotti avatar freddrake avatar gpshead avatar gvanrossum avatar jackjansen avatar jeremyhylton avatar loewis avatar mdickinson avatar merwok avatar miss-islington avatar ned-deily avatar nnorwitz avatar orsenthil avatar pitrou avatar rhettinger avatar serhiy-storchaka avatar terryjreedy avatar tim-one avatar tiran avatar vsajip avatar vstinner avatar warsaw avatar zooba avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

cpython-mingw's Issues

PY3_DLLNAME not correct?

I noticed a warning in the ARM64 build:

In file included from <built-in>:374:
<command line>:5:9: warning: 'PY3_DLLNAME' macro redefined [-Wmacro-redefined]
#define PY3_DLLNAME L"libpython3.10.dll"
        ^
<command line>:3:9: note: previous definition is here
#define PY3_DLLNAME "libpython3.dll"
        ^

There was another warning elsewhere:

../Python/pathconfig.c:338:45: warning: incompatible pointer types passing 'char [15]' to parameter of type 'LPCWSTR' (aka 'const unsigned short *') [-Wincompatible-pointer-types]
        HMODULE hPython3 = GetModuleHandleW(PY3_DLLNAME);
                                            ^~~~~~~~~~~
<command line>:3:21: note: expanded from here
#define PY3_DLLNAME "libpython3.dll"
                    ^~~~~~~~~~~~~~~~
C:/msys64/clangarm64/aarch64-w64-mingw32/include/libloaderapi.h:140:53: note: passing argument to parameter 'lpModuleName' here
WINBASEAPI HMODULE WINAPI GetModuleHandleW (LPCWSTR lpModuleName);

I think that PY3_DLLNAME should be L"libpython3.10.dll" not "libpython3.dll"

fail build if a module fails to build?

After msys2/MINGW-packages#9645, maybe it would make sense to make the build fail if a module fails to build? I did some nasty hacks with tee and sed in the MSYS2-packages/python PKGBUILD to detect a failed module and fail the build, but I know that Gentoo patches python to make it fail there instead.

Upstream things

We should create a meta upstream bug tracker entry for small mingw related cleanups and start creating PRs.

This of course only works for things which can be considered cleanups/refactorings from the upstream POV and not mingw specific changes. Also excluding distutils, since upstream no longer takes changes for that.

profile tests fail with KeyError '2' on arm64

LLVM_PROFILE_FILE="code-%p.profclangr"  ./python.exe -m test --pgo || true
Traceback (most recent call last):
  File "C:/_/M/mingw-w64-python/src/Python-3.9.6/Lib/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:/_/M/mingw-w64-python/src/Python-3.9.6/Lib/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:/_/M/mingw-w64-python/src/Python-3.9.6/Lib/test/__main__.py", line 2, in <module>
    main()
  File "C:/_/M/mingw-w64-python/src/Python-3.9.6/Lib/test/libregrtest/main.py", line 716, in main
    Regrtest().main(tests=tests, **kwargs)
  File "C:/_/M/mingw-w64-python/src/Python-3.9.6/Lib/test/libregrtest/main.py", line 638, in main
    self._main(tests, kwargs)
  File "C:/_/M/mingw-w64-python/src/Python-3.9.6/Lib/test/libregrtest/main.py", line 684, in _main
    self.win_load_tracker = WindowsLoadTracker()
  File "C:/_/M/mingw-w64-python/src/Python-3.9.6/Lib/test/libregrtest/win_utils.py", line 41, in __init__
    self.start()
  File "C:/_/M/mingw-w64-python/src/Python-3.9.6/Lib/test/libregrtest/win_utils.py", line 70, in start
    counter_name = self._get_counter_name()
  File "C:/_/M/mingw-w64-python/src/Python-3.9.6/Lib/test/libregrtest/win_utils.py", line 88, in _get_counter_name
    system = counters_dict['2']
KeyError: '2'

I don't know if this happens on any other arches.

smoketests fails from build tree if python3.9 is not already installed

Updated smoketests.py in mingw-w64-python from the version in this repo and attempted to build with check enabled. Got this output:

check output
==> Starting check()...
Looking in links: c:/Users/ADMINI~1/AppData/Local/Temp/tmplubvt8of
Processing c:/users/administrator/appdata/local/temp/tmplubvt8of/setuptools-56.0.0-py3-none-any.whl
Processing c:/users/administrator/appdata/local/temp/tmplubvt8of/pip-21.1.3-py3-none-any.whl
Installing collected packages: setuptools, pip
  WARNING: The scripts pip3.9.exe and pip3.exe are installed in 'C:\Users\Administrator/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-21.1.3 setuptools-56.0.0
Collecting wheel
  Downloading wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel
Successfully installed wheel-0.36.2
Processing c:/users/admini~1/appdata/local/temp/tmplfi0sqnk
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
Building wheels for collected packages: cwrapper
  Building wheel for cwrapper (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: T:/mingw-w64-python3.9/src/build-aarch64/python.exe -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/setup.py'"'"'; __file__='"'"'C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d C:/Users/Administrator/AppData/Local/Temp/pip-wheel-mnp_u_w2
       cwd: C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/
  Complete output (13 lines):
  running bdist_wheel
  running build
  running build_ext
  building 'cwrapper' extension
  creating build
  creating build/temp.mingw_aarch64-3.9
  clang -fno-strict-aliasing -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -D__USE_MINGW_ANSI_STDIO=1 -IC:/msys32/clangarm64/include/ncurses -IT:/mingw-w64-python3.9/src/Python-3.9.6/Include -IT:/mingw-w64-python3.9/src/build-aarch64 -c cwrapper.c -o build/temp.mingw_aarch64-3.9/cwrapper.o
  writing build/temp.mingw_aarch64-3.9/cwrapper.cp39-mingw_aarch64.def
  creating build/lib.mingw_aarch64-3.9
  clang -shared -Wl,--enable-auto-image-base -pipe -Wl,--dynamicbase -pipe -Wl,--dynamicbase -pipe -Wl,--dynamicbase -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -D__USE_MINGW_ANSI_STDIO=1 -IC:/msys32/clangarm64/include/ncurses -s build/temp.mingw_aarch64-3.9/cwrapper.o build/temp.mingw_aarch64-3.9/cwrapper.cp39-mingw_aarch64.def -L. -L. -lpython3.9 -lm -lversion -lshlwapi -o build/lib.mingw_aarch64-3.9/cwrapper.cp39-mingw_aarch64.pyd
  lld: error: unable to find library -lpython3.9
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  error: command 'C:\\msys32\\clangarm64\\bin/clang.exe' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for cwrapper
  Running setup.py clean for cwrapper
Failed to build cwrapper
Installing collected packages: cwrapper
    Running setup.py install for cwrapper ... error
    ERROR: Command errored out with exit status 1:
     command: T:/mingw-w64-python3.9/src/build-aarch64/python.exe -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/setup.py'"'"'; __file__='"'"'C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record C:/Users/Administrator/AppData/Local/Temp/pip-record-_d0pvkfe/install-record.txt --single-version-externally-managed --compile --install-headers C:/msys32/clangarm64/include/python3.9/cwrapper
         cwd: C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/
    Complete output (13 lines):
    running install
    running build
    running build_ext
    building 'cwrapper' extension
    creating build
    creating build/temp.mingw_aarch64-3.9
    clang -fno-strict-aliasing -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -D__USE_MINGW_ANSI_STDIO=1 -IC:/msys32/clangarm64/include/ncurses -IT:/mingw-w64-python3.9/src/Python-3.9.6/Include -IT:/mingw-w64-python3.9/src/build-aarch64 -c cwrapper.c -o build/temp.mingw_aarch64-3.9/cwrapper.o
    writing build/temp.mingw_aarch64-3.9/cwrapper.cp39-mingw_aarch64.def
    creating build/lib.mingw_aarch64-3.9
    clang -shared -Wl,--enable-auto-image-base -pipe -Wl,--dynamicbase -pipe -Wl,--dynamicbase -pipe -Wl,--dynamicbase -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -D__USE_MINGW_ANSI_STDIO=1 -IC:/msys32/clangarm64/include/ncurses -s build/temp.mingw_aarch64-3.9/cwrapper.o build/temp.mingw_aarch64-3.9/cwrapper.cp39-mingw_aarch64.def -L. -L. -lpython3.9 -lm -lversion -lshlwapi -o build/lib.mingw_aarch64-3.9/cwrapper.cp39-mingw_aarch64.pyd
    lld: error: unable to find library -lpython3.9
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    error: command 'C:\\msys32\\clangarm64\\bin/clang.exe' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: T:/mingw-w64-python3.9/src/build-aarch64/python.exe -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/setup.py'"'"'; __file__='"'"'C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record C:/Users/Administrator/AppData/Local/Temp/pip-record-_d0pvkfe/install-record.txt --single-version-externally-managed --compile --install-headers C:/msys32/clangarm64/include/python3.9/cwrapper Check the logs for full command output.
E...............
======================================================================
ERROR: test_c_ext_build (__main__.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "T:\mingw-w64-python3.9\src\smoketests.py", line 218, in test_c_ext_build
    subprocess.check_call(
  File "T:/mingw-w64-python3.9/src/Python-3.9.6/Lib/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['T:/mingw-w64-python3.9/src/build-aarch64/python.exe', '-m', 'pip', 'install', 'C:/Users/ADMINI~1/AppData/Local/Temp/tmplfi0sqnk']' returned non-zero exit status 1.

----------------------------------------------------------------------
Ran 16 tests in 77.263s

FAILED (errors=1)
==> ERROR: A failure occurred in check().

It seems to have been missing the -L option to find the libpython3.9.dll.a import library

TZPATH not converted to windows paths

I have a sample to test zoneinfo module in py39. It doesn't run in mingw. After some debugging I discovered that the sysconfig.get_config_var("TZPATH") is the source of the issue.
I think that a change in Makefile.pre.in like in this commit ae4be49#diff-d593bd299ba58e440ba411ffa0640ccd9d20d518b0cf2644ed4bdb75a82a3e70 should fix this issue.
To test is easy:

import zoneinfo
zoneinfo.TZPATH
zoneinfo.ZoneInfo("America/Sao_Paulo")
Python 3.9.6 (default, Jul 12 2021, 05:46:11)  [GCC 10.3.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import zoneinfo
>>> zoneinfo.TZPATH
('/usr/share/zoneinfo:/usr/lib/zoneinfo:/usr/share/lib/zoneinfo:/etc/zoneinfo',)
>>> zoneinfo.ZoneInfo("America/Sao_Paulo")
Traceback (most recent call last):
  File "C:/msys64/mingw64/lib/python3.9/zoneinfo/_common.py", line 12, in load_tzdata
    return importlib.resources.open_binary(package_name, resource_name)
  File "C:/msys64/mingw64/lib/python3.9/importlib/resources.py", line 88, in open_binary
    package = _get_package(package)
  File "C:/msys64/mingw64/lib/python3.9/importlib/resources.py", line 49, in _get_package
    module = _resolve(package)
  File "C:/msys64/mingw64/lib/python3.9/importlib/resources.py", line 40, in _resolve
    return import_module(name)
  File "C:/msys64/mingw64/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'tzdata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:/msys64/mingw64/lib/python3.9/zoneinfo/_common.py", line 24, in load_tzdata
    raise ZoneInfoNotFoundError(f"No time zone found with key {key}")
zoneinfo._common.ZoneInfoNotFoundError: 'No time zone found with key America/Sao_Paulo'

MACHDEP and sys.platform mismatch

I came across this problem when testing lldb with python. Apparently MACHDEP and sys.platform don't match, and it causes an error when starting the interpreter

(lldb) script
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\llvm-mingw-extras\python\lib\python3.9\pydoc.py", line 458, in <module>
    class Doc:
  File "C:\llvm-mingw-extras\python\lib\python3.9\pydoc.py", line 488, in Doc
    def getdocloc(self, object, basedir=sysconfig.get_path('stdlib')):
  File "C:\llvm-mingw-extras\python\lib\python3.9\sysconfig.py", line 554, in get_path
    return get_paths(scheme, vars, expand)[name]
  File "C:\llvm-mingw-extras\python\lib\python3.9\sysconfig.py", line 544, in get_paths
    return _expand_vars(scheme, vars)
  File "C:\llvm-mingw-extras\python\lib\python3.9\sysconfig.py", line 179, in _expand_vars
    _extend_dict(vars, get_config_vars())
  File "C:\llvm-mingw-extras\python\lib\python3.9\sysconfig.py", line 594, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "C:\llvm-mingw-extras\python\lib\python3.9\sysconfig.py", line 460, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata__win32_'
Traceback (most recent call last):
  File "<input>", line 1, in <module>
NameError: name 'pydoc' is not defined
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> sys.platform
'win32'
>>>

I looked into configure.ac and saw this comment:

cpython-mingw/configure.ac

Lines 386 to 390 in 91ea75c

case $host in
*-*-mingw*)
# On those hosts MACHDEP is 'win', as platform specific files
# go in plat-$(MACHDEP), but runtime platform is 'win32'
test -z "$MACHDEP" && MACHDEP=win

Did you encounter this on the native build, or this is just a problem of cross-compiling?

Runtime error: not found 'posix' module

I can build with MinGW but when running, Python.exe outputs errors:

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1207, in _install_external_importers
ModuleNotFoundError: No module named 'posix'
Fatal Python error: init_importlib_external: external importer setup failed
Python runtime state: core initialized

Current thread 0x0000372c (most recent call first):
  <no Python frame>

That's when I go to the build directory and try the Executable.

The building process is not finished though, here is the output:

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1207, in _install_external_importers
ModuleNotFoundError: No module named 'posix'
Fatal Python error: init_importlib_external: external importer setup failed
Python runtime state: core initialized

Current thread 0x0000372c (most recent call first):
  <no Python frame>
make[3]: *** [Makefile:651: pybuilddir.txt] Error 1
make[2]: *** [Makefile:547: build_all_generate_profile] Error 2
make[1]: *** [Makefile:523: profile-gen-stamp] Error 2
make: *** [Makefile:535: profile-run-stamp] Error 2

Last time I used with POSIX threading model, this error also occurs. Tried with Win32 threading too, still similar output.
My build configuration is simply : ./configure --enable-shared --enable-optimizations

Clean up CFLAGS

Most (if not all?) CFLAGS set before configure should be CFLAGS_NODIST instead, so they don't get used when building extensions with distutils.

AC_RUN_IFELSE not suitable for cross compiling

There are 2 instances of AC_RUN_IFELSE in the configure script that prevents cpython from cross-compiling

cpython-mingw/configure.ac

Lines 5135 to 5140 in b38e504

AC_RUN_IFELSE([AC_LANG_SOURCE([[
#include <stdio.h>
int main(){
_UCRT;
}
]])],[linking_to_ucrt=yes],[linking_to_ucrt=no])

This one can be changed to AC_COMPILE_IFELSE by modifying the test code to

#ifndef _UCRT
    #error "no ucrt"
#endif

cpython-mingw/configure.ac

Lines 5950 to 5960 in b38e504

AC_RUN_IFELSE([AC_LANG_SOURCE([[
#include <stdio.h>
#include "patchlevel.h"
int main() {
fprintf (
fopen ("conftest.out","w"),
"%d",
PY_MICRO_VERSION * 1000 + PY_RELEASE_LEVEL * 10 + PY_RELEASE_SERIAL);
return 0;
}
]])], [FIELD3=`cat conftest.out`])

This is trickier, but still doable. We can use grep to grab each macro from patchlevel.h, then another grep to grab the value of it.

PY_MICRO_VERSION=$(grep 'PY_MICRO_VERSION' Include/patchlevel.h | grep -o '[0-9]')
PY_RELEASE_LEVEL=15 # hardcoded since we only use stable versions
PY_RELEASE_SERIAL=$(grep 'PY_RELEASE_SERIAL' Include/patchlevel.h | grep -o '[0-9]')

and either use expr or let to calculate the final FIELD3 value.

Failure with LTO enabled in GCC

I ran this locally, and I'm seeing heap corruption:

WinDbgX analysis
Critical error detected c0000374
(218.4ec): Break instruction exception - code 80000003 (first chance)
ntdll!RtlReportCriticalFailure+0x56:
00007ff9`aa02f122 cc              int     3
0:000> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 1718

    Key  : Analysis.DebugAnalysisManager
    Value: Create

    Key  : Analysis.Elapsed.mSec
    Value: 22192

    Key  : Analysis.Init.CPU.mSec
    Value: 499

    Key  : Analysis.Init.Elapsed.mSec
    Value: 27069

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 52

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 92809

    Key  : Timeline.Process.Start.DeltaSec
    Value: 26

    Key  : WER.OS.Branch
    Value: vb_release

    Key  : WER.OS.Timestamp
    Value: 2019-12-06T14:06:00Z

    Key  : WER.OS.Version
    Value: 10.0.19041.1

    Key  : WER.Process.Version
    Value: 3.9.6000.1013


NTGLOBALFLAG:  70

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00007ff9aa02f122 (ntdll!RtlReportCriticalFailure+0x0000000000000056)
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 1
   Parameter[0]: 0000000000000000

FAULTING_THREAD:  000004ec

PROCESS_NAME:  python.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE_STR:  80000003

EXCEPTION_PARAMETER1:  0000000000000000

ADDITIONAL_DEBUG_TEXT:  Enable Pageheap/AutoVerifer ; Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]

STACK_TEXT:  
00000000`00000000 00000000`00000000 heap_corruption!python.exe+0x0


STACK_COMMAND:  ** Pseudo Context ** ManagedPseudo ** Value: ffffffff ** ; kb

SYMBOL_NAME:  heap_corruption!python.exe

MODULE_NAME: heap_corruption

IMAGE_NAME:  heap_corruption

FAILURE_BUCKET_ID:  HEAP_CORRUPTION_80000003_heap_corruption!python.exe

OS_VERSION:  10.0.19041.1

BUILDLAB_STR:  vb_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

FAILURE_ID_HASH:  {ba8abe75-133c-bc8e-6aca-284ac0a108d6}

Followup:     MachineOwner
---------

I will try pageheap/autoverifier as recommended by windbgx, and building with debug symbols.

build in MSYS2+Mingw64 on windows 10 64 bits?

as the title says, I am using windows 10, but I can't make a fresh python:

Any hints? Thanks

$ ./configure --enable-optimizations
...
configure: creating ./config.status
config.status: creating Makefile.pre
config.status: creating Misc/python.pc
config.status: creating Misc/python-embed.pc
config.status: creating Misc/python-config.sh
config.status: creating Modules/ld_so_aix
config.status: creating pyconfig.h
config.status: pyconfig.h is unchanged
creating Modules/Setup.local
creating Makefile


$ make
make: *** No rule to make target 'Modules/Setup.config', needed by 'Makefile'.  Stop.

Buffer overflow in Py_GetSepW

When building python with ASAN, it detected a buffer overflow here:

if (name != NULL && memcmp(name, L"\\\\?\\", sizeof(L"\\\\?\\") - sizeof(wchar_t)) == 0)

Looks like this code assumes that name is longer than \\?\, hence the buffer overflow.

For fixing this I used this shamelessly copypasted function (python might have such function already, but I couldn't find it quickly):

static int _starts_with(const wchar_t * str, const wchar_t * prefix)
{
    while(*prefix)
    {
        if(*prefix++ != *str++)
            return 0;
    }

    return 1;
}
wchar_t
Py_GetSepW(const wchar_t *name)
{
    /* ... */
    if (name != NULL && _starts_with(name, L"\\\\?\\"))

This change seems to make ASAN happy, and compiled python works fine as well.

If someone could suggest a better starts_with() function to use here, or if it's fine to keep this one, I'll submit the PR.

Python 3.11 Update

There are a lot of build system changes/improvements because of wasm, which means more work to rebase, but should help us in the long run (more auto detection, more pkg-config). The new frozen core modules also look like that could be tricky... we'll see.

Checklist:

  • Drop windows 7 related patches
  • Drop iscygpty patches (??)
  • Regenerate Importlib
  • Fix #90

Don't expose a new API

Currently, 8264c28 exposes around 4 new API, ie Py_GetSepW, Py_GetSepA, Py_NormalizeSepsW, Py_NormalizeSepsA. They should be private.

Update to v3.10.4

Python 3.10 is released a few weeks ago. We should update it probably when Arch Linux does. I have rebased most of the patches in this branch mingw-v3.10.0. It still has some problems though.

Should we drop support for Windows 7 from this version?

Todos

  • Use os.winver for installing user packages. See _get_path() in Lib\site.py. (#68 should fix this)
  • Add 4ad86b7af back.
  • Fix ensurepip and venv module. See smoke tests for what happens.
  • Change of \ to / doesn't work. Probally issue with 088ba0b, should check, (done 2ea5fa3)
  • cygwin tty detection not working #76
  • #80
  • #81
  • #82 (see #90, not worth fixing for this version)

error: implicit declaration of function 'setenv

../../cpython-mingw/Python/pylifecycle.c: In function '_coerce_default_locale_settings':
../../cpython-mingw/Python/pylifecycle.c:320:9: error: implicit declaration of function 'setenv'; did you mean 'getenv'? [-Werror=implicit-function-declaration]
320 | if (setenv("LC_CTYPE", newloc, 1)) {
| ^~~~~~
| getenv
../../cpython-mingw/Python/sysmodule.c:1445:42: warning: missing braces around initializer [-Wmissing-braces]
1445 | static PyTypeObject WindowsVersionType = {0, 0, 0, 0, 0, 0};
| ^
| {{ } }

python 3.10

New venv launchers are broken

Got the recent build from staging created the virtual environment with python3.10. The venv was created as expected. Then I ran the executable bin/python.exe and it launched python 3.9 (??). I then set and an env var to debug those launcher PYLAUNCH_DEBUG=1 and ended up with these logs

❯ .venv-3.10/bin/python --version
launcher build: 32bit
launcher executable: Console
File 'C:\dev\temp-test\.venv-3.10\bin\pyvenv.cfg' non-existent
Using venv configuration file 'C:\dev\temp-test\.venv-3.10\pyvenv.cfg'
Called with command line: --version
run_child: about to run 'C:/msys64/mingw32/bin\python.exe --version'
Python 3.9.10
child process exit code: 0

It launched the python.exe instead of python3.10.exe. This is the first broken behaviour I noticed.

Then I copied python3.10.exe to python.exe and then recreated that virtual environment. While recreating it ended up running the ensurepip module (the one that bootstraps pip and setuptools) globally and not in the virtual environment

❯ python -m venv .venv-3.10
launcher build: 32bit
launcher executable: Console
File 'C:\dev\temp-test\.venv-3.10\bin\pyvenv.cfg' non-existent
Using venv configuration file 'C:\dev\temp-test\.venv-3.10\pyvenv.cfg'
Called with command line: -Im ensurepip --upgrade --default-pip
run_child: about to run 'C:/msys64/mingw32/bin\python.exe -Im ensurepip --upgrade --default-pip'
Looking in links: c:\Users\naveen\AppData\Local\Temp\tmp_dm1_pvu
Processing c:\users\naveen\appdata\local\temp\tmp_dm1_pvu\setuptools-58.1.0-py3-none-any.whl
Processing c:\users\naveen\appdata\local\temp\tmp_dm1_pvu\pip-21.2.4-py3-none-any.whl
Installing collected packages: setuptools, pip
Successfully installed pip-21.2.4 setuptools-58.1.0
child process exit code: 0

This is the second broken behaviour I noticed.

Next I ran bin/python -c 'import sys;print(sys.executable)' and it ended up printing

❯ .venv-3.10/bin/python -c 'import sys; print(sys.executable)'
C:/msys64/mingw32/bin\python.exe

which is again wrong, because it should be printing the virtual environment's executable and not the original one. This is the third broken behaviour I noticed.

Now, the last one made me think that somewhere our python can't detect it's a venv when launched through those launchers, but it could detect it when those executables are copied. I looked into where sys.executable is set (it's in Modules/getpath.c) and found that the file is only for UNIX like system and MSVC python uses PC/getpathp.c which I think is the root cause of the problems (from this comment below).

* NOTE: Windows MSVC builds use PC/getpathp.c instead!

I think the solutions for these problems requires more work, essentially copying stuff from PC/getpathp.c into Modules/getpath.c for supporting these new launchers. Or maybe use PC/getpathp.c rather than just Modules/getpath.c, but it still requires work to move patches from the latter. Also, upstream in Python 3.11 ended up changing these C modules to Python python#29041, so the work we would do would be wasted when upgrading to Python 3.11. Not sure what is the best thing to do here.

Also, I think those patches from #82 should be reverted until these broken behaviours are fixed.

Questions about "building Python stdlib" checks inside distutils

if sys.platform[:6] == 'cygwin' or self.plat_name.startswith(('mingw')):
if sys.executable.startswith(os.path.join(sys.exec_prefix, "bin")):
# building third party extensions
config_dir_name = os.path.basename(sysconfig.get_config_var('LIBPL'))
self.library_dirs.append(os.path.join(sys.prefix, "lib",
"python" + get_python_version(),
config_dir_name))
else:
# building python standard extensions
self.library_dirs.append('.')

I'm having a problem with this check when cross-compiling. Somehow sys.executable.startswith(os.path.join(sys.exec_prefix, "bin")): returns True, and it incorrectly appends the native Python library path to the linker path, which causes linker errors due to a library file mismatch.

However, a few lines down, I saw another check, which seems to do the same thing, but with another condition

if (sysconfig.get_config_var('Py_ENABLE_SHARED')):
if not sysconfig.python_build:
# building third party extensions
self.library_dirs.append(sysconfig.get_config_var('LIBDIR'))
else:
# building python standard extensions
self.library_dirs.append('.')

My question is, how does the first check work, and why does it fail for us but not for you?

DistutilsChangeRootError not defined

Traceback (most recent call last):
  File "D:\a\cpython-mingw\cpython-mingw\Lib\distutils\tests\test_install.py", line 239, in test_debug_mode
    self.test_record()
  File "D:\a\cpython-mingw\cpython-mingw\Lib\distutils\tests\test_install.py", line 185, in test_record
    cmd.ensure_finalized()
  File "D:\a\cpython-mingw\cpython-mingw\Lib\distutils\cmd.py", line 107, in ensure_finalized
    self.finalize_options()
  File "D:\a\cpython-mingw\cpython-mingw\Lib\distutils\command\install.py", line 362, in finalize_options
    self.change_roots('libbase', 'lib', 'purelib', 'platlib',
  File "D:\a\cpython-mingw\cpython-mingw\Lib\distutils\command\install.py", line 529, in change_roots
    setattr(self, attr, change_root(self.root, getattr(self, attr)))
  File "D:\a\cpython-mingw\cpython-mingw\Lib\distutils\util.py", line 183, in change_root
    raise DistutilsChangeRootError("root and pathname not on same drive (%s, %s)"
NameError: name 'DistutilsChangeRootError' is not defined

It was added in c96759d

mingw_smoketests test_ctypes_find_library fails on ARM

The condition for _UCRT isn't right. I was thinking of changing it to

_UCRT = any(w in sys.version.lower() for w in ("clang", "ucrt"))

but I'm thinking now it may be easier to say what isn't UCRT:

_UCRT = sysconfig.get_platform() not in ('mingw_x86_64', 'mingw_i686')

3.10: cygwin tty detection not working

running "python3.10" does not start it in interactive mode, unlike with 3.9. We hacked this in by replacing all isatty calls. maybe there are new ones we need to patch.

Remove workaround to support AddDllDirectory

We've basically reverted the change documented here for 3.8: https://docs.python.org/3/library/os.html#os.add_dll_directory

4eb2649#diff-53cfc4d59c58055e49fd0354379fe7caead9f3b2868d7d1a5b2dd85232b65178L194

to keep .dll loading working as before. Which breaks os.add_dll_directory.

We should instead add the relocated prefix to the search path by default somehow and revert that patch to LoadLibraryExW. So our Python behaves more like upstream.

IIRC anaconda did something similar.

Build venv launcher

These are missing and should be built from PC/launcher.c and named as venvlauncher.exe and venvwlauncher.exe

Clean up the branch

Might be nice to squash some things, and maybe create a "mingw-v3.9.6-v2" branch for it

cross-compiling issue finding libraries

I used this project to test msys2/MINGW-packages#8762 x86_64 to aarch64 cross compiling using clangarm64 packages. It seemed to succeed, except for the ctypes module. I added some print statements and narrowed it down to

cpython-mingw/setup.py

Lines 2265 to 2269 in 8736112

if ffi_lib is None and ffi_inc:
for lib_name in ('ffi', 'ffi_pic'):
if (self.compiler.find_library_file(self.lib_dirs, lib_name)):
ffi_lib = lib_name
break

Printing out self.lib_dirs was ['.', '.']

It kind of looks like the add_cross_compiling_paths is intended to help here, but clang is not outputing any library path from its -E -v output, only include paths.

As a workaround, just setting ffi_lib = 'ffi' allows ctypes module to build successfully, showing it is just this setup.py detection logic having problems.

Default to -O2

The tests assume that we build with -O2 because that reduces the stack size. We should default to it and add a comment why.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.