pypa / pyproject-hooks Goto Github PK

View Code? Open in Web Editor NEW

121.0 16.0 49.0 363 KB

A low-level library for calling build-backends in `pyproject.toml`-based project

Home Page: https://pyproject-hooks.readthedocs.io/

License: MIT License

Python 100.00%

python packaging

pyproject-hooks's Introduction

`pyproject-hooks`

This is a low-level library for calling build-backends in pyproject.toml-based project. It provides the basic functionality to help write tooling that generates distribution files from Python projects.

If you want a tool that builds Python packages, you'll want to use https://github.com/pypa/build instead. This is an underlying piece for pip, build and other "build frontends" use to call "build backends" within them.

You can read more in the documentation.

Note: The pep517 project has been replaced by this project (low level) and the build project (high level).

pyproject-hooks's People

Contributors

Stargazers

Watchers

pyproject-hooks's Issues

Don't pip install without user knowledge and allow to opt out

It seems like pep517.build and pep517.check will run pip install on their own without ever telling anything to the user, and there's no way to disable this.

Linux distributions often need a tool that simply builds a wheel, one that does not go run things like pip install on our backs. A lot of the reasons have been outlined in pypa/setuptools#2080. The problem is not just it is using pip, it's that it is fetching sources from the internet and installing packages without I ever tell it to do it. When I make a distribution package I want to be the one fetching and verifying all sources, and I want to be in control of what is installed to the filesystem.

All I need is a simple build system without many dependencies (because we will need to bootstrap those, which is a lot of work). It should not do any package management, just build what I ask it to. It will fail if there are missing build dependencies.

And I believe this should be the default behavior, I think python -m pep517.* should be a build system first (actually, I think it should only be a build system but I don't think you agree). If a user wants to install a project from source, pip works, but please let's not add pip as a dependency at this level. If you really want pip to be run I would suggest adding a command-line option that would enable that behavior.

Right now python -m pep517.build will error out because pip is not installed:

/usr/bin/python: No module named pip
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.8/site-packages/pep517/build.py", line 124, in <module>
    main(parser.parse_args())
  File "/usr/lib/python3.8/site-packages/pep517/build.py", line 120, in main
    build(args.source_dir, dist, args.out_dir)
  File "/usr/lib/python3.8/site-packages/pep517/build.py", line 87, in build
    env.pip_install(system['requires'])
  File "/usr/lib/python3.8/site-packages/pep517/envbuild.py", line 100, in pip_install
    check_call(
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '-m', 'pip', 'install', '--ignore-installed', '--prefix', '/tmp/pep517-build-env-v568jxh0', 'flit_core >=2,<3']' returned non-zero exit status 1.

There is no reason for pip to be there if I have every build dependency already.

Just a note on the shape python packaging is taking. I believe everything here should be modular, having a build system and installer, and then a package manager. I think pip should offload the work to these tools and just deal it the package management (fetching sources, managing dependencies, etc.).

Automated releases broken due to ./pep517 in path

In #32, I added support for automatically cutting releases from tagged commits based on a pattern I've adopted in other projects. This technique fails for this project in particular because (a) the technique relies on functionality not yet merged into master (#27) and (b) because the project is checked out locally, the local checkout takes precedence.

Pep517HookCaller's prepare_metadata_for_build_wheel throws subprocess.CalledProcessError instead of something nice

Pep517HookCaller's prepare_metadata_for_build_wheel throws subprocess.CalledProcessError if wheel is not installed, and that it throws an error in that case is of course expected, but wouldn't some sort of other error make more sense? I'm a bit afraid to catch this specific one now since this error seems extremely implementation-specific (related to how it's internally called) rather than problem-descriptive (telling me that wheel is missing), but I don't want to catch Exception either. So I'm not sure how to best catch this sort of problem...?

Expose command to build package metadata

In pypa/packaging-problems#224 and #19, @Jonast and I are trying to work out how to use PEP 517 to query the package metadata for an unbuilt package. PEP 517 has an interface for prepare_metadata_for_build_wheel and a protocol for falling back to build_wheel to generate that metadata.

This package should present a command that will follow that protocol for producing such metadata from a source checkout, similar to what pep517.check does. It should probably present both as a command, leaving the metadata somewhere on the filesystem, and as an API - returning a path to the generated metadata.

No information on the README about the future of this package

This package looks very interesting, and I'd love to make use of it in a real non-testing project.

However, it is quite unclear if after the testing period 1. this will vanish entirely, 2. be available with the same name on pypi, 3. be available with a different name/part of something else on pypi, ...

Can you add some information to the README clarifying where this project is heading, and how much sense it makes to use this for more than just playing around at this stage?

pep517.check doesn't work under Py2

Not sure if this is intentional or not -- if it is maybe specifying a minimum Python version in the setup.py would be a nice outcome, but e.g. #51 makes me think there is intention that pep517.check works on Py2.

But, trying to use it results in:

~/Desktop
⊙  virtualenv venv; venv/bin/python -m pip install pep517; venv/bin/python -m pep517.check ~/Development/jsonschema                                                     julian@Air
Using real prefix '/usr/local/Cellar/pypy/7.1.1/libexec'
  No LICENSE.txt / LICENSE found in source
Path not in prefix '/Users/julian/.local/share/virtualenvs/venvs/include' '/usr/local/Cellar/pypy/7.1.1/libexec'
New pypy executable in /Users/julian/Desktop/venv/bin/python
Also creating executable in /Users/julian/Desktop/venv/bin/pypy
Installing setuptools, pip, wheel...
done.
Collecting pep517
/Users/julian/Desktop/venv/site-packages/pip/_vendor/msgpack/fallback.py:133: DeprecationWarning: encoding is deprecated, Use raw=False instead.
  unpacker = Unpacker(None, max_buffer_size=len(packed), **kwargs)
  Using cached https://files.pythonhosted.org/packages/a2/1f/d99c4cb3c46cfe0c356e884ef26a23dd73b5d17d2b4efca201e03e63cc28/pep517-0.6.0-py2.py3-none-any.whl
Collecting pytoml (from pep517)
  Using cached https://files.pythonhosted.org/packages/a5/47/c7f8a0f210ad18576840922e0b504f0b7f5f73aea4a52ab14c5b58517edf/pytoml-0.1.21-py2.py3-none-any.whl
Collecting importlib_metadata (from pep517)
  Using cached https://files.pythonhosted.org/packages/ad/aa/25fcbded2ab4ed4ff3071d1e000cd4f8f9c65653d2d7157dd105a8e81d42/importlib_metadata-0.19-py2.py3-none-any.whl
Collecting zipp (from pep517)
  Using cached https://files.pythonhosted.org/packages/74/3d/1ee25a26411ba0401b43c6376d2316a71addcc72ef8690b101b4ea56d76a/zipp-0.6.0-py2.py3-none-any.whl
Collecting contextlib2; python_version < "3" (from importlib_metadata->pep517)
  Using cached https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting configparser>=3.5; python_version < "3" (from importlib_metadata->pep517)
  Using cached https://files.pythonhosted.org/packages/ab/1a/ec151e5e703ac80041eaccef923611bbcec2b667c20383655a06962732e9/configparser-3.8.1-py2.py3-none-any.whl
Collecting pathlib2; python_version == "3.4.*" or python_version < "3" (from importlib_metadata->pep517)
  Using cached https://files.pythonhosted.org/packages/67/c6/4dbf5dfdbe1140cadf765c3896acc098578626c35721bc7d3eb35f6a8fc1/pathlib2-2.3.4-py2.py3-none-any.whl
Collecting more-itertools (from zipp->pep517)
  Using cached https://files.pythonhosted.org/packages/2f/9d/dcfe59e213093695f108508af1214cf9cd95cc5489e46877ec5cb56369e5/more_itertools-5.0.0-py2-none-any.whl
Collecting six (from pathlib2; python_version == "3.4.*" or python_version < "3"->importlib_metadata->pep517)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting scandir; python_version < "3.5" (from pathlib2; python_version == "3.4.*" or python_version < "3"->importlib_metadata->pep517)
Installing collected packages: pytoml, six, more-itertools, zipp, contextlib2, configparser, scandir, pathlib2, importlib-metadata, pep517
Successfully installed configparser-3.8.1 contextlib2-0.5.5 importlib-metadata-0.19 more-itertools-5.0.0 pathlib2-2.3.4 pep517-0.6.0 pytoml-0.1.21 scandir-1.10.0 six-1.12.0 zipp-0.6.0
Traceback (most recent call last):
  File "/usr/local/Cellar/pypy/7.1.1/libexec/lib-python/2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/pypy/7.1.1/libexec/lib-python/2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/julian/Desktop/venv/site-packages/pep517/check.py", line 202, in <module>
    main()
  File "/Users/julian/Desktop/venv/site-packages/pep517/check.py", line 175, in main
    enable_colourful_output()
  File "/Users/julian/Desktop/venv/site-packages/pep517/colorlog.py", line 113, in enable_colourful_output
    handler.setFormatter(LogFormatter())
  File "/Users/julian/Desktop/venv/site-packages/pep517/colorlog.py", line 81, in __init__
    curses.tparm(fg_color, code), "ascii")
TypeError: __new__() takes at most 2 arguments (3 given)

where the hidden line there is pep517.check trying to do str(foo, "ascii"), which does not work on Py2.

Callers need better control over backend output

From the PEP, "The build frontend may capture stdout and/or stderr from the backend". As a specific example, pip currently captures setuptools output and displays it only in debug mode or when an error occurs. Also, if not displaying output, a progress spinner is updated as lines of output are produced. Similar control will be needed under PEP 517.

So, the requirements are, to be able to (on a per-call basis) capture hook output (stdout and/or stderr - pip simply merges the two, which should be sufficient for most cases) and react to it in real time as it's generated line by line.

It's not immediately obvious what the best design here would be. My initial suggestion (not yet validated to confirm that it would satisfy pip's requirements) would be for the wrapper to support a context manager that allows temporarily using a different function to call the subprocess:

def my_runner(cmdline, cwd, env):
    # This is the default, we'd customise this as needed
    # Callers must supply a function that runs the command with the given CWD and env.
    # No return value.
    # Raise an exception if the subprocess errors, or the invocation fails.
    subprocess.check_call(cmdline, cwd, env)

with backend.subprocess_runner(my_runner):
    backend.build_wheel(...)

Does anyone have any comments, or alternative design ssuggestions?

Shouldn't build-backend be optional?

Test automation

Setting up the tests is a little cumbersome at the moment. You need to:

Create a virtualenv
Install the test dependencies (which aren't documented anywhere)
Run pytest by hand from the project directory
Repeat for as many versions of Python as you ant to test for

Ideally, a test automation tool should be used (a dev-requirements.txt file would also help with (2), and I assume that's not an issue to add). There are two that I know of:

Tox, by far the most commonly used/known
Nox, a newcomer that is more flexible, but has some rough edges still (see wntrblm/nox#93 in particular)

The biggest problem with tox is that it expects a setup.py file, which flit doesn't use. I have a configuration for tox that works, but it needs a dummy (empty) setup.py added to the project.

What are your thoughts? Until nox can better handle missing Python versions, I'm in favour of tox. But I don't want to get into a philosophical debate over setup.py and alternative build systems, I just want something that lets me run the tests locally without jumping through hoops :-(

Add release instructions

Allow calling hooks from a custom Python interpreter

I want to call the hooks from a Python interpreter different from sys.executable. The use case would be to run it in an isolated environment.

Either allowing me to configure the executable in Pep517HookCaller or replacing sys.executable with python, so that I can control the executable via PATH, would be fine.

Need a PyPI release

In order to integrate with pip's vendoring mechanism, we need to be able to pip install this package. Ultimately, this means a pip release. In the short term, we can't vendor from github, because prior to the adoption of PEP 517, pip can only install setup.py based projects from source...

This is a short term issue during the development of pip's PEP 517 support, but I wanted to note it here so that it doesn't get forgotten.

Restructure this library to better separate low-level and high-level features

People seem to regularly expect this library to be a higher-level tool than it is. As a result, we get requests for functionality that we don't want to support, and users are disappointed that features that do exist don't meet their expectations.

I suggest that this library should be reorganised, and potentially split up, to better reflect different features.

A very low-level library, that has no external dependencies, that just handles calling backend hooks. Essentially, this would just be the wrappers.py file and support modules.
A slightly higher level library that depends on a particular toml library, and parses pyproject.toml and calls the backend. It should probably also return the build dependencies. I'm not 100% sure this is useful, but it would satisfy the needs of people wanting to manage dependencies themselves, but still be able to call something that does all the steps of building a project - see for example #81.
Maybe a build environment management library - but my feeling is that getting such a thing right is pretty complex. I'd be inclined to say that if we were to provide a library interface to build environment management, it should be based on pip's implementation, not on the existing code.
A command line tool doing what is currently pep517.build, etc. Personally, this isn't something I particularly want to touch, but the growing popularity of pep517.build suggests that there's a demand for this.

I'd suggest that this project focus on being a pure library, covering (1) and (2) initially, with (3) a longer-term goal.

The command line functionality can be spun off into an independent project. I'd argue that the current build environment management code should probably go with it - it's not really robust enough to be a usable library at the moment, but it's needed for the command line tools.

Wrappers have to be run from a filesystem

Because the _in_process.py script is located via __file__, the wrappers module won't work from within a zipfile (or other PEP 302 loader). Ideally, that should be fixed (pip runs itself from within a zipfile when run via get-pip.py, although it shouldn't be installing anything from source, so it's not clear that it needs PEP 517 support in that scenario).

Just adding this as a reminder for now because I'm not sure what a good solution would be (the wrapper script may be too long to be passed to Python via -c).

TypeError: init() missing 1 required positional argument: 'build_backend'

I tried some of the example code in the readme, but I get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jonas/e/python-for-android/pythonforandroid/pythonpackage.py", line 26, in get_dependencies_from_package_folder
    from pep517.wrappers import Pep517HookCaller
ModuleNotFoundError: No module named 'pep517'
>>> get_dependencies_from_package_folder("/home/e/Develop/testproject/")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/e/Develop/python-for-android/pythonforandroid/pythonpackage.py", line 28, in get_dependencies_from_package_folder
    hooks = Pep517HookCaller(path)
TypeError: __init__() missing 1 required positional argument: 'build_backend'

Is the constructor for Pep517HookCaller now different than in the README? If yes, an updated usage example would be very useful

setup.py in PyPI tarball fails flake8 test

Running pytest inside the pypi release tarball fails with flake8 errors in setup.py:

============================= test session starts ==============================
platform linux -- Python 3.7.1, pytest-4.0.2, py-1.7.0, pluggy-0.8.0
rootdir: /build/python-pep517/src/pep517-0.5.0, inifile: pytest.ini
plugins: flake8-1.0.2
collected 31 items

install-pip-master.py .                                                  [  3%]
setup.py F                                                               [  6%]
pep517/__init__.py .                                                     [  9%]
pep517/_in_process.py .                                                  [ 12%]
pep517/build.py .                                                        [ 16%]
pep517/check.py .                                                        [ 19%]
pep517/colorlog.py .                                                     [ 22%]
pep517/compat.py .                                                       [ 25%]
pep517/envbuild.py .                                                     [ 29%]
pep517/wrappers.py .                                                     [ 32%]
tests/__init__.py .                                                      [ 35%]
tests/test_call_hooks.py .........                                       [ 64%]
tests/test_envbuild.py ...                                               [ 74%]
tests/test_hook_fallbacks.py ....                                        [ 87%]
tests/samples/buildsys_pkgs/buildsys.py .                                [ 90%]
tests/samples/buildsys_pkgs/buildsys_minimal.py .                        [ 93%]
tests/samples/pkg1/pkg1.py .                                             [ 96%]
tests/samples/pkg2/pkg2.py .                                             [100%]

=================================== FAILURES ===================================
_________________________________ FLAKE8-check _________________________________
/build/python-pep517/src/pep517-0.5.0/setup.py:7:1: E122 continuation line missing indentation or outdented
/build/python-pep517/src/pep517-0.5.0/setup.py:10:1: E122 continuation line missing indentation or outdented
/build/python-pep517/src/pep517-0.5.0/setup.py:13:1: E122 continuation line missing indentation or outdented
/build/python-pep517/src/pep517-0.5.0/setup.py:24:6: E124 closing bracket does not match visual indentation

===================== 1 failed, 30 passed in 3.60 seconds ======================

typeerror init() missing 1 required positional argument

Problem installing package in an isolated environment

I'm trying to install a package in an AWS instance which is not accessible to the pypi.org.
I had to upload it to a private repository using s3pypi. (With all dependencies of course)

when trying to install said package, this is the stack trace I get:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in main
    status = self.run(options, args)
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 345, in run
    resolver.resolve(requirement_set)
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 196, in resolve
    self._resolve_one(requirement_set, req)
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 359, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 307, in _get_abstract_dist_for
    self.require_hashes
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 215, in prepare_linked_requirement
    finder, self.build_isolation,
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/distributions/source.py", line 70, in prepare_distribution_metadata
    reqs = self.req.pep517_backend.get_requires_for_build_wheel()
  File "/opt/conda/lib/python3.7/site-packages/pip/_vendor/pep517/wrappers.py", line 71, in get_requires_for_build_wheel
    'config_settings': config_settings
  File "/opt/conda/lib/python3.7/site-packages/pip/_vendor/pep517/wrappers.py", line 162, in _call_hook
    raise BackendUnavailable
pip._vendor.pep517.wrappers.BackendUnavailable

Is there maybe a way to work around this?
Thanks

Add --quiet options to build and check

It'd be nice, particularly for pep517.check, to selectively show less noisy output (e.g. simply whether things were successfully built).

Release 0.3

I'm going to release a version 0.3, to fix a couple of issues I encountered in pip.

Steps (as far as I recall):

Edit pep517/__init__.py to update the version, and commit the change.
git tag 0.3
git push --tags
flit publish
Create a release on github from tag 0.3

Shout if I forgot anything, otherwise I'll do a release tonight.

Add type hints to public API

Hey, would it be possible to add type hints to public API? Thanks.

Async hook wrappers?

The wrappers to call PEP 517 hooks in a subprocess are completely synchronous: they write files, start a subprocess, wait for that to exit, and read output files before returning. For many applications, I expect that is what we want.

This issue is to collect info on whether there is a need for machinery to call hooks asynchronously, and if so whether this package should provide the machinery or some lower-level pieces to build it out of.

If you want to call PEP 517 hooks asynchronously, please comment here, but start building the machinery you need as part of your own project. We probably won't add async parts to this package before we have a couple of potential users to figure out what code it's useful to share.

No way of passing pip arguments when creating an environment

When creating an environment, pip is invoked to install the build requirements. But there is no way of passing custom arguments for pip, to control how packages are downloaded (for example --extra-index-url, --find-links, --proxy, --no-index, --cert, --trusted-host, ...) Technically it would be possible to set the equivalent environment variables (PIP_*) but that runs the risk of overriding any environment variables the user might have set (which may be a good thing or a bad thing, it's not immediately clear).

My original need for this was to set --find-links when writing tests, to load a dummy backend from the test directory, but the general problem will need to be addressed if the library is to be used by pip, to copy over the options passed to pip.

This probably needs some discussion - it's not obvious what the best way of handling this would be, and pip has rather a lot of options that are potentially relevant here. It's possible that the best solution would require changes to pip, as well.

hooks.get_requires_for_build_wheel may return pep 508 url, but BuildEnvironment.pip_install() cannot process that

The hooks.get_requires_for_build_wheel function may return a pep 508 url if that is used in install_requires() (which works completely fine outside of pep517 usage, I can install this package just fine), but BuildEnvironment.pip_install() cannot process that. However, this is exactly how envbuild.py uses things:

...
with BuildEnvironment() as env:
        env.pip_install(requires)
        reqs = hooks.get_requires_for_build_wheel(config_settings)
        env.pip_install(reqs)

And this will lead to the following error:

Requirement 'e @ https://github.com/e/e/archive/master.zip' looks like a filename, but the file does not exist
Processing ./e @ https:/github.com/e/e/archive/master.zip
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/home/e/Develop/python-for-android/e @ https:/github.com/e/e/archive/master.zip'

So it looks to me like either envbuild.py or pip are broken here

Document interface of subprocess_runner callback

Carried over from #52 (cc @pradyunsg): the machinery for calling the hooks takes an optional 'subprocess runner'. We should document what is expected of this - what arguments it should accept, and its behaviour when the subprocess fails.

Part of #13.

add console script entry point for build

Without this pipx cannot install it sadly 👍

AttributeError: 'Pep517HookCaller' object has no attribute 'build_sys_requires'

To reproduce:

$ git clone https://github.com/takluyver/flit.git
...

$ python -m pep517.check flit
Traceback (most recent call last):
  File "pep517/check.py", line 154, in check
    sdist_ok = check_build_sdist(hooks)
  File "check.py", line 24, in check_build_sdist
    env.pip_install(hooks.build_sys_requires)
AttributeError: 'Pep517HookCaller' object has no attribute 'build_sys_requires'

I believe check_build_wheel also contains this non-existent attribute reference.

Running pep517.build (only with setuptools?) on PyPy does not fetch the backend / fails with BackendUnavailable

Full reproducer:

somepython=pypy
rm -rf venv pep517-venv foo && \
"$somepython" ~/Development/virtualenv/virtualenv.py --quiet venv && \
venv/bin/python -m pip install --quiet mkpkg && \
venv/bin/mkpkg foo && \
venv/bin/python ~/Development/virtualenv/virtualenv.py --no-setuptools pep517-venv && \
pep517-venv/bin/python -m pip install --quiet pep517 && \
pep517-venv/bin/python -m pep517.build foo

fails with:

  No LICENSE.txt / LICENSE found in source
/Users/julian/Desktop/venv/site-packages/pip/_vendor/msgpack/fallback.py:133: DeprecationWarning: encoding is deprecated, Use raw=False instead.
  unpacker = Unpacker(None, max_buffer_size=len(packed), **kwargs)
Set up CI at: https://travis-ci.com/Julian/foo
Using real prefix '/usr/local/Cellar/pypy/7.2.0/libexec'
  No LICENSE.txt / LICENSE found in source
Path not in prefix '/Users/julian/Desktop/venv/include' '/usr/local/Cellar/pypy/7.2.0/libexec'
New pypy executable in /Users/julian/Desktop/pep517-venv/bin/python
Also creating executable in /Users/julian/Desktop/pep517-venv/bin/pypy
Installing pip, wheel...
done.
/Users/julian/Desktop/pep517-venv/site-packages/pip/_vendor/msgpack/fallback.py:133: DeprecationWarning: encoding is deprecated, Use raw=False instead.
  unpacker = Unpacker(None, max_buffer_size=len(packed), **kwargs)
No handlers could be found for logger "pep517.envbuild"
Traceback (most recent call last):
  File "/usr/local/Cellar/pypy/7.2.0/libexec/lib-python/2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/pypy/7.2.0/libexec/lib-python/2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/julian/Desktop/pep517-venv/site-packages/pep517/build.py", line 124, in <module>
    main(parser.parse_args())
  File "/Users/julian/Desktop/pep517-venv/site-packages/pep517/build.py", line 120, in main
    build(args.source_dir, dist, args.out_dir)
  File "/Users/julian/Desktop/pep517-venv/site-packages/pep517/build.py", line 88, in build
    _do_build(hooks, env, dist, dest)
  File "/Users/julian/Desktop/pep517-venv/site-packages/pep517/build.py", line 61, in _do_build
    reqs = get_requires({})
  File "/Users/julian/Desktop/pep517-venv/site-packages/pep517/wrappers.py", line 213, in get_requires_for_build_sdist
    'config_settings': config_settings
  File "/Users/julian/Desktop/pep517-venv/site-packages/pep517/wrappers.py", line 265, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
BackendUnavailable

which seems to only fail on PyPy, because on CPython changing somepython=python3.6 and rerunning seems to succeed.

For ease, the (generated) pyproject.toml in that stub package is:

[build-system]
requires = [
    # The minimum setuptools version is specific to the PEP 517 backend,
    # and may be stricter than the version required in `setup.py`
    "setuptools>=40.6.0",
    "setuptools_scm",
    "wheel",
]
build-backend = "setuptools.build_meta"

is .bumpversion.cfg used?

I was poking around the codebase and noticed that .bumpversion.cfg says the current version is 0.5.0, but there's been two releases since then.

Is bumpversion still being used?

Default Pep517HookCaller subprocess runner not reverted on exception

If the scope wrapped by Pep517HookCaller.subprocess_runner raises an exception then the overridden _subprocess_runner will not get reverted back to the default. We should have a try: ... finally: ... around the yield.

Unable to install from local checkout

In troubleshooting other issues, I'm trying to install from a local checkout of this project, but using pip 18.1, the project doesn't install:

$ pip install ~/p/pypa/pep517
Processing /Users/jaraco/p/pypa/pep517
  Installing build dependencies ... done
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tokenize.py", line 447, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-g1rb926g/setup.py'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-g1rb926g/

The triple verbose output is here.

Better handling of errors in backends

If a backend raises an unexpected exception, the subprocess being run by the wrapper will exit with an error, which will get reported back to the caller, but it's not as informative as it could be. We should look into capturing the actual error and traceback, and re-raising it in the calling process.

This is a particular issue when debugging pip issues, as it's hard to locate or identify what the backend might be doing, with all the layers of capturing subprocess output involved (particularly in pip's CI, where tox and pytest add extra layers).

Just noting this here for now, as I don't have any immediate ideas on how to do this, and while it's frustrating while I'm developing the PEP 517 interface for pip, it's not exactly an urgent issue.

Master installs as 0.2 despite 0.3 release

Installing the code in master results in pep517==0.2.

pep517 master $ pip install .
Processing /Users/jaraco/code/public/pypa/pep517
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting pytoml (from pep517==0.2)
Building wheels for collected packages: pep517
  Building wheel for pep517 (PEP 517) ... done
  Stored in directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-ephem-wheel-cache-87af9797/wheels/49/cf/b4/c8c7c92c61ca20ee053675caa65bc02da8212c078d75ea49e1
Successfully built pep517
Installing collected packages: pytoml, pep517
Successfully installed pep517-0.2 pytoml-0.1.20

I would have expected 0.3+post or 0.4dev or similar.

sdist cannot be installed on Python 2

Attempting to install pep517 from an sdist on Python 2.7 results in an error:

~ $ python2.7 -m pip install --no-binary=:all: pep517                                                                                                                          
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting pep517
  Using cached https://files.pythonhosted.org/packages/79/27/1e6a8ba83dab2b243e1020e340552c129b0076fbb5518e0c2a9c44c7d056/pep517-0.5.0.tar.gz
  Installing build dependencies ... error
  Complete output from command /usr/local/opt/python@2/bin/python2.7 /usr/local/lib/python2.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-sq1HqN/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- flit:
  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
  Collecting flit
    Could not find a version that satisfies the requirement flit (from versions: )
  No matching distribution found for flit
  
  ----------------------------------------
Command "/usr/local/opt/python@2/bin/python2.7 /usr/local/lib/python2.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-sq1HqN/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- flit" failed with error code 1 in None

Automate releases of tagged commits through CI

Not obvious if `get_requires_for_build_wheel` will modify installed packages state, or how - and other questions

I am very interested in the following code, since it is what I want to use in my project:

import os
from pep517.wrappers import Pep517HookCaller

src = 'path/to/source'  # Folder containing 'pyproject.toml'
hooks = Pep517HookCaller(src)
print(hooks.build_sys_requires)  # List of static requirements

config_options = {}   # Optional parameters for backend
# List of dynamic requirements:
print(hooks.get_requires_for_build_wheel(config_options))

However, I have a couple of questions, which I suggest should be possibly answered in the README:

What happens when the package doesn't provide the new functionality to get this metadata without building the wheel? If it builds the wheel in that case, can I possibly configure it to error out instead?
Can I somehow prevent any sort of state modification outside of the package dir (installed site packages) by this command? Will it do such modifications ever at all, e.g. when building the wheel as fallback or in other situations. e.g. to satisfy setup_requires?
If the package examined with this command is already installed in the site-packages, will this in any way interfere with obtaining the information from it in the exact version as found at the src path?

AttributeError: 'Pep517HookCaller' object has no attribute 'build_sys_requires'

I'm trying to build a setuptools project using pep517 to avoid depending on setup_requires and easy_install for the build dependencies.

When I do, I get the error.

jaraco.mongodb master $ cat pyproject.toml
[build-system]
requires = ["setuptools>=34.4", "wheel", "setuptools_scm>=1.15"]
build-backend = 'setuptools.build_meta'
jaraco.mongodb master $ pip-run pep517 -- -m pep517.check .
Collecting pep517
  Using cached https://files.pythonhosted.org/packages/24/ec/e506e3bfdde3ef449d8fe6de7c0ff0f3292d6cc071f676d75dc8d5f98375/pep517-0.3-py2.py3-none-any.whl
Collecting pytoml (from pep517)
Installing collected packages: pytoml, pep517
Successfully installed pep517-0.3 pytoml-0.1.20
Found pyproject.toml                                                                                                                                                           I-__main__
Loaded pyproject.toml                                                                                                                                                          I-__main__
Temporary build environment: /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pep517-build-env-ep4t2o2k                                                                 I-pep517.envbuild
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-shftz686/pep517/check.py", line 194, in <module>
    main()
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-shftz686/pep517/check.py", line 173, in main
    ok = check(args.source_dir)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-shftz686/pep517/check.py", line 154, in check
    sdist_ok = check_build_sdist(hooks)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-shftz686/pep517/check.py", line 24, in check_build_sdist
    env.pip_install(hooks.build_sys_requires)
AttributeError: 'Pep517HookCaller' object has no attribute 'build_sys_requires'

I don't see mention of that hook in PEP 517. What am I missing?

System packages leak into build environment

PEP 517 recommends that the build environment should only contain the standard library and packages from build-system.requires. However, system packages are exposed to the build environment using pep517. This behaviour has been implicated in jaraco/skeleton#12.

For an MRE:

$ python -m pip list
Package Version
------- -------
pep517  0.8.1
pip     20.0.1
toml    0.10.0
$ cat pyproject.toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
$ cat setup.py
import toml   # Does not raise

from setuptools import setup

setup(name='test')
$ python -m pep517.build --source .
running egg_info
creating test.egg-info
writing test.egg-info/PKG-INFO
writing dependency_links to test.egg-info/dependency_links.txt
writing top-level names to test.egg-info/top_level.txt
writing manifest file 'test.egg-info/SOURCES.txt'
reading manifest file 'test.egg-info/SOURCES.txt'
writing manifest file 'test.egg-info/SOURCES.txt'
running sdist
running egg_info
writing test.egg-info/PKG-INFO
writing dependency_links to test.egg-info/dependency_links.txt
writing top-level names to test.egg-info/top_level.txt
reading manifest file 'test.egg-info/SOURCES.txt'
writing manifest file 'test.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md

running check
warning: check: missing required meta-data: url

warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied

creating test-0.0.0
creating test-0.0.0/test.egg-info
copying files to test-0.0.0...
copying pyproject.toml -> test-0.0.0
copying setup.py -> test-0.0.0
copying test.egg-info/PKG-INFO -> test-0.0.0/test.egg-info
copying test.egg-info/SOURCES.txt -> test-0.0.0/test.egg-info
copying test.egg-info/dependency_links.txt -> test-0.0.0/test.egg-info
copying test.egg-info/top_level.txt -> test-0.0.0/test.egg-info
Writing test-0.0.0/setup.cfg
Creating tar archive
removing 'test-0.0.0' (and everything under it)

ERROR: Double requirement given: wheel>=0.34.2 (already in wheel, name='wheel')

Hi guys, thank you very much for all the work you have put on this.
I run into an weird error this morning while trying to build a project with pep517, so I tried to recreate a reduced version that demonstrates the error, to report the issue.

It seems that pep517 tries to include wheel in the build environment but does not reconcile the version when the user specifies a range in setup.cfg :: options.setup_requires or pyproject.toml :: build-system.requires.

This is the error message I am having:

ERROR: Double requirement given: wheel>=0.34.2 (already in wheel, name='wheel')
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/example/.venv/lib/python3.6/site-packages/pep517/build.py", line 124, in <module>
    main(parser.parse_args())
  File "/tmp/example/.venv/lib/python3.6/site-packages/pep517/build.py", line 120, in main
    build(args.source_dir, dist, args.out_dir)
  File "/tmp/example/.venv/lib/python3.6/site-packages/pep517/build.py", line 88, in build
    _do_build(hooks, env, dist, dest)
  File "/tmp/example/.venv/lib/python3.6/site-packages/pep517/build.py", line 64, in _do_build
    env.pip_install(reqs)
  File "/tmp/example/.venv/lib/python3.6/site-packages/pep517/envbuild.py", line 103, in pip_install
    stderr=LoggerWrapper(log, logging.ERROR),
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/tmp/example/.venv/bin/python', '-m', 'pip', 'install', '--ignore-installed', '--prefix', '/tmp/pep517-build-env-24brzdll', 'wheel', 'setuptools_scm>=4.1.2', 'wheel>=0.34.2']' returned non-zero exit status 1.

And these are the files to reproduce the test:

# pyproject.toml
[build-system]
requires = ["setuptools>=46.1.0", "wheel>=0.34.2", "setuptools_scm[toml]>=4.1.2"]
build-backend = "setuptools.build_meta"

[tool.setuptools_scm]
version_scheme = "post-release"

# setup.cfg
[metadata]
name = example
description = example
platforms = any

[options]
zip_safe = False
packages = find:
include_package_data = True
package_dir =
    =src
setup_requires =
    setuptools_scm>=4.1.2
    wheel>=0.34.2

[options.packages.find]
where = src
exclude =
    tests

[bdist_wheel]
universal = 1

# setup.py
from setuptools import setup

if __name__ == "__main__":
    setup(use_scm_version={"version_scheme": "post-release"})

# src/example.py
def main():
    print("hello world")

if __name__ == "__main__":
    main()

After creating the files I did:

git init .
git add .
git commit -a -m "initial commit'
virtualenv .venv
.venv/bin/pip install -U pip setuptools  pep517
.venv/bin/python -m pep517.build .

Please let me know if you need any other information, or if it is more appropriate to report this issue elsewhere.

Option to not fallback for `wrappers.prepare_metadata_for_build_wheel`

Currently prepare_metadata_for_build_wheel builds a wheel if the hook is not implemented by the given backend.

It would be nice to be able to disable the automatic fallback behavior. The specific use cases are:

More consistent user experience - a package that uses a backend that does not implement preparing_metadata_for_build_wheel may show "Preparing wheel metadata" for a long time and "Building wheel for (PEP 517)" for a short time. Another package that does implement preparing_metadata_for_build_wheel would show the opposite behavior.
When refactoring pip, we would like to align the actual state of the project on disk with the type representing that project in memory. See pypa/pip#6607 (comment) for reference. Specifically, a "Local modern project" may have its metadata queried but if that fails then we want to know about it and explicitly advance the state to a "Local wheel" by building a wheel. Mocking this behavior for unit tests is very straightforward, but those sorts of tests get further from the truth if the library is keeping a wheel off to the side.

pytoml dependency is unmaintained

According to pyproject.toml, this library depends on pytoml.

However, pytoml is explicitly unmaintained and lacks support for dotted keys.

It'd be great if this switched a still-supported toml library. I've found uiri/toml works well.

Context on how I discovered this:

When using bork (which relies on this library), I tried putting this in pyproject.toml:

[tool.bork]
zipapp.enabled = true
zipapp.main = "bork.cli:main"

but it results in a hilariously vague error message of pyproject.toml(8, 1): msg (that's it).

I wound up having to do this instead:

[tool.bork.zipapp]
enabled = true
main = "bork.cli:main"

typeerror init() missing 1 required positional argument 'value' tkinter

Expose command to build packages

Currently, there's pep517.check that checks that a package can successfully build sdist and wheels for a package. Similar to that, it should be possible to actually build a package, either the sdist or the wheel or both... something that a CI tool could use to build the package prior to using twine to upload it. This would address pypa/packaging-problems#219.

we need a way to specify the use of a custom package index for a build dependency

Given the following pyproject.toml

[build-system]
requires = [
    "setuptools",
    "custom_buildpackage"
]
build-backend = "setuptools.build_meta"

how do I specify that custom_buildpackage (and only it) is to be installed from a package-index other than the one at pypi.org?

I know that these are options:
1.) Set environment variables for pip to use an --extra-index-url or --index-url
2.) Set the (extra-)index in ~/.config/pip.conf
3.) Install the package beforehand from the other index

pep517.wrappers.BackendUnavailable doesn't actually print out which backend

I just ran into this error in travis:

E               Traceback (most recent call last):
E                 File "<string>", line 9, in <module>
E                 File "/home/travis/build/kivy/python-for-android/.tox/py3/lib/python3.5/site-packages/pythonforandroid/pythonpackage.py", line 444, in _extract_metainfo_files_from_package_unsafe
E                   reqs = hooks.get_requires_for_build_wheel({})
E                 File "/home/travis/build/kivy/python-for-android/.tox/py3/lib/python3.5/site-packages/pep517/wrappers.py", line 71, in get_requires_for_build_wheel
E                   'config_settings': config_settings
E                 File "/home/travis/build/kivy/python-for-android/.tox/py3/lib/python3.5/site-packages/pep517/wrappers.py", line 162, in _call_hook
E                   raise BackendUnavailable
E               pep517.wrappers.BackendUnavailable

But now I feel left guessing. Which backend? I'm guessing setuptools, but it would be extremely helpful if the error could actually print that out

pep517.build warns about missing author_email field

I ported a very old package to Python 3, of which I know the author, but not his email address. Also, I'd think he does not want to get involved any more with this 16 year old package.

In order to create a package I followed Hynek's guide at
https://hynek.me/articles/sharing-your-labor-of-love-pypi-quick-and-dirty/

My setup.py contains an author, but no author_email field and me as maintainer.

When I do a python -m pep517.build . I get the following warning:

warning: Check: missing meta-data: if 'author' supplied, 'author_email' must be supplied too

There was a quick discussion over at
https://discuss.python.org/t/which-fields-are-required-for-a-setup-py-especially-is-author-required/2705/2
with different proposed solutions.

I created a test package and uploaded it at test.pypi.org with the author field, but without an author_email field - looks totally valid to me:
https://test.pypi.org/project/Products.ZopeTree/2.0.2.dev0/

Where is the information coming from that author_email must be supplied - while it need not as it seems.

Is it possible to drop this warning? Or change its wording to "should be provided"?

If it is a "must" requirement I'd expect the build to fail.

Build/check should fallback to setuptools unconditionally

In this comment, @gaborbernat suggests that the project should unconditionally fallback to setuptools if PEP 517 is not indicated.

That seems like a plausible suggestion, but I'd like to separate that concern from the concern in the PR.

Set up webhook for RTD

I've just set up docs on Readthedocs: https://pep517.readthedocs.io/en/latest/index.html

However, I don't have permission to create the webhook for new commits to be built automatically. Please could someone with the right access go to https://readthedocs.org/dashboard/pep517/integrations/135801/ and click the 'Resync' button to wire it up?

Thanks!

Moving users away using pep517 as a CLI tool

This is the issue for tracking our plans for nudging users to transition away from using pep517 from the CLI. These changes would go into the inc-pep517.build branch and not the default branch.

Revert #83 (see #96)
Point to python-build in the README
{maybe} Point to python-build from issue templates
Add a deprecation warning when pep517.build is used
Add a deprecation warning when pep517.checkis used
Release a version with the warning - possibly after a bit of a wait, to give people a chance to start moving to python-build and flush out any bugs there.

Originally posted by @takluyver in #83 (comment) and edited by @pradyunsg

pep517 should either have wheel as an install dependency, or provide methods to get package dependencies without the wheel tool

pep517 should either have wheel as an install dependency, or provide methods to get package dependencies without the wheel tool. Right now, all the examples in the README don't work properly (except maybe hooks.build_sys_requires but that's broken #14 and I'm not sure what it's even supposed to do) without wheel installed.

This is why pep517 really should install wheel:

Basically, my package builds & installs just fine right now.
The promise of pep517 is that for a properly defined package, I should be able to get metadata (that's the point of the project if I got that right)
Therefore, if I install pep517 and clearly have a working package, I should be able to get its dependencies

This doesn't seem to be possible unless I manually install wheel in addition, which I don't think I should be required to do

pypa / pyproject-hooks Goto Github PK

pyproject-hooks's Introduction

pyproject-hooks

pyproject-hooks's People

Contributors

Stargazers

Watchers

Forkers

pyproject-hooks's Issues

Recommend Projects

Recommend Topics

Recommend Org

`pyproject-hooks`