Giter Club home page Giter Club logo

rez-pip's Introduction

My name is Jean-Christophe Morin and I'm a software developer and DevOps with lots of interests in builds, tests, CI and packaging (and other things too).

People usually call me JC or Jean-Christophe. The majority of people prefer JC since it's a lot shorter. Christophe is not optional, so it’s either JC or Jean-Christophe, but not Jean or Christophe.

Where to find me

You can find me here on GitHub (right here) and also on https://slack.aswf.io. These are the two places where I'm the most active.

  • LinkedIn: For professional purposes. I don't monitor it actively.
  • Email: This is the easiest way to contact me if you are not on the ASWF's Slack Workspace. I read all emails that I receive.
  • ASWF related things, you can also talk to me on Slack at https://slack.aswf.io. I'm active there, so I usually respond quickly.

Projects

Contributions

Here is a non exhaustive list of projects/repositories I've contributed to (ordered by first contribution, from newest to oldest, where applicable).

rez-pip's People

Contributors

dependabot[bot] avatar jeanchristophemorinperso avatar mrlixm avatar rez-pip-update-bot[bot] avatar supervalou avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

ennosch zachlewis supervalou mrlixm

rez-pip's Issues

Installs for multiple pythons?

When I issue this command:

rez-pip2 --pip <path/to/pip.pyz> --python-version 3.9 pyRXP==3.0.1

what happens is that it seems to want to go through all the versions of 3.9 that are installed, so I get

Installing requested packages for Python 3.9.13
...
Installing requested packages for Python 3.9.16
...
etc

Is this expected? Is there a way to not have it do this?

rez-pip should cleanly fail if resolving a python package fails

See https://academysoftwarefdn.slack.com/archives/C0321B828FM/p1697125828936919?thread_ts=1697123068.952239&cid=C0321B828FM.

Seems my issue is that I had an issue that caused Rez to be able to find a "pip" package, but when performing the resolve, it broke. There doesn't appear to be any sort of check in the code to confirm that the ResolvedContext.status == ResolverStatus.solved

This applies to the original rez-pip, but it also applies to us.

I don't know in which case a solve would fail for such simple requests, but we should support failures just in case. See

rez-pip/src/rez_pip/rez.py

Lines 246 to 262 in 9c61a5d

for package in packages:
resolvedContext = rez.resolved_context.ResolvedContext(
[f"{package.name}=={package.version}"]
)
# Make sure that system PATH doens't interfere with the "which" method.
resolvedContext.append_sys_path = False
for trimmedVersion in map(package.version.trim, [2, 1, 0]):
path = resolvedContext.which(f"python{trimmedVersion}", parent_environ={})
if path:
pythons[str(package.version)] = pathlib.Path(path)
break
else:
_LOG.warning(
f"Failed to find a Python executable in the {package.qualified_name!r} rez package"
)
.

rez-pip fails due to Windows short file names

When using rez-pip to install a package on Windows, the installation fails.

Environment

  • Windows 10.0
  • rez-pip 0.2.0
  • rez-pip being installed in its own virtual environment, not part of a rez installation

To Reproduce

  • Rezify a python 3.7.7 interpreter through this utility
  • Install rez-pip into its own virtualenv
  • open a terminal
  • run cd C:\rez-pip\Scripts then .\activate to enter the virtualenv of rez-pip
  • run .\rez-pip2 urllib3
  • see the error:
(rez-pip) PS C:\rez-pip\Scripts> rez-pip2 urllib3
INFO     Installing requested packages for Python 3.7.7
INFO     Resolved 1 dependencies for python 3.7.7
INFO     Downloading...
INFO     Downloaded urllib3-2.0.6 to
         'C:\\Users\\SUPERV~1\\AppData\\Local\\Temp\\rez-pip-targetod9cqugz\\wheels\\urllib3-2.0.6-py3-none-any.whl'
         (123799 bytes)
INFO     Downloaded 1 wheels
INFO     Installing urllib3-2.0.6 wheel
INFO     Creating rez package for urllib3
INFO     Installing urllib3-2.0.6 []
Traceback (most recent call last):
  File "c:\users\supervalou\packages\python\3.7.7\platform-windows\arch-AMD64\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\supervalou\packages\python\3.7.7\platform-windows\arch-AMD64\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\rez-pip\Scripts\rez-pip2.exe\__main__.py", line 9, in <module>
  File "c:\rez-pip\lib\site-packages\rez_pip\cli.py", line 318, in run
    _run(args, pipArgs, pipWorkArea)
  File "c:\rez-pip\lib\site-packages\rez_pip\cli.py", line 219, in _run
    release=args.release,
  File "c:\rez-pip\lib\site-packages\rez_pip\rez.py", line 127, in createPackage
    pkg.pip["metadata"] = remainingMetadata
  File "c:\users\supervalou\packages\python\3.7.7\platform-windows\arch-AMD64\lib\contextlib.py", line 119, in __exit__
    next(self.gen)
  File "c:\rez-pip\lib\site-packages\rez\package_maker.py", line 237, in make_package
    make_root(variant_, root)
  File "c:\rez-pip\lib\site-packages\rez_pip\rez.py", line 74, in make_root
    path, srcAbsolute.relative_to(os.path.realpath(installedWheelsDir))
  File "c:\users\supervalou\packages\python\3.7.7\platform-windows\arch-AMD64\lib\pathlib.py", line 895, in relative_to
    .format(str(self), str(formatted)))
ValueError: 'C:\\Users\\supervalou\\AppData\\Local\\Temp\\rez-pip-targetod9cqugz\\installed\\3.7.7\\python\\urllib3\\__init__.py' does not start with 'C:\\Users\\SUPERV~1\\AppData\\Local\\Temp\\rez-pip-targetod9cqugz\\installed\\3.7.7'

Expected behavior
Program installs urllib3 as a rez package.

Actual behavior
The error above occurs and nothing gets installed.

Investigation
This clearly seems to be due to a string comparison between a windows short name (C:/Users/SUPERV~1/...) and its corresponding long name (C:/Users/supervalou/...).

Adding some os.realpath around this line may fix this issue.

rez-pip reinstalls numpy everytime 

(.venv) [jcmorin@arch01 rez-pip]$ REZ_PACKAGES_PATH=~/rez_packages:/tmp/asd rez-pip2 numpy -p /tmp/asd --python-version 3.9
INFO     Installing requested packages for Python 3.9.18                                                                                                                                 
INFO     Resolved 1 dependencies for python 3.9.18                                                                                                                                       
INFO     Downloading...                                                                                                                                                                  
INFO     Downloaded numpy-1.26.0 to '/tmp/rez-pip-targetf1fombqc/wheels/numpy-1.26.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl' (18185894 bytes)                          
INFO     Downloaded 1 wheels                                                                                                                                                             
INFO     Installing numpy-1.26.0 wheel                                                                                                                                                   
INFO     Creating rez package for numpy                                                                                                                                                  
INFO     Installing numpy-1.26.0 [platform-linux, arch-x86_64, python-3.9]                                                                                                               
INFO     Created 1 variants and skipped 0

                                                                                                                                          
(.venv) [jcmorin@arch01 rez-pip]$ REZ_PACKAGES_PATH=~/rez_packages:/tmp/asd rez-pip2 numpy -p /tmp/asd --python-version 3.9
INFO     Installing requested packages for Python 3.9.18                                                                                                                                 
INFO     Resolved 1 dependencies for python 3.9.18                                                                                                                                       
INFO     Downloading...                                                                                                                                                                  
INFO     Downloaded numpy-1.26.0 to '/tmp/rez-pip-targetxiwjbwt3/wheels/numpy-1.26.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl' (18185894 bytes)                          
INFO     Downloaded 1 wheels                                                                                                                                                             
INFO     Installing numpy-1.26.0 wheel                                                                                                                                                   
INFO     Creating rez package for numpy                                                                                                                                                  
INFO     Installing numpy-1.26.0 [platform-linux, arch-x86_64, python-3.9]                                                                                                               
INFO     Created 1 variants and skipped 0

I have no idea why it gets re-installed.

Add package name remapping functionality

In relation to #55, it would be handy to have a way to control the names of the packages that we create. This would probably be a mapping/dict either defined through a CLI, or a config setting.

We might also need a way to automatically prefix all package names?

Missing 'archive_info' while converting pip package causes rez-pip crash

When running rez-pip2 on ShotGrid's latest version of their tk-core package, the process crashes trying to get a archive_info key in the package's dictionary.

Environment

  • Windows 10
  • Rez version 2.112.0
  • Rez python version 3.7
  • rez-pip 0.3.1

To Reproduce

  1. Install new rez-pip
  2. rez-pip2 git+https://github.com/shotgunsoftware/[email protected] --python-version 3.7

Expected behavior
Successful install.

Actual behavior

(Rez) λ rez-pip2 git+https://github.com/shotgunsoftware/[email protected] --python-version 3.7
INFO     Installing requested packages for Python 3.7.9
Traceback (most recent call last):
  File "C:\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Rez\Scripts\rez-pip2.exe\__main__.py", line 7, in <module>
  File "c:\program files\rez\lib\site-packages\rez_pip\cli.py", line 318, in run
    _run(args, pipArgs, pipWorkArea)
  File "c:\program files\rez\lib\site-packages\rez_pip\cli.py", line 183, in _run
    pipArgs,
  File "c:\program files\rez\lib\site-packages\rez_pip\pip.py", line 138, in getPackages
    packageInfo = PackageInfo.from_dict(rawPackage)
  File "c:\program files\rez\lib\site-packages\dataclasses_json\api.py", line 70, in from_dict
    return _decode_dataclass(cls, kvs, infer_missing)
  File "c:\program files\rez\lib\site-packages\dataclasses_json\core.py", line 217, in _decode_dataclass
    infer_missing)
  File "c:\program files\rez\lib\site-packages\dataclasses_json\core.py", line 172, in _decode_dataclass
    field_value = kvs[field.name]
KeyError: 'archive_info'

Cyclic dependencies when installing sphinx.

Hello,

I am trying to install sphinx using rez-pip. The installation itself doesn't raise any error, but the result of that installation is not as expected and doesn't resolve to a proper rez environment.

steps to reproduce

We first assume the issue described in #55 has been fixed manually (which sphinx is subject to)

rez python -m rez_pip sphinx==7.2.6 --python-version ==3.10.11
rez env sphinx

produce the following

resolve failed, by lcoll@PC-Liam-Home, on Fri Jan 12 14:32:15 2024, using Rez v2.113.0

requested packages:
sphinx
~platform==windows  (implicit)
~arch==AMD64        (implicit)
~os==windows-10     (implicit)

The context failed to resolve:
A cyclic dependency was detected: sphinx-7.2.6 --> sphinxcontrib_applehelp-1.0.7 --> sphinx-7.2.6

reason

Looking at sphinx rez package produced we can see its requires :

requires = [
    'sphinxcontrib_applehelp',
    'sphinxcontrib_devhelp',
    'sphinxcontrib_jsmath',
    'sphinxcontrib_htmlhelp-2.0.0+',
    'sphinxcontrib_serializinghtml-1.1.9+',
    'sphinxcontrib_qthelp',
    'jinja2-3.0+',
    'pygments-2.14+',
    'docutils-0.18.1+<0.21',
    'snowballstemmer-2.0+',
    'babel-2.9+',
    'alabaster-0.7+<0.8',
    'imagesize-1.3+',
    'requests-2.25.0+',
    'packaging-21.0+',
    'python-3.9+'
]

Having a look at the first sphix extension sphinxcontrib_applehelp we can also check its requires :

requires = [
    'sphinx-5+',
    'python-3.9+'
]

which explain the cyclic dependency

suggested fix

I have the feeling that this is not a simple problem with probably a lot of pitfall.

EDIT: not a good idea
However I was thinking of a simple fix by using weak references on dependencies (package that have not been directly requested to pip).

I will try to apply this fix on my side and see how it works.

Cheers.
Liam.

Package names inconsistent between requirements and installed packages

See AcademySoftwareFoundation/rez#1545.

Basically, there is an inconsistency between the packages we install, and the packages that we put in the requirements. On top of the issue reported in the linked issue, the new rez-pip also has problems with uppercased vs lowercased packages.

Package name canonization is annoying... https://blog.piwheels.org/canonicalise-all-the-things/.

We can fix this issue by changing

rez-pip/src/rez_pip/utils.py

Line 39 in 9c61a5d

return name.replace("-", "_")
to always lower-case the name and also replace . with _.

To confirm that it really works, we can run REZ_PACKAGES_PATH=~/rez_packages rez-pip2 prefect==2.13.6 -p /tmp/asd --python-version 3.9 -- -v and then rez-env prefect and it should work.

We might want to also print a report at the end with a mapping between the original names and the new names if it's possible to do. We might also want to harmonize the names across the logs.

Unable to install sdist package

When using rez-pip to install the future package, the install fails.

Environment

  • Windows 10.0
  • rez-pip 0.2.0
  • rez-pip being installed in its own virtual environment, not part of a rez installation

To Reproduce

  • Rezify a python 3.7.7 interpreter through this utility
  • Install rez-pip into its own virtualenv
    • open a terminal
    • run rez env python-3.7.7 -- python -m venv C:\rez-pip to generate a virtualenv that will host the rez-pip package
    • run cd C:\rez-pip\Scripts then .\activate to enter the virtualenv
    • run python -m pip install rez-pip to install rez-pip in the virtualenv
    • close the terminal
  • open a terminal
  • run cd C:\rez-pip\Scripts then .\activate to enter the virtualenv of rez-pip
  • run .\rez-pip2.exe future
  • see the error:
> rez-pip2 future
INFO     Installing requested packages for Python 3.7.7
ERROR: Could not find a version that satisfies the requirement future (from versions: none)
ERROR: No matching distribution found for future
rez_pip.exceptions.PipError: Failed to run pip command: '.\\python.exe c:\\rez-pip\\lib\\site-packages\\rez_pip\\data\\pip.pyz install -q future --disable-pip-version-check --dry-run --ignore-installed --python-version=3.7.7 --only-binary=:all: --target=/tmp/asd --disable-pip-version-check --report C:\\<tmp_folder_path>\\pip-install-output9cm63vpe'

Pip reported this:

ERROR: Could not find a version that satisfies the requirement future (from versions: none)
ERROR: No matching distribution found for future

Expected behavior
Program installs future as a rez package (as future seems compatible with python 3.7.7).

Actual behavior
The error above occurs and nothing gets installed at all. When having the error with python-3.7.7, no further python version (like 3.9.7, 3.10.8, etc.) gets attempted by rez-pip in its loop.

Investigation
This is probably a pip-trick issue. This doesn't work:

(rez-pip) PS C:\rez-pip\Scripts> .\python.exe c:\rez-pip\lib\site-packages\rez_pip\data\pip.pyz install future --disable-pip-version-check --dry-run --ignore-installed --python-version=3.7.7 --only-binary=:all: --target=/tmp/

ERROR: Could not find a version that satisfies the requirement future (from versions: none)
ERROR: No matching distribution found for future

but this seems to be okay:

(rez-pip) PS C:\rez-pip\Scripts> .\python.exe c:\rez-pip\lib\site-packages\rez_pip\data\pip.pyz install future --disable-pip-version-check --dry-run --ignore-installed --target=/tmp/

Collecting future
  Using cached future-0.18.3-py3-none-any.whl
Would install future-0.18.3

I'm not too familiar with the --python-version and --only-binary args of pip yet so I'll probably add more comments with further investigation.

Fail to install boxsdk

When running a rez-pip2 install on the boxsdk PyPi package, the install fails with a UnicodeDecodeError.

Environment

  • Windows
  • Rez version 2.112.0
  • Rez python version 3.7
  • rez-pip 0.3.0

To Reproduce

  1. Install new rez-pip
  2. rez-pip2 boxsdk==3.9.2 --python-version 3.7 (seems to fail with at least all 3.x.x versions)

Expected behavior
Successful install.

Actual behavior

INFO     Installing requested packages for Python 3.7.9
Traceback (most recent call last):
  File "C:\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Rez\Scripts\rez-pip2.exe\__main__.py", line 7, in <module>
  File "c:\program files\rez\lib\site-packages\rez_pip\cli.py", line 318, in run
    _run(args, pipArgs, pipWorkArea)
  File "c:\program files\rez\lib\site-packages\rez_pip\cli.py", line 183, in _run
    pipArgs,
  File "c:\program files\rez\lib\site-packages\rez_pip\pip.py", line 131, in getPackages
    rawData = json.load(fd)
  File "C:\Python37\lib\json\__init__.py", line 293, in load
    return loads(fp.read(),
  File "C:\Python37\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 3565: character maps to <undefined>

Failling with custom versioning for python package.

Hello, my issue is related to my workflow so I don't except much, but reporting in case it can be of any use.

traceback

 ./rez-pip2.exe progress
INFO     Installing requested packages for Python 3.9.13.1

Usage:
  pip.pyz install [options] <requirement specifier> [package-index-options] ...
  pip.pyz install [options] [-e] <vcs project url> ...
  pip.pyz install [options] [-e] <local project path> ...
  pip.pyz install [options] <archive url/path> ...

--python-version error: invalid --python-version value: '3.9.13.1': at
rez_pip\\data\\pip.pyz install -q progress --disable-pip-version-check --dry-run --ignore-installed --python-version=3.9.13.1 --only-binary=:all: --target=/tmp/asd --disable-pip-version-check --report C:\\Users\\lcoll\\AppData\\Local\\Temp\\pip-install-outputb2h27l70'

reason

I am using a custom versioning schema for all vendor packages by appending an additional version token. Example a python-3.9.13 can become python-3.9.13.1.
It seems that rez-pip has no issue by founding my python version, though pip of course doesn't like the non-standard version and fail.

potential-fix

  1. I'm not sure it would be desirable to include a python version "sanitize" that would strip out any non standard token ?
  2. Or perhaps it could read the python version from a custom package attribute instead ?
  3. I think the best solution would just to let the use provide the actual version to pip, i.e 
    rez-pip2 whatever --python-version 3.9.13.1 -- --python-version 3.9.13
    which currently doesn't work as pip args are append to the existing pip command and doesn't override the existing args

context

  • windows 10
  • rez-pip 0.2.0 from pip
  • rez 2.113.0

Fail to install PySide6

Hello !
I have been trying to use rez-pip to install PySide6 :

rez python -m rez_pip PySide6 --python-version ==3.9.13

Which fail with the following :

INFO     Installing requested packages for Python 3.9.13
INFO     Resolved 4 dependencies for python 3.9.13
INFO     Downloading...
INFO     Downloaded PySide6-6.6.0 to
         'C:\\Users\\lcoll\\AppData\\Local\\Temp\\rez-pip-targetce9bd26z\\wheels\\PySide6-6.6.0-cp
         38-abi3-win_amd64.whl' (7225 bytes)
INFO     Downloaded shiboken6-6.6.0 to
         'C:\\Users\\lcoll\\AppData\\Local\\Temp\\rez-pip-targetce9bd26z\\wheels\\shiboken6-6.6.0-
         cp38-abi3-win_amd64.whl' (1066845 bytes)
INFO     Downloaded PySide6-Essentials-6.6.0 to
         'C:\\Users\\lcoll\\AppData\\Local\\Temp\\rez-pip-targetce9bd26z\\wheels\\PySide6_Essentia
         ls-6.6.0-cp38-abi3-win_amd64.whl' (76703502 bytes)
INFO     Downloaded PySide6-Addons-6.6.0 to
         'C:\\Users\\lcoll\\AppData\\Local\\Temp\\rez-pip-targetce9bd26z\\wheels\\PySide6_Addons-6
         .6.0-cp38-abi3-win_amd64.whl' (110273318 bytes)
INFO     Downloaded 4 wheels
INFO     Installing PySide6-6.6.0 wheel
INFO     Installing PySide6-Addons-6.6.0 wheel
Traceback (most recent call last):
  File "Z:\packages-dev\python\3.10.7\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "Z:\packages-dev\python\3.10.7\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "c:\program files\rez\lib\site-packages\rez_pip\__main__.py", line 3, in <module>
    rez_pip.cli.run()
  File "c:\program files\rez\lib\site-packages\rez_pip\cli.py", line 318, in run
    _run(args, pipArgs, pipWorkArea)
  File "c:\program files\rez\lib\site-packages\rez_pip\cli.py", line 200, in _run
    dist, isPure = rez_pip.install.installWheel(
  File "c:\program files\rez\lib\site-packages\rez_pip\install.py", line 89, in installWheel
    installer.install(
  File "c:\program files\rez\lib\site-packages\installer\_core.py", line 109, in install
    record = destination.write_file(
  File "c:\program files\rez\lib\site-packages\installer\destinations.py", line 207, in write_file
    return self.write_to_fs(scheme, path_, stream, is_executable)
  File "c:\program files\rez\lib\site-packages\installer\destinations.py", line 167, in write_to_fs
    raise FileExistsError(message)
FileExistsError: File already exists: C:\Users\lcoll\AppData\Local\Temp\rez-pip-targetce9bd26z\installed\3.9.13\python\PySide6/__init__.py

It seems that PySide6-Addons is overriding some stuff in the existing __init__.py of the just-installed PySide6 which does not seem to be expected by rez.

I have not investigated further but will report anything I find below.

context

  • windows 10
  • rez-pip 0.2.0 from pip
  • rez 2.113.0

Metadata is not stored on a per variant basis

I recently realized that the metadata we stored in the package definition is not done on a per variable basis. This is extremely problematic because rez-pip's installs wheels, which can have different metadata for a given version. The first thing that comes to mind is the URL of the wheel, then if could be the license or whatever else.

This means that the metadata we store in the pip attribute gets overridden every time a new variant is added/installed. This is very bad.

Rez's design is unfortunately playing against us here. There is only a single package definition for a package (keeping in mind that a package is a version and a version can contain variants). There is no native way to store variant specific metadata.

`rez_pip_version` metadata missing from produced package.py files

When running rez-pip2 on any package, the rez_pip_version metadata is missing from the produced package.py file.

This metadata is shown to be present in the docs but does not appear in produced package.py files.

Environment

  • Windows 10
  • Rez version 2.112.0
  • Rez python version 3.7
  • rez-pip 0.3.1

To Reproduce

  1. Install new rez-pip
  2. rez-pip2 semver==3.0.2 (can also try requests==2.31.0 or any other package at all)

Expected Behavior
rez_pip_version metadata present in pip attribute of package.py.

Actual Behavior
rez_pip_version is missing from pip attribute of package.py

Extras not translated to rez

When running a rez-pip2 on the boxsdk PyPi package with the extra "jwt" specified, the extra dependencies are not included in the resulting package.py file.

Environment

  • Windows 10
  • Rez version 2.112.0
  • Rez python version 3.7
  • rez-pip 0.4.0.dev1

To Reproduce

  1. Install new rez-pip
  2. rez-pip2 boxsdk[jwt]==3.9.2 --python-version 3.7

Expected behavior
"jwt" and "cryptography" packages listed in "extra_requires" on boxsdk's setup.py file are included in requires portion of resulting package.py.

Actual behavior
"jwt" and "cryptography" packages are missing from the resulting requires portion of the package.py.

Unexpected python executable can be used by rez-pip

In some cases, rez-pip can unexpectedly use a python executable found on your machine instead of a rezified python package.

Environment

  • Windows 10.0
  • rez-pip 0.2.0
  • rez-pip being installed in its own virtual environment, not part of a rez installation

To Reproduce

  • Create an empty file named python3.exe in some arbitrary folder, e.g. "C:\foo\python3.exe"
  • Rezify a python interpreter through this utility
  • Install rez-pip into its own virtualenv
    • open a terminal
    • run rez env python -- python -m venv C:\rez-pip to generate a virtualenv that will host the rez-pip package
    • run cd C:\rez-pip\Scripts then .\activate to enter the virtualenv
    • run python -m pip install rez-pip to install rez-pip in the virtualenv
  • Through the Windows Configuration panel, add "C:\foo" at the end of your PATH environment variable
  • open a terminal
  • run cd C:\rez-pip\Scripts then .\activate to enter the virtualenv of rez-pip
  • run .\rez-pip2.exe future
  • see the error:
> .\rez-pip2.exe future
INFO     Installing requested packages for Python 3.7.7
Traceback (most recent call last):
  File "\\ ... \packages\python\3.9.10\platform-windows\arch-AMD64\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "\\ ... \packages\python\3.9.10\platform-windows\arch-AMD64\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\rez-pip\Scripts\rez-pip2.exe\__main__.py", line 7, in <module>
  File "C:\rez-pip\lib\site-packages\rez_pip\cli.py", line 318, in run
    _run(args, pipArgs, pipWorkArea)
  File "C:\rez-pip\lib\site-packages\rez_pip\cli.py", line 176, in _run
    packages = rez_pip.pip.getPackages(
  File "C:\rez-pip\lib\site-packages\rez_pip\pip.py", line 106, in getPackages
    process = subprocess.Popen(
  File "\\ ... \packages\python\3.9.10\platform-windows\arch-AMD64\lib\subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "\\ ... \packages\python\3.9.10\platform-windows\arch-AMD64\lib\subprocess.py", line 1420, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
OSError: [WinError 193] %1 not a valid Win32 application

Expected behavior
Program installs future as a rez package.

Actual behavior
The exception above occurs.

Investigation
The error is due to this line checking for a "python3.exe" in PATH before checking for "python.exe". The empty python3.exe we created gets found before the python.exe of our rezified python.

Possible suggestions

  • If the rezified python package provides python in its tools section, directly use "python" instead of attempting "python3.7", "python3" then "python"
  • check that the result of which is a under the root of a rezified python package

Note
IMO being able to use an arbitrary python executable instead of the one provided by a rezified python package could be interesting, but it should be an explicit behaviour requested by the user; the default behavior of rez-pip should be to only use a rezified python package.

rez-pip can't handle python_version requirements in packages

elastictransport 8.12.0 has the following requirements:

'urllib3 <3,>=1.26.2',
'certifi',
'dataclasses ; python_version < "3.7"',
'importlib-metadata ; python_version < "3.8"'
...
...

When running rez-pip2 elastictransport==8.12.0, I end up getting a ResourceError like this:

Traceback (most recent call last):
  File "C:\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Rez\Scripts\rez-pip2.exe\__main__.py", line 7, in <module>
  File "c:\program files\rez\lib\site-packages\rez_pip\cli.py", line 324, in run
    _run(args, pipArgs, pipWorkArea)
  File "c:\program files\rez\lib\site-packages\rez_pip\cli.py", line 225, in _run
    release=args.release,
  File "c:\program files\rez\lib\site-packages\rez_pip\rez.py", line 126, in createPackage
    pkg.pip["metadata"] = remainingMetadata
  File "C:\Python37\lib\contextlib.py", line 119, in __exit__
    next(self.gen)
  File "c:\program files\rez\lib\site-packages\rez\package_maker.py", line 209, in make_package
    variant_ = variant.install(path, dry_run=True)
  File "c:\program files\rez\lib\site-packages\rez\packages.py", line 457, in install
    overrides=overrides)
  File "c:\program files\rez\lib\site-packages\rezplugins\package_repository\filesystem.py", line 951, in install_variant

    variant = _create_variant()
  File "c:\program files\rez\lib\site-packages\rezplugins\package_repository\filesystem.py", line 947, in _create_variant

    overrides=overrides
  File "c:\program files\rez\lib\site-packages\rezplugins\package_repository\filesystem.py", line 1246, in _create_variant
    % (variant, package))
rez.exceptions.ResourceError: Attempting to install a package without variants (MemoryVariantResource({'location': 'memory{0x1ea5f20c0e8}', 'name': 'elastic_transport', 'version': '8.12.0', 'index': None, 'repository_type': 'memory'})) into an existing package with variants (FileSystemPackageResource({'location': '\\\\vm-fs-003\\prod\\sww\\rez\\packages', 'name': 'elastic_transport', 'version': '8.12.0', 'repository_type': 'filesystem'}))

It seems like the python version dependent dependency was put in the variant list in the output package.py file:

variants = [['importlib_metadata']]

I am doing the rez-pip install with both Python 3.7.9 and Python 3.9.10 at the same time (by not specifying --python-version).

The issue seems to be that the Python 3.7 variant requires importlib-metadata but the 3.9 variant does not, and so the above crash happens.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.