Scripts for setting up Pants in your repo
License: Apache License 2.0
This repository contains the bootstrap needed to get you up and running with Pants.
Follow the instructions at https://www.pantsbuild.org/docs/installation.
We use Pants for test automation (so recursive!). To run the test suite, linters and formatters:
$ pants fmt lint check test ::
I don't think the -p option of mktemp is supported, as OSX ships with a BSD version of mktemp from 2005.
(I'll update this with a proposed OSX workaround if I find one)
asdf is a CLI for managing multiple runtime versions of other tools (i.e. python). To use an asdf-managed tool, you:
asdf install it (i.e. asdf install python 3.7.12).tool-versions file.tool-versions are hierarchical - for example if you have ~/.tool-versions:
python 3.7.12
and also ~/project/.tool-versions:
python 3.8.9
Then running python --version from within ~/project will print 3.8.9 instead of 3.7.12.
Pants' python backend has dedicated support for locating asdf-installed Pythons within both ~/.tool-versions and in <project-root>/.tool-versions, but my team has found that the bootstrapping process only ever uses versions from ~/.tool-versions. This happens because the bootstrapping process cds into temp dirs at various points, preventing asdf from seeing the project-local .tool-versions.
I think we can refactor the script to avoid the cds and stay within the project root, allowing local asdf to work.
In the past month, we've had two users confused from trying to use macOS 10.14. The error message is quite cryptic:
$ ./pants test algorithms::
Bootstrapping Pants using /Users/user/.pyenv/shims/python3.9
Installing pantsbuild.pants==2.5.0 into a virtual environment at /Users/user/.cache/pants/setup/bootstrap-Darwin-x86_64/2.5.0_py39
created virtual environment CPython3.9.2.final.0-64 in 353ms
creator CPython3Posix(dest=/Users/user/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.xGjbPP/install, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/sohum/Library/Application Support/virtualenv)
added seed packages: pip==21.1.2, setuptools==57.0.0, wheel==0.36.2
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Requirement already satisfied: pip in /Users/user/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.xGjbPP/install/lib/python3.9/site-packages (21.1.2)
Collecting pip
Using cached pip-21.2.4-py3-none-any.whl (1.6 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 21.1.2
Uninstalling pip-21.1.2:
Successfully uninstalled pip-21.1.2
Successfully installed pip-21.2.4
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==2.5.0 (from versions: 0.0.17, 0.0.18, 0.0.20, 0.0.21, 0.0.22, 0.0.23, 0.0.24, 0.0.25, 0.0.26, 0.0.27, 0.0.28, 0.0.29, 0.0.30, 0.0.31, 0.0.32, 0.0.33, 0.0.34, 0.0.35, 0.0.36, 0.0.37, 0.0.38, 0.0.39, 0.0.40, 0.0.41, 0.0.42, 0.0.43, 0.0.44, 0.0.45, 0.0.46, 0.0.47, 0.0.48, 0.0.49, 0.0.50, 0.0.51, 0.0.52, 0.0.53, 0.0.54, 0.0.55, 0.0.56, 0.0.57, 0.0.58, 0.0.59, 0.0.60, 0.0.61, 0.0.62, 0.0.63, 0.0.64, 0.0.65, 0.0.66, 0.0.67, 0.0.68, 0.0.69, 0.0.70, 0.0.71, 0.0.72, 0.0.73, 0.0.74, 0.0.75, 0.0.76, 0.0.77, 0.0.79, 0.0.80, 0.0.81, 0.0.82, 1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.19.0, 1.20.0, 1.21.0, 1.22.0, 1.23.0, 1.24.0, 1.25.0, 1.26.0, 1.27.0, 1.28.0)
ERROR: No matching distribution found for pantsbuild.pants==2.5.0
Instead, we can eagerly detect if using an invalid OS and print a nice error message with your options.
From a user:
When I have a line in pants.toml
pants_version = "2.6.1rc1" # using rc1 because blah
Pants will try to download a version called "2.6.1rc1 # using rc1 because blah"
The bootstrap-update command would fetch the latest setup script and save it over $0.
Originally posted by @kaos in #129 (comment)
I ran into an issue w/ PyMock==1.3.0 and OSX Sierra w/ pantsbuild==1.2.0rc4 and I'll send a PR shortly. It appears that OSX Sierra comes w/ six==1.4 and that is getting pulled into the virtual environment when pants runs. This manifests itself with pants repl and pants test ... here's what I see.
Dumping requirement: mock==1.3.0
Dumping requirement: requests==2.8.1
Dumping requirement: six==1.10.0
Dumping requirement: enum34==1.0
Dumping requirement: kafka-python==1.0.2
Dumping requirement: python-snappy==0.5
Dumping requirement: kazoo==2.2.1
Dumping distribution: .../enum34-1.0-py2-none-any.whl
Dumping distribution: .../pbr-1.10.0-py2.py3-none-any.whl
Dumping distribution: .../funcsigs-1.0.2-py2.py3-none-any.whl
Dumping distribution: .../python_snappy-0.5-cp27-none-linux_x86_64.whl
Dumping distribution: .../kazoo-2.2.1-py2.py3-none-any.whl
Dumping distribution: .../mock-1.3.0-py2.py3-none-any.whl
Dumping distribution: .../requests-2.8.1-py2.py3-none-any.whl
Dumping distribution: .../six-1.10.0-py2.py3-none-any.whl
Dumping distribution: .../kafka_python-1.0.2-py2.py3-none-any.whl
Dumping distribution: .../python_snappy-0.5-cp27-none-macosx_10_11_intel.whl
09:27:01 00:02 [repl]
Python 2.7.10 (default, Jul 30 2016, 18:31:42)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from six import wraps
Traceback (most recent call last):
File "<console>", line 1, in <module>
ImportError: cannot import name wraps
>>> import six
>>> six.__version__
'1.4.1'
>>> six.__file__
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six.pyc'
>>> import sys
>>> sys.path
['/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.bootstrap', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/funcsigs-1.0.2-py2.py3-none-any.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/enum34-1.0-py2-none-any.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/requests-2.8.1-py2.py3-none-any.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/mock-1.3.0-py2.py3-none-any.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/kafka_python-1.0.2-py2.py3-none-any.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/pbr-1.10.0-py2.py3-none-any.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/python_snappy-0.5-cp27-none-macosx_10_11_intel.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/six-1.10.0-py2.py3-none-any.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/kazoo-2.2.1-py2.py3-none-any.whl']
>>>I've tested w/ manually setting up the PYTHONPATH and this is what I see.
Dumping requirement: mock==1.3.0
Dumping requirement: requests==2.8.1
Dumping requirement: six==1.10.0
Dumping requirement: enum34==1.0
Dumping requirement: kafka-python==1.0.2
Dumping requirement: python-snappy==0.5
Dumping requirement: kazoo==2.2.1
Dumping distribution: .../enum34-1.0-py2-none-any.whl
Dumping distribution: .../pbr-1.10.0-py2.py3-none-any.whl
Dumping distribution: .../funcsigs-1.0.2-py2.py3-none-any.whl
Dumping distribution: .../python_snappy-0.5-cp27-none-linux_x86_64.whl
Dumping distribution: .../kazoo-2.2.1-py2.py3-none-any.whl
Dumping distribution: .../mock-1.3.0-py2.py3-none-any.whl
Dumping distribution: .../requests-2.8.1-py2.py3-none-any.whl
Dumping distribution: .../six-1.10.0-py2.py3-none-any.whl
Dumping distribution: .../kafka_python-1.0.2-py2.py3-none-any.whl
Dumping distribution: .../python_snappy-0.5-cp27-none-macosx_10_11_intel.whl
09:36:42 00:17 [repl]
Python 2.7.10 (default, Jul 30 2016, 18:31:42)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from six import wraps
>>> import six
>>> six.__version__
'1.10.0'
>>> six.__file__
'/Users/jvenus/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0rc4/lib/python2.7/site-packages/six.pyc'
>>> import sys
>>> sys.path
['/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.bootstrap', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a', '/Users/jvenus/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0rc4/lib/python2.7', '/Users/jvenus/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0rc4/lib/python2.7/site-packages', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/funcsigs-1.0.2-py2.py3-none-any.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/enum34-1.0-py2-none-any.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/requests-2.8.1-py2.py3-none-any.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/mock-1.3.0-py2.py3-none-any.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/kafka_python-1.0.2-py2.py3-none-any.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/pbr-1.10.0-py2.py3-none-any.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/python_snappy-0.5-cp27-none-macosx_10_11_intel.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/six-1.10.0-py2.py3-none-any.whl', '/Users/jvenus/dev/brighttag/pypr/.pants.d/python-setup/chroots/08376bd6219bf5e92e06d0476c8634d350ebfb3a/.deps/kazoo-2.2.1-py2.py3-none-any.whl']As reported over in pantsbuild/pants#12984, the pants script uses the Pex PEX to create a virtual environment for pantsbuild.pants and this is thwarted if an alternate PEX entrypoint is specified via environment variables:
$ PEX_MODULE=foo ./pants -V
Bootstrapping Pants using /usr/local/bin/python3.8
Creating the virtualenv PEX.
Downloading the Pex PEX.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 609 100 609 0 0 3736 0 --:--:-- --:--:-- --:--:-- 3736
100 3529k 100 3529k 0 0 3211k 0 0:00:01 0:00:01 --:--:-- 4346k
SHA256 fingerprint of https://github.com/pantsbuild/pex/releases/download/v2.1.42/pex verified.
Traceback (most recent call last):
File "/root/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 484, in execute
exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
File "/root/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
return runner(*args)
File "/root/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
return runner(*args)
File "/root/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 536, in _execute
return self.execute_entry(self._pex_info_overrides.entry_point)
File "/root/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 667, in execute_entry
return self.execute_module(entry_point, alter_sys)
File "/root/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 679, in execute_module
runpy.run_module(module_name, run_name="__main__", alter_sys=alter_sys)
File "/usr/local/lib/python3.8/runpy.py", line 203, in run_module
mod_name, mod_spec, code = _get_module_details(mod_name)
File "/usr/local/lib/python3.8/runpy.py", line 138, in _get_module_details
raise error("No module named %s" % mod_name)
ImportError: No module named foo
mv: cannot stat '/root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.fvP0e0/virtualenv.pex': No such file or directory
Installing pantsbuild.pants==2.6.1 into a virtual environment at /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.6.1_py38
/usr/local/bin/python3.8: can't open file '/root/.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-20.4.7/virtualenv.pex': [Errno 2] No such file or directory
The command '/bin/sh -c ./pants --version' returned a non-zero code: 1I didn't realize that the install script had its own repo.
We use this script almost word-for-word in Fsq.io. The only change is that we add a FOURSQUARE_REQUIREMENTS variable that points to an additional requirements.txt. We have added two lines to the install script so that it installs those python deps into the pants virtualenv.
Internally, we set up a whole additional virtualenv for our 3rdparty deps, called fs-env. From what I understand, that is what Twitter does as well and Square may be about to create their own. But maintaining that additional virtualenv is a lot to ask a user that may just want to add a couple python libraries.
Could we add the ability to pass additional requirements.txt to the install script? My first thought would be an environmental variable but perhaps there is a better idea.
I am willing to write the patch if it (or something like it) sounds acceptable.
Failures don't hint that this is the issue except very indirectly if you know how pex goes about its business:
$ mkdir /tmp/test2 && cd /tmp/test2 && curl -sSL -O https://pantsbuild.github.io/setup/pants && chmod +x pants && cat << EOF > pants.ini
[GLOBAL]
pants_version: 1.2.1
plugins: [
'pantsbuild.pants.contrib.go==%(pants_version)s',
'pantsbuild.pants.contrib.scrooge==%(pants_version)s',
]
EOF
$ rm -rf .pants.d/ ~/.pex/ ~/.cache/pants/{setup,plugins,python_cache} ~/.cache/pip/
$ PEX_VERBOSE=5 ./pants -V
...
Installing setuptools, pip, wheel...done.
Requirement already satisfied (use --upgrade to upgrade): setuptools<31.0,>=5.4.1 in /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.wsboUf/install/lib/python2.7/site-packages
...
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x7fa3f385f290>
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x7fa3f37eec50>
pex: crawling link i=0 link=Link('file:///home/jsirois/.cache/pants/plugins') follow_links=False
pex: crawling link i=0 link=Link('https://pypi.python.org/simple/pantsbuild.pants.contrib.go/') follow_links=False
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x7fa3f37eed90>
pex: crawling link i=0 link=Link('https://pypi.python.org/simple/pantsbuild.pants.contrib.scrooge/') follow_links=False
pex: Validated pantsbuild.pants.contrib.scrooge-1.2.1.tar.gz (md5=8f6f45cb794f5d4887d46730d73f7862)
pex: Validated pantsbuild.pants.contrib.scrooge-1.2.1.tar.gz (md5=8f6f45cb794f5d4887d46730d73f7862)
pex: Fetching https://pypi.python.org/packages/66/3d/633f69deee202cb1397c5c3f11721696891e8680a2ff29c930fbd4cd3094/pantsbuild.pants.contrib.scrooge-1.2.1.tar.gz#md5=8f6f45cb794f5d4887d46730d73f7862: 63.8ms
pex: Fetching file:///home/jsirois/.cache/pants/plugins/pantsbuild.pants.contrib.scrooge-1.2.1.tar.gz: 1.9ms
pex: Fetching file:///home/jsirois/.cache/pants/plugins/pantsbuild.pants.contrib.scrooge-1.2.1.tar.gz: 0.2ms
pex: Failed to translate SourcePackage(u'file:///tmp/tmpP1KhHj/pantsbuild.pants.contrib.scrooge-1.2.1.tar.gz')
pex: Traceback (most recent call last):
File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.2.1/lib/python2.7/site-packages/pex/translator.py", line 101, in translate
strict=(package.name not in ('distribute', 'setuptools')))
File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.2.1/lib/python2.7/site-packages/pex/installer.py", line 61, in __init__
self._interpreter.binary, self.__class__.__name__))
IncapableInterpreter: Interpreter /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.wsboUf/install/bin/python2.7 not capable of running EggInstaller
pex: Translating /tmp/tmpP1KhHj/pantsbuild.pants.contrib.scrooge-1.2.1.tar.gz into distribution: 6.8ms
Exception caught: (<class 'pex.resolver.Untranslateable'>)
File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.2.1/bin/pants", line 11, in <module>
sys.exit(main())
...
File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.2.1/lib/python2.7/site-packages/pex/resolver.py", line 168, in build
raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
Exception message: Package SourcePackage(u'file:///home/jsirois/.cache/pants/plugins/pantsbuild.pants.contrib.scrooge-1.2.1.tar.gz') is not translateable by ChainedTranslator(EggTranslator, SourceTranslator)
Here, an existing setuptools version:
$ ls -l /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.wsboUf/install/lib/python2.7/site-packages | grep setuptools
drwxr-xr-x 4 jsirois jsirois 4096 Jan 30 19:19 setuptools
drwxr-xr-x 2 jsirois jsirois 4096 Jan 30 19:19 setuptools-21.2.1.dist-info
Saisfied the setup script constraint of setuptools<31.0,>=5.4.1, but failed the pex==1.1.13 constaint of:
$ cat /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.wsboUf/install/lib/python2.7/site-packages/pex/version.py
# Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).
__version__ = '1.1.13'
SETUPTOOLS_REQUIREMENT = 'setuptools>=2.2,<20.11'
WHEEL_REQUIREMENT = 'wheel>=0.24.0,<0.30.0'
Currently pants supports both of these environments and pantsbuild/pants has prior art for getting an OSX ci working.
It would be nice to avoid basic issues like those noted in #1 before changes hit mainline.
When running curl -L -O https://pantsbuild.github.io/setup/pants && chmod +x pants && touch pants.ini I am seeing the following issue in the openjdk docker container based on Debian Jessie:
# ./pants -V
New python executable in /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.Y6bv1k/install/bin/python2.7
Also creating executable in /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.Y6bv1k/install/bin/python
Installing pip, wheel...done.
Collecting pantsbuild.pants
Using cached pantsbuild.pants-1.1.0.tar.gz
Could not import setuptools which is required to install from a source distribution.
Please install setuptools.
./pants: line 84: /root/.cache/pants/setup/bootstrap-Linux-x86_64/unspecified/bin/python: No such file or directoryI am able to resolve this issue by reversing the change made in commit a4b5f74 and installing setuptools in the venv
A regression was reported on the Pants Slack in the ./pants wrapper script after a virtualenv upgrade. https://pantsbuild.slack.com/archives/C046T6T9U/p1608991742321800.
I reproduced it with the example-python repository:
~/TC/example-python$ ./pants --version
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 122 100 122 0 0 606 0 --:--:-- --:--:-- --:--:-- 610
100 281 100 281 0 0 747 0 --:--:-- --:--:-- --:--:-- 747
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 8861k 100 8861k 0 0 7348k 0 0:00:01 0:00:01 --:--:-- 14.2M
Traceback (most recent call last):
File "/Users/tdyas/.cache/pants/setup/bootstrap-Darwin-x86_64/virtualenv-20.2.2/src/virtualenv/__main__.py", line 77, in <module>
run_with_catch() # pragma: no cov
File "/Users/tdyas/.cache/pants/setup/bootstrap-Darwin-x86_64/virtualenv-20.2.2/src/virtualenv/__main__.py", line 58, in run_with_catch
from virtualenv.config.cli.parser import VirtualEnvOptions
ModuleNotFoundError: No module named 'virtualenv'
./pants: line 293: /Users/tdyas/.cache/pants/setup/bootstrap-Darwin-x86_64/2.0.0rc1_py38/bin/python: No such file or directory
We should revert until a solution is found. #93 has the reversion.
It's an error to fall back on 3.8 and 3.7, resulting in pantsbuild/pants#15332.
For example:
~ $ cd /tmp
/tmp $ mkdir setup-venv-corrupted
/tmp $ cd setup-venv-corrupted/
/tmp/setup-venv-corrupted $ printf '[GLOBAL]\npants_version = "2.11.0.dev0"\n' > pants.toml
/tmp/setup-venv-corrupted $ curl -L -O https://static.pantsbuild.org/setup/pants && chmod +x ./pants
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 162 100 162 0 0 111 0 0:00:01 0:00:01 --:--:-- 111
100 14844 100 14844 0 0 8502 0 0:00:01 0:00:01 --:--:-- 70018
/tmp/setup-venv-corrupted $ pyenv install 3.9.2
Downloading Python-3.9.2.tar.xz...
-> https://www.python.org/ftp/python/3.9.2/Python-3.9.2.tar.xz
Installing Python-3.9.2...
Installed Python-3.9.2 to /home/jsirois/.pyenv/versions/3.9.2
/tmp/setup-venv-corrupted $ PATH=~/.pyenv/versions/3.9.2/bin:$PATH ./pants -V
Bootstrapping Pants using /home/jsirois/.pyenv/versions/3.9.2/bin/python3.9
Installing pantsbuild.pants==2.11.0.dev0 into a virtual environment at /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/2.11.0.dev0_py39
...
New virtual environment successfully created at /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/2.11.0.dev0_py39.
08:49:37.56 [INFO] Initializing scheduler...
08:49:37.62 [INFO] Scheduler initialized.
08:49:37.63 [WARN] Please either set `enabled = true` in the [anonymous-telemetry] section of pants.toml to enable sending anonymous stats to the Pants project to aid development, or set `enabled = false` to disable it. No telemetry sent for this run. An explicit setting will get rid of this message. See https://www.pantsbuild.org/v2.11/docs/anonymous-telemetry for details.
2.11.0.dev0
/tmp/setup-venv-corrupted $ pyenv uninstall 3.9.2
pyenv: remove /home/jsirois/.pyenv/versions/3.9.2? [y|N] y
pyenv: 3.9.2 uninstalled
/tmp/setup-venv-corrupted $ ./pants -V
./pants: line 387: /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/2.11.0.dev0_py39/bin/python: No such file or directory
/tmp/setup-venv-corrupted $Pants version: 1.21.0
Pants installed via: curl -L -O https://pantsbuild.github.io/setup/pants && chmod +x pants
Update the pants.ini to pants_version: 1.21.0 then run ./pants
C02XV41NJHD2:mlp-ark zhanwu.xiong$ ./pants
pyenv: python3.7: command not found
The `python3.7' command exists in these Python versions:
3.7.4
pyenv: python3.7: command not found
The `python3.7' command exists in these Python versions:
3.7.4
./pants: line 170: /Users/zhanwu.xiong/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.35pSjk/install/bin/pip: No such file or directory
./pants: line 171: /Users/zhanwu.xiong/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.35pSjk/install/bin/pip: No such file or directory
New virtual environment successfully created at /Users/zhanwu.xiong/.cache/pants/setup/bootstrap-Darwin-x86_64/1.21.0_py.
./pants: line 193: /Users/zhanwu.xiong/.cache/pants/setup/bootstrap-Darwin-x86_64/1.21.0_py/bin/python: No such file or directory
Full pants.ini after update:
[GLOBAL]
pants_version: 1.21.0
print_exception_stacktrace: True
pythonpath: ["%(buildroot)s/plugins"]
plugins: [
'pantsbuild.pants.contrib.go==%(pants_version)s',
'pantsbuild.pants.contrib.python.checks==%(pants_version)s',
'docker==4.0.2']
backend_packages : [
'pants.backend.graph_info',
'pants.backend.python',
'pants.backend.native',
'pants.backend.project_info',
'pants.backend.codegen.grpcio.python',
'pants.rules.core',
'cruise.docker']
[python-setup]
interpreter_constraints: ["CPython>=3.6"]
[source]
source_root_patterns = ['src/python/*', 'src/*', 'src/main/*', 'src/scala/*', 'src/java/*']
# Per language reverse key map.
# Go lang is designed to be built from Monorepo structure
source_roots = {u'go/src': (u'go',), u'go/main/src': (u'go',)}
[python-repos]
indexes = ['https://artifactory.robot.car/artifactory/api/pypi/wheels-python-local/simple', 'https://pypi.org/simple/']
repos = []
[repl.py]
ipython = True
[go-distribution]
version: 1.12
[lint.python-eval]
skip: True
The immediate cause of the issue is pants setup believes python3.7 is a valid python version, but actually it is not. See the demonstration below:
C02XV41NJHD2:mlp-ark zhanwu.xiong$ pyenv versions
system
2.7.15
* 3.6.9 (set by /Users/zhanwu.xiong/cruise/mlp-ark/.python-version)
3.7.4
C02XV41NJHD2:mlp-ark zhanwu.xiong$ which python3.7
/Users/zhanwu.xiong/.pyenv/shims/python3.7
C02XV41NJHD2:mlp-ark zhanwu.xiong$ python3.7
pyenv: python3.7: command not found
The `python3.7' command exists in these Python versions:
3.7.4
Arguably, this could be a pyenv issue
Right now the pants version is embedded in the setup.sh script. Instead, it should live in pants.ini [DEFAULT] pants_version where the intellij plugin expects it and its easier to discover and change by an end user.
The script currently contains:
pants_requirement="pantsbuild.pants"
pants_version=$(
grep -E "^[[:space:]]*pants_version" pants.ini 2>/dev/null | \
sed -E 's|^[[:space:]]*pants_version[:=][[:space:]]*([^[:space:]]+)|\1|'
)
CentOS 5 ships with:
$ sed --version
GNU sed version 4.1.5
Copyright (C) 2003 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE,
to the extent permitted by law.
which does not support the -E parameter:
$ echo a | sed -E "s|a|b|"
sed: invalid option -- E
....
The use of -E should be removed so this script can work on systems that ship with older versions of coreutils.
Failure looks like so (here trying to run Pants 1.30.1)
AssertionError: Bootstrapping Pants using /home/jsirois/.pyenv/shims/python3.6
Creating the virtualenv PEX.
Downloading the Pex PEX.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 640 100 640 0 0 1774 0 --:--:-- --:--:-- --:--:-- 1777
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 3609k 100 3609k 0 0 2810k 0 0:00:01 0:00:01 --:--:-- 3953k
SHA256 fingerprint of https://github.com/pantsbuild/pex/releases/download/v2.1.62/pex verified.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15.0)",)': /simple/filelock/
Installing pantsbuild.pants==1.30.5rc1 into a virtual environment at /tmp/pytest-of-jsirois/pytest-7/test_only_bootstraps_the_first0/PANTS_SETUP_CACHE/bootstrap-Linux-x86_64/1.30.5rc1_py36
created virtual environment CPython3.6.15.final.0-64 in 90ms
creator CPython3Posix(dest=/tmp/pytest-of-jsirois/pytest-7/test_only_bootstraps_the_first0/PANTS_SETUP_CACHE/bootstrap-Linux-x86_64/pants.dEsSDq/install, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/jsirois/.local/share/virtualenv)
added seed packages: pip==21.3.1, setuptools==59.6.0, wheel==0.36.2
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Collecting pantsbuild.pants==1.30.5rc1
Using cached pantsbuild.pants-1.30.5rc1-cp36-cp36m-manylinux2014_x86_64.whl (47.8 MB)
Collecting typing-extensions==3.7.4.2
Using cached typing_extensions-3.7.4.2-py3-none-any.whl (22 kB)
Collecting setuptools==44.0.0
Using cached setuptools-44.0.0-py2.py3-none-any.whl (583 kB)
Collecting twitter.common.dirutil<0.4,>=0.3.11
Using cached twitter.common.dirutil-0.3.11.tar.gz (9.5 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting pex==2.1.24
Using cached pex-2.1.24-py2.py3-none-any.whl (2.5 MB)
Collecting dataclasses==0.6
Using cached dataclasses-0.6-py3-none-any.whl (14 kB)
Collecting Pygments==2.6.1
Using cached Pygments-2.6.1-py3-none-any.whl (914 kB)
Collecting toml==0.10.1
Using cached toml-0.10.1-py2.py3-none-any.whl (19 kB)
Collecting requests[security]>=2.20.1
Using cached requests-2.27.1-py2.py3-none-any.whl (63 kB)
Collecting PyYAML<5.4,>=5.3.1
Using cached PyYAML-5.3.1.tar.gz (269 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting pathspec==0.8.0
Using cached pathspec-0.8.0-py2.py3-none-any.whl (28 kB)
Collecting packaging==20.3
Using cached packaging-20.3-py2.py3-none-any.whl (37 kB)
Collecting Markdown==2.1.1
Using cached Markdown-2.1.1.zip (341 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting ansicolors==1.1.8
Using cached ansicolors-1.1.8-py2.py3-none-any.whl (13 kB)
Collecting www-authenticate==0.9.2
Using cached www-authenticate-0.9.2.tar.gz (2.4 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting python-Levenshtein==0.12.1
Using cached python-Levenshtein-0.12.1.tar.gz (50 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting pystache==0.5.4
Using cached pystache-0.5.4.tar.gz (75 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /tmp/pytest-of-jsirois/pytest-7/test_only_bootstraps_the_first0/PANTS_SETUP_CACHE/bootstrap-Linux-x86_64/pants.dEsSDq/install/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-pz56_4vk/pystache_fac0619cc279482e866112764b0be8c4/setup.py'"'"'; __file__='"'"'/tmp/pip-install-pz56_4vk/pystache_fac0619cc279482e866112764b0be8c4/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-r4qf9kxc
cwd: /tmp/pip-install-pz56_4vk/pystache_fac0619cc279482e866112764b0be8c4/
Complete output (2 lines):
pystache: using: version '59.6.0' of <module 'setuptools' from '/tmp/pytest-of-jsirois/pytest-7/test_only_bootstraps_the_first0/PANTS_SETUP_CACHE/bootstrap-Linux-x86_64/pants.dEsSDq/install/lib/python3.6/site-packages/setuptools/__init__.py'>
error in pystache setup command: use_2to3 is invalid.
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/d6/fd/eb8c212053addd941cc90baac307c00ac246ac3fce7166b86434c6eae963/pystache-0.5.4.tar.gz#sha256=f7bbc265fb957b4d6c7c042b336563179444ab313fb93a719759111eabd3b85a (from https://pypi.org/simple/pystache/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pystache==0.5.4 (from pantsbuild-pants) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 0.4.1, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.0)
ERROR: No matching distribution found for pystache==0.5.4
As @traviscrawford pointed out, the pants script should have a header that makes it clear:
As pointed out by @traviscrawford this might allow for a more seamless pants install. For one, it ensures the user has python, if not python2.7. For another, it could be a good way to keep the pants script from getting checked into each repo.
If I delete $XDG_CACHE_DIR/pants, a directory with a working pants setup stops working. This seems wrong: clearing cache directories should not change behaviour.
Global version is system which is 3.10.5:
/tmp $ asdf current python
python system /home/kamal/.tool-versions
/tmp $ python --version
Python 3.10.5
Create a directory with local version 3.9.13:
/tmp $ mkdir pants-test
/tmp $ cd pants-test
/t/pants-test $ asdf local python 3.9.13
/t/pants-test $ python --version
Python 3.9.13
Follow installation instructions:
/t/pants-test $ printf '[GLOBAL]\npants_version = "2.12.0"\n' > pants.toml
/t/pants-test $ curl -sL -O https://static.pantsbuild.org/setup/pants && chmod +x ./pants
/t/pants-test $ ./pants --level=error --version
2.12.0
For extra precision, the version of the script corresponds to the current HEAD commit, eb887ad:
/t/pants-test $ sha256sum pants
6a1371d1035900c792815e5c7cee6a54c264bd72958f87dba4afec8b60e00cda pants
/t/pants-test $ curl -sL https://raw.githubusercontent.com/pantsbuild/setup/eb887ad5fd80766214e8c38102a5bc92d1342beb/pants | sha256sum
Here's where things go wrong: I delete the cache directory. I expect the script to reinstall pants and work as before, but instead:
/t/pants-test $ rm -r ~/.cache/pants
/t/pants-test $ ./pants --level=error --version
Bootstrapping Pants using /home/kamal/.asdf/shims/python3.9
Creating the virtualenv PEX.
Downloading the Pex PEX.
No python3.9 executable found for python system
SHA256 of https://github.com/pantsbuild/pex/releases/download/v2.1.62/pex is not as expected. Aborting.
At least two issues here:
I don't know what the right fix is. Some ideas:
sha256sum instead of python's hashlib (this may break compatibility with some systems; does WSL2 include that binary?)For complex reasons, ./pants is being invoked as a subprocess of some other Python process. If I wipe the bootstrap dir (and get rid of --quiet from the relevant pip install in the bootstrap script) I see:
Requirement already satisfied: typing-extensions==4.0.1 in /path/to/typing_extensions (from pantsbuild.pants==2.11.0rc1) (4.0.1)
...
for > 1 requirements. Then if I invoke Pants outside the subprocess I'll get an import error for those packages.
Should pants use the following pip option?
-I, --ignore-installed Ignore the installed packages, overwriting them. This can break your system if the existing package is of a different version or was
installed with a different package manager!
(which works in my use case)
If the bootstrap script is run from outside the repo root (or wherever pants.ini is present) it will fallback to the unspecified version. This will then cause a Version mismatch exception when pants actually runs because pants will search for pants.ini at the root and finds it is supposed to be using a different version.
We can work around this locally with git rev-parse --show-toplevel/pantsbut that's obviously a git specific solution and I'm unsure if there is a good path forward without re-writing the code for findingpants.ini` in bash.
We run into this because humans run pants from scripts inside our repository (for example a local fab.sh that wraps fabric and the dependencies for this fabfile) and those scripts might be at an arbitrary location in the repo (we certainly don't want to stick a hundred bash scripts in the root).
This can be seen with the following test edit:
diff --git a/tests/test_first_time_install.py b/tests/test_first_time_install.py
index c42dce3..860b8a3 100755
--- a/tests/test_first_time_install.py
+++ b/tests/test_first_time_install.py
@@ -23,6 +23,7 @@ def test_only_bootstraps_the_first_time(build_root: Path) -> None:
cwd=str(build_root),
).stderr
assert "Collecting pantsbuild.pants==" in first_run_pants_script_logging
+ assert "PEXWarning" not in first_run_pants_script_logging
second_run_pants_script_logging = subprocess.run(
["./pants", "--version"],
check=True,Looks like so:
$ rm -rf ~/.pex && tox -etest -- -vvsk test_only_bootstraps_the_first_time
...
E 'PEXWarning' is contained here:
E Bootstrapping Pants using /home/jsirois/.pyenv/shims/python3.6
E Creating the virtualenv PEX.
E Downloading the Pex PEX.
E % Total % Received % Xferd Average Speed Time Time Time Current
E Dload Upload Total Spent Left Speed
E
E 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
E 100 640 100 640 0 0 1764 0 --:--:-- --:--:-- --:--:-- 1767
E
E 6 3529k 6 223k 0 0 267k 0 0:00:13 --:--:-- 0:00:13 267k
E 100 3529k 100 3529k 0 0 2055k 0 0:00:01 0:00:01 --:--:-- 3743k
E SHA256 fingerprint of https://github.com/pantsbuild/pex/releases/download/v2.1.42/pex verified.
E WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15.0)",)': /simple/filelock/
E /home/jsirois/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/tools/commands/venv.py:141: PEXWarning: Encountered collision building venv at /home/jsirois/.pex/venvs/short/2b8dbee4 from /home/jsirois/.pex/pip.pex/46820cb5af0dcf9295a4e7f30184cc0e9fa063dc:
E ? ++++++++++
E 1. /home/jsirois/.pex/venvs/b031755f2051fa8ce456b892c522e0cded89f771/79e5dde819fafe12be815b7591ae1d09dcaca28a.59820b7fc23d46059b5f3fb153271805/lib/python3.6/site-packages/constraints.txt was provided by:
E /home/jsirois/.pex/pip.pex/46820cb5af0dcf9295a4e7f30184cc0e9fa063dc/.deps/setuptools/constraints.txt
E /home/jsirois/.pex/pip.pex/46820cb5af0dcf9295a4e7f30184cc0e9fa063dc/.deps/wheel/constraints.txt
E pex_warnings.warn(message)
E Installing pantsbuild.pants==1.30.1 into a virtual environment at /tmp/pytest-of-jsirois/pytest-13/test_only_bootstraps_the_first0/PANTS_SETUP_CACHE/bootstrap-Linux-x86_64/1.30.1_py36
...Currently in the function determine_pants_version a plain python command is used to fetch the pants version from the JSON blob returned by the curl command.
This obviously won't work if python is not available.
In my case I had a Docker image, where Python3.7 was installed and I had to register it as an alternative for the python command.
Wouldn't it be better to instead use the same Python executable as returned by determine_python_exe?
I changed my pants script locally, so that determine_pants_version would take the path to the Python executable as an input argument and called it like this:
python="$(determine_python_exe)"
pants_version="$(determine_pants_version "${python}")"
This also seemed to work like a charm.
I'm running python on macOS and have installed python3.9 as well as python3.8 using homebrew. The app I'm currently using pants for uses 3.8 whereas the default is now 3.9 (and many apps require that). Since I don't want to unlink and re-link 3.8 every time I do a brew upgrade, I keep 3.9 as the default and tried to setup my environment for said app to use a virtualenv with 3.8 instead.
For most tools & IDEs this was pretty easy, but the pants wrapper script doesn't seem to respect that version. Although python and python3 link to the correct python3.8 binary (/usr/local/Cellar/[email protected]/3.8.11/bin/python3), the wrapper is hard coded to iterate through executables that are named pythonX.X. And since virtualenv doesn't create a link or shim in that format, I don't have a python3.8 executable. I do however have a python3.9 executable (linked by homebrew), which then (wrongfully) becomes pants next best guess (despite having set interpreter_constraints = ["==3.8.*"])*.
In my opinion, if interpreter_search_paths = ["<PATH>"] is set, pants should also consider python / python3, check their versions and see if they satisfy the configured constraints. Moreover, it should rather check the actual interpreter version instead of its bare name.
*I believe this might be a bug that is already resolved by now. The wrapper we're using was created sometime around pants 2.2 or 2.3
With Pants soon to be released with Python 3 wheels pantsbuild/pants#7197 and the Python 3 migration nearing completion pantsbuild/pants#6062, we need to modify this setup script to allow Pants end users to consume the upgrade.
.bashrc with their individual preference. It should be an organization-level decision set for the whole repo.rm -rf their cache../pants2 and ./pants. We are not expecting the average Pants user to worry about which Python interpreter they use to run Pants as both should work equally as well and it is more cognitive load than necessary.Users should add pants_runtime_python_version: {2.7,3.6,3.7} to their pants.ini [GLOBAL] entry, just as they set pants_version.
We would update the website at https://www.pantsbuild.org/install.html to provide instructions, along the lines of:
Pants may be ran with Python 2.7, 3.6, or 3.7. This does not mean your code has to use this Python
version, as Pants will spawn subprocesses to run with the interpreter you want to use for your own
codebase; rather, this is what Pants uses for its engine.
While Pants defaults to Python 2.7, we recommend pinning the interpreter you prefer in `pants.ini` by
adding an entry like this:
``ini
[GLOBAL]
pants_version: 1.15.0
pants_runtime_python_version: 3.6
``
Any time you want to change the interpreter Pants uses, update this value in your `pants.ini`.
Some notes when choosing an interpreter:
- Ensure you have this Python version accessible on your PATH everywhere Pants is used in your organization.
- Pants will require Python 3.6+ to run as of release 1.17.0.
pants.ini. Use this to set $PYTHON and determine bootstrap_venv.This could probably be done in a new setup script recommended to users over the old.
Released pexes are hosted on github: https://github.com/pantsbuild/pants/releases/tag/release_1.9.0.dev1
I was trying to automate some Pants updates and was looking to use this invocation to update the pants script itself:
curl --proto "=https" \
--tlsv1.3 \
--location \
--verbose \
--output ./pants \
https://pantsbuild.github.io/setup/pants
However, when I run that, it appears that at some point in the flow, I get redirected to an http site:
HTTP/2 301
< server: GitHub.com
< content-type: text/html
< permissions-policy: interest-cohort=()
< location: http://v1.pantsbuild.org/setup/pants
< x-github-request-id: 4DDA:0AD7:24AE8:607545:61F8148F
< accept-ranges: bytes
< date: Mon, 31 Jan 2022 16:57:01 GMT
< via: 1.1 varnish
< age: 78
< x-served-by: cache-fty21360-FTY
< x-cache: HIT
< x-cache-hits: 1
< x-timer: S1643648221.348955,VS0,VE1
< vary: Accept-Encoding
< x-fastly-request-id: de662d2ef47f018f97d6677537bf7d7ea081cf13
< content-length: 162
<
* Ignoring the response-body
{ [162 bytes data]
100 162 100 162 0 0 2793 0 --:--:-- --:--:-- --:--:-- 2842
* Connection #0 to host pantsbuild.github.io left intact
* Issue another request to this URL: 'http://v1.pantsbuild.org/setup/pants'
* Protocol "http" not supported or disabled in libcurl
* Closing connection -1
Right now the script only provides (clunky) support for pantsbuild.pants at a version. It needs to support easy specification of a list of requirements for installs that want plugins found outside of the main pantsbuild.pants sdist.
I don't seem to be able to bootstrap pants on RHEL 7.4 . Below are steps to reproduce and the output I get when running the bootstrap script. Gcc keeps giving no such file or directory and fails to build many of the wheels, and then it appears that perhaps a virtualenv is not created properly with which to run pants, as the final error is ./pants: line 102: /home/ec2-user/.cache/pants/setup/bootstrap-Linux-x86_64/unspecified/bin/python: No such file or directory
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)
$ curl https://raw.githubusercontent.com/pantsbuild/setup/gh-pages/pants > ./pants && bash ./pants
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3711 100 3711 0 0 15186 0 --:--:-- --:--:-- --:--:-- 15209
New python executable in /home/ec2-user/.cache/pants/setup/bootstrap-Linux-x86_64/pants.zaFQYS/install/bin/python2.7
Also creating executable in /home/ec2-user/.cache/pants/setup/bootstrap-Linux-x86_64/pants.zaFQYS/install/bin/python
Installing pip, wheel...done.
Collecting setuptools==5.4.1
Using cached setuptools-5.4.1-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-5.4.1
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pantsbuild.pants
Collecting pex==1.2.6 (from pantsbuild.pants)
Using cached pex-1.2.6-py2.py3-none-any.whl
Collecting twitter.common.dirutil<0.4,>=0.3.1 (from pantsbuild.pants)
Collecting fasteners==0.14.1 (from pantsbuild.pants)
Using cached fasteners-0.14.1-py2.py3-none-any.whl
Collecting packaging==16.8 (from pantsbuild.pants)
Using cached packaging-16.8-py2.py3-none-any.whl
Collecting pathspec==0.5.0 (from pantsbuild.pants)
Collecting futures==3.0.5 (from pantsbuild.pants)
Using cached futures-3.0.5-py2-none-any.whl
Collecting ansicolors==1.0.2 (from pantsbuild.pants)
Collecting Markdown==2.1.1 (from pantsbuild.pants)
Collecting twitter.common.collections<0.4,>=0.3.1 (from pantsbuild.pants)
Collecting scandir==1.2 (from pantsbuild.pants)
Using cached scandir-1.2.zip
Collecting pywatchman==1.3.0 (from pantsbuild.pants)
Using cached pywatchman-1.3.0.tar.gz
Collecting psutil==4.3.0 (from pantsbuild.pants)
Using cached psutil-4.3.0.tar.gz
Collecting pystache==0.5.3 (from pantsbuild.pants)
Collecting docutils<0.13,>=0.12 (from pantsbuild.pants)
Collecting requests<2.6,>=2.5.0 (from pantsbuild.pants)
Using cached requests-2.5.3-py2.py3-none-any.whl
Collecting cffi==1.7.0 (from pantsbuild.pants)
Using cached cffi-1.7.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting six<2,>=1.9.0 (from pantsbuild.pants)
Using cached six-1.11.0-py2.py3-none-any.whl
Collecting Pygments==1.4 (from pantsbuild.pants)
Collecting twitter.common.confluence<0.4,>=0.3.1 (from pantsbuild.pants)
Collecting setuptools==30.0.0 (from pantsbuild.pants)
Using cached setuptools-30.0.0-py2.py3-none-any.whl
Collecting setproctitle==1.1.10 (from pantsbuild.pants)
Using cached setproctitle-1.1.10.tar.gz
Collecting twitter.common.lang==0.3.9 (from twitter.common.dirutil<0.4,>=0.3.1->pantsbuild.pants)
Collecting monotonic>=0.1 (from fasteners==0.14.1->pantsbuild.pants)
Using cached monotonic-1.3-py2.py3-none-any.whl
Collecting pyparsing (from packaging==16.8->pantsbuild.pants)
Using cached pyparsing-2.2.0-py2.py3-none-any.whl
Collecting pycparser (from cffi==1.7.0->pantsbuild.pants)
Collecting twitter.common.log==0.3.9 (from twitter.common.confluence<0.4,>=0.3.1->pantsbuild.pants)
Collecting twitter.common.options==0.3.9 (from twitter.common.log==0.3.9->twitter.common.confluence<0.4,>=0.3.1->pantsbuild.pants)
Building wheels for collected packages: scandir, pywatchman, psutil, setproctitle
Running setup.py bdist_wheel for scandir ... error
Complete output from command /home/ec2-user/.cache/pants/setup/bootstrap-Linux-x86_64/pants.zaFQYS/install/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-jr5Pmm/scandir/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmpW80rNtpip-wheel- --python-tag cp27:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
copying scandir.py -> build/lib.linux-x86_64-2.7
running build_ext
building '_scandir' extension
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c _scandir.c -o build/temp.linux-x86_64-2.7/_scandir.o
unable to execute gcc: No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for scandir
Running setup.py clean for scandir
Running setup.py bdist_wheel for pywatchman ... error
Complete output from command /home/ec2-user/.cache/pants/setup/bootstrap-Linux-x86_64/pants.zaFQYS/install/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-jr5Pmm/pywatchman/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmp5uFcjHpip-wheel- --python-tag cp27:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/pywatchman
copying pywatchman/__init__.py -> build/lib.linux-x86_64-2.7/pywatchman
copying pywatchman/capabilities.py -> build/lib.linux-x86_64-2.7/pywatchman
copying pywatchman/pybser.py -> build/lib.linux-x86_64-2.7/pywatchman
running build_ext
building 'pywatchman.bser' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/pywatchman
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c pywatchman/bser.c -o build/temp.linux-x86_64-2.7/pywatchman/bser.o
unable to execute gcc: No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for pywatchman
Running setup.py clean for pywatchman
Running setup.py bdist_wheel for psutil ... error
Complete output from command /home/ec2-user/.cache/pants/setup/bootstrap-Linux-x86_64/pants.zaFQYS/install/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-jr5Pmm/psutil/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmpe2BWJXpip-wheel- --python-tag cp27:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/psutil
copying psutil/__init__.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_compat.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_common.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_psosx.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_pssunos.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_psbsd.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_psposix.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_pslinux.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_pswindows.py -> build/lib.linux-x86_64-2.7/psutil
creating build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/__init__.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_misc.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_system.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_windows.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_osx.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_process.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/runner.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_linux.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_bsd.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_memory_leaks.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_sunos.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_posix.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_testutils.py -> build/lib.linux-x86_64-2.7/psutil/tests
running build_ext
building 'psutil._psutil_linux' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/psutil
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DPSUTIL_VERSION=430 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
unable to execute gcc: No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for psutil
Running setup.py clean for psutil
Running setup.py bdist_wheel for setproctitle ... error
Complete output from command /home/ec2-user/.cache/pants/setup/bootstrap-Linux-x86_64/pants.zaFQYS/install/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-jr5Pmm/setproctitle/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmptLX3rJpip-wheel- --python-tag cp27:
running bdist_wheel
running build
running build_ext
building 'setproctitle' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DHAVE_SYS_PRCTL_H=1 -DSPT_VERSION=1.1.10 -I/usr/include/python2.7 -c src/setproctitle.c -o build/temp.linux-x86_64-2.7/src/setproctitle.o
unable to execute gcc: No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for setproctitle
Running setup.py clean for setproctitle
Failed to build scandir pywatchman psutil setproctitle
Installing collected packages: setuptools, pex, twitter.common.lang, twitter.common.dirutil, six, monotonic, fasteners, pyparsing, packaging, pathspec, futures, ansicolors, Markdown, twitter.common.collections, scandir, pywatchman, psutil, pystache, docutils, requests, pycparser, cffi, Pygments, twitter.common.options, twitter.common.log, twitter.common.confluence, setproctitle, pantsbuild.pants
Found existing installation: setuptools 5.4.1
Uninstalling setuptools-5.4.1:
Successfully uninstalled setuptools-5.4.1
Running setup.py install for scandir ... error
Complete output from command /home/ec2-user/.cache/pants/setup/bootstrap-Linux-x86_64/pants.zaFQYS/install/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-jr5Pmm/scandir/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-VFjRoC-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/ec2-user/.cache/pants/setup/bootstrap-Linux-x86_64/pants.zaFQYS/install/include/site/python2.7/scandir:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
copying scandir.py -> build/lib.linux-x86_64-2.7
running build_ext
building '_scandir' extension
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c _scandir.c -o build/temp.linux-x86_64-2.7/_scandir.o
unable to execute gcc: No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/home/ec2-user/.cache/pants/setup/bootstrap-Linux-x86_64/pants.zaFQYS/install/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-jr5Pmm/scandir/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-VFjRoC-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/ec2-user/.cache/pants/setup/bootstrap-Linux-x86_64/pants.zaFQYS/install/include/site/python2.7/scandir" failed with error code 1 in /tmp/pip-build-jr5Pmm/scandir/
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
./pants: line 102: /home/ec2-user/.cache/pants/setup/bootstrap-Linux-x86_64/unspecified/bin/python: No such file or directory
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.