pex-tool / pex Goto Github PK

View Code? Open in Web Editor NEW

2.5K 2.5K 254.0 10.93 MB

A tool for generating .pex (Python EXecutable) files, lock files and venvs.

Home Page: https://docs.pex-tool.org

License: Apache License 2.0

Python 99.75% Shell 0.18% Dockerfile 0.07%

pex's People

Contributors

Stargazers

Watchers

Forkers

wickman jsirois kamilchm zyga yasumoto pfmoore adamtheturtle rhysparry rouge8 jmdcal moult ugodiggi sholsapp warsaw kecaps mikekap pombredanne ewdurbin mbachry kostyll msabramo jamesbroadhead yasoob lorencarvalho jumaphelix rayleyva imoapps davinirjr lexigram steveniemitz kwlzn jdanbrown baroquebobcat gogobook huskyeder linearregression sethp-jive landism malloc47 digwanderlust houzz davefnbuck jjjake guoshimin ewang katielucas-grapeshot pikeas kirklg mzdaniel adamchainz epu billonahill brbsix lvzhichang smorin clam-gpsw pjonnala-eab come-maiz spfjr eliotwinchell justinvenus dan-blanchard mkai yolken cevaris toumorokoshi ldkingvivi datawire olivierh59500 burtgulash sveitser kxepal butlern benjyw toddgardner autotraderuk snyaggarwal thoughtworksinc rowhit peterebden thesamet cheister cuulee kennydo fvinas borgstrom georgi-yanchev zvezdan kkoralsky amylavara cmlivingston jimi-c t-cas markyen unrememberme mgedigian-fitbit wei-n-ning nmost ttreptow narensh

pex's Issues

Enable HTTP retries

Often pex will fail to resolve a dependency because it will fail fetch to a package within a deadline. This will cause a build to fail with a twitter.common.python.resolver.Unsatisfiable exception. It would be nice if pex could retry fetching the package so a bit of network latency will not break the build.

Implement extras parsing for `ResolvablePackages`

From the review in https://rbcommons.com/s/twitter/r/2044

default caching in the pex tool is incorrect

if you do pex --repo=/tmp/path/to/my/repo -r req and that requirement gets pulled into $PEX_ROOT/build, subsequent calls to 'pex -r req' will resolve to the version in $PEX_ROOT/build if 'req' is ahead of the PyPI tip. this is unexpected default behavior. the pex tool should only do this if --cache/--cache-ttl are both specified.

Can we make pex read from a requirements.txt file instead of specifying each requirement separately?

start using pep440 identifiers.

specifically switch to using dstufft's packaging package to manage version numbers instead of pkg_resources.Requirement.

change PEX_INTERPRETER to PEX_REPL

this is to allow PEX_INTERPRETER to be specified to select a specific interpreter class. alternately we call it PEX_INTERPRETER_SPEC and keep PEX_INTERPRETER.

Adjust setup.py's install_requires

setup.py limits setuptools in its install_requires to >=2.2,<8. This breaks the build of pex on Ubuntu 15.04 where we have setuptools 12.2. It'll break on Debian too after Jessie is released.

I added the following very simple patch, which so far seems to fix the problem (more testing ahead).

--- a/setup.py
+++ b/setup.py
@@ -36,7 +36,7 @@
     'pex.bin',
   ],
   install_requires = [
-    'setuptools>=2.2,<8',
+    'setuptools>=2.2',
   ],
   tests_require = [
     'mock',

FEATURE - Add ability to specify major version of python for PEX to run with

When I have created a PEX it recorded the interpreter as python3.3. becasue that was the version we
had installed.
But we have recently installed python 3.4.1 and the question is that if the python 3.3 will get removed, then all PEX files built with py33 will break becasue of the "hardcoded" python interpreter value with minor version.

I thought if I'd use --python option and specify interperter as "python3" it would allow to use/run PEX with any python3 version that is present in the $PATH but it doesn't seem to be the case.

Please add the support for major version of python specification (python3) that would effectively use first python3 found in the $PATH

pex --python seems to ignore the selected interpreter path

% ls -l /usr/bin/python3
lrwxrwxrwx 1 root root 9 Dec  1 00:12 /usr/bin/python3 -> python3.4*
% pex -v --python=/usr/bin/python3 -o world.pex -r world --no-wheel -e worldlib.__main__:main
pex: Resolving distributions: 517.2ms          
pex:     Packaging world: 354.3ms
  world 3.1
Setting entry point to worldlib.__main__:main
Saving PEX file to world.pex
% head -1 world.pex
#!/usr/bin/env python3.4

It doesn't seem to matter if --python=/usr/bin/python3.4 is used (i.e. it's not related to python3 being a symlink).

Can't pass 'cwd' to pex.run()

cwd is an option for subprocess.Popen, but the way pants.run() is written, you can't override it.

I don't really need this feature, I just ran into it when trying to call it from pants with cwd=None.

Link Travis CI build badge in README

https://travis-ci.org/pantsbuild/pex

pex resolver crashes when using local cache for certain transitive requirements

packages_from_requirement_cached doesn't check if everything in the existing= kwarg is local, which means it may attempt to os.stat a remote file in certain circumstances.

mba=pex=; pex -r pytest -r setuptools -r py==1.4.25 -o /tmp/pt.pex --cache-ttl=3600 -v -v -v -v -v
pex: Package cache hit (inexact): pytest        
pex: Package cache hit (inexact): setuptools        
pex: Package cache miss: py==1.4.25                 
pex: Resolving distributions :: Fetching https://pypi.python.org/packages/source/p/py/py-1.4.25.tar.gz#md5pex: Validated py-1.4.25.tar.gz (md5=1f0b84dab1681fc1bf07c0528e091efb)                                                                     
pex: Validated py-1.4.25.tar.gz (md5=1f0b84dab1681fc1bf07c0528e091efb)
pex: Resolving distributions :: Packaging py :: Installing /var/folders/rd/_tjz8zts3g14md1kmf38z6w80000gn/pex: Resolving distributions :: Packaging pytest                                                          pex: Resolving distributions :: Packaging pytest :: Installing /var/folders/rd/_tjz8zts3g14md1kmf38z6w8000pex: Resolving distributions :: Resolving py>=1.4.25                                                      Traceback (most recent call last):
  File "/Users/wickman/Local/bin/pex/.bootstrap/_pex/pex.py", line 272, in execute
    self.execute_entry(entry_point, args)
  File "/Users/wickman/Local/bin/pex/.bootstrap/_pex/pex.py", line 320, in execute_entry
    runner(entry_point)
  File "/Users/wickman/Local/bin/pex/.bootstrap/_pex/pex.py", line 343, in execute_pkg_resources
    runner()
  File "/Users/wickman/.pex/install/pex-0.8.0.dev0-py2.py3-none-any.whl.0d5be588022f0aad6de2b84b77a8a890c43ad942/pex-0.8.0.dev0-py2.py3-none-any.whl/pex/bin/pex.py", line 325, in main
    pex_builder = build_pex(args, options)
  File "/Users/wickman/.pex/install/pex-0.8.0.dev0-py2.py3-none-any.whl.0d5be588022f0aad6de2b84b77a8a890c43ad942/pex-0.8.0.dev0-py2.py3-none-any.whl/pex/bin/pex.py", line 303, in build_pex
    cache_ttl=options.cache_ttl)
  File "/Users/wickman/.pex/install/pex-0.8.0.dev0-py2.py3-none-any.whl.0d5be588022f0aad6de2b84b77a8a890c43ad942/pex-0.8.0.dev0-py2.py3-none-any.whl/pex/resolver.py", line 215, in resolve
    existing=distribution_set.get(requirement.key))
  File "/Users/wickman/.pex/install/pex-0.8.0.dev0-py2.py3-none-any.whl.0d5be588022f0aad6de2b84b77a8a890c43ad942/pex-0.8.0.dev0-py2.py3-none-any.whl/pex/resolver.py", line 94, in packages_from_requirement_cached
    packages = [package for package in packages if (now - os.path.getmtime(package.path)) < ttl]
  File "/Users/wickman/Python/CPython-2.7.8/lib/python2.7/genericpath.py", line 54, in getmtime
    return os.stat(filename).st_mtime
OSError: [Errno 2] No such file or directory: '/packages/source/p/py/py-1.4.25.tar.gz'

Add timeout support when fetching requirements.

It would be useful if pex allowed for a configurable timeout when downloading requirements.

--allow-external/--allow-all-external doesn't work

Looks like pypi changed the way it tags external packages. Now you can have a direct download link from pypi.python.org/simple, just that it's tagged with an 'ext-download' tag. We need to filter ext-download packages if follow_links is False.

no errors keyword for decode in python 2.6

Attempting to run pex on Python 2.6.8 fails. I had to add -v to see the error, otherwise it just says it can't resolve the requirement.

[root@08b6ba92511a ~]# pex --python=/usr/bin/python2.6 -r fabric -v
pex: Unknown exception encountered: decode() takes no keyword arguments
pex: Traceback (most recent call last):
pex:   File "/usr/lib/python2.6/site-packages/pex/crawler.py", line 110, in execute
pex:     roots, rels = self.crawl_link(self.context, link)
pex:   File "/usr/lib/python2.6/site-packages/pex/crawler.py", line 87, in crawl_link
pex:     return cls.crawl_remote(context, link)
pex:   File "/usr/lib/python2.6/site-packages/pex/crawler.py", line 74, in crawl_remote
pex:     content = context.content(link)
pex:   File "/usr/lib/python2.6/site-packages/pex/http.py", line 122, in content
pex:     return fp.read().decode(encoding, errors='replace')
pex: TypeError: decode() takes no keyword arguments
Traceback (most recent call last):
  File "/usr/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.6/site-packages/pex/bin/pex.py", line 325, in main
    pex_builder = build_pex(args, options)
  File "/usr/lib/python2.6/site-packages/pex/bin/pex.py", line 303, in build_pex
    cache_ttl=options.cache_ttl)
  File "/usr/lib/python2.6/site-packages/pex/resolver.py", line 220, in resolve
    raise Unsatisfiable('Cannot satisfy requirements: %s' % requirement_set[requirement.key])
pex.resolver.Unsatisfiable: Cannot satisfy requirements: [Requirement.parse('fabric')]

Release pex 0.8.4

Thanks to a nudge from @Moult, we should tag a new release and ship to PyPI.

Allow selecting different interpreter at runtime in a PEX

PythonInterpreter.replace supports runtime replacement of interpreters. We should plumb through a way e.g. via the environment to allow users to select a specific interpreter.

Create a way to merge PEX environments at runtime

e.g. create PEX_PATH that is the analogue to PYTHONPATH and do something like:

pex -r pantsbuild.pants -e pants.bin.pants_exe:main -o pants.pex
pex -r pantsbuild.plugin1 -r pantsbuild.plugin2 -r ... -o pants_plugins.pex
PEX_PATH=$PEX_PATH:pants_plugins.pex ./pants.pex

and have all the distributions from pants_plugins.pex mixed into pants.pex (providing that there are no conflicts.)

PyPI tar.gz is missing some files

The PyPI tar.gz is missing some files which are useful for downstream packagers (e.g. Debian).

The entire docs/ directory (.md should be .rst, etc.)
LICENSE
rbt (may or may not be useful)
scripts/
tests/
tox.ini

There might be others, but that's as far as I've gotten so far. Probably just need an updated MANIFEST.in and I'll add PR if I can get it working.

unhandled AttributeError during pex bootstrapping

looks like bootstrapping might be broken in the current version of pex:

>>> from pex import pex_bootstrapper
>>> pex_bootstrapper.bootstrap_pex_env('/home/kwilson/test_flask_wsgi.pex')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/kwilson/.pex/install/pex-0.8.4-py2.py3-none-any.whl.92fa056b8fc3f5f9ade278207ae2d42306515ec8/pex-0.8.4-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 80, in bootstrap_pex_env
    PEXEnvironment(entry_point, PexInfo.from_pex(entry_point)).activate()
  File "/home/kwilson/.pex/install/pex-0.8.4-py2.py3-none-any.whl.92fa056b8fc3f5f9ade278207ae2d42306515ec8/pex-0.8.4-py2.py3-none-any.whl/pex/environment.py", line 134, in activate
    self._working_set = self._activate()
  File "/home/kwilson/.pex/install/pex-0.8.4-py2.py3-none-any.whl.92fa056b8fc3f5f9ade278207ae2d42306515ec8/pex-0.8.4-py2.py3-none-any.whl/pex/environment.py", line 140, in _activate
    self.update_candidate_distributions(self.load_internal_cache(self._pex, self._pex_info))
  File "/home/kwilson/.pex/install/pex-0.8.4-py2.py3-none-any.whl.92fa056b8fc3f5f9ade278207ae2d42306515ec8/pex-0.8.4-py2.py3-none-any.whl/pex/environment.py", line 124, in update_candidate_distributions
    if self.can_add(dist):
  File "/home/kwilson/.pex/install/pex-0.8.4-py2.py3-none-any.whl.92fa056b8fc3f5f9ade278207ae2d42306515ec8/pex-0.8.4-py2.py3-none-any.whl/pex/environment.py", line 129, in can_add
    return distribution_compatible(dist, self._interpreter, self.platform)
  File "/home/kwilson/.pex/install/pex-0.8.4-py2.py3-none-any.whl.92fa056b8fc3f5f9ade278207ae2d42306515ec8/pex-0.8.4-py2.py3-none-any.whl/pex/package.py", line 247, in distribution_compatible
    package = Package.from_href(dist.location)
AttributeError: 'NoneType' object has no attribute 'location'

pip does not install extras_requires for console_scripts

This seems like a bug in pip, however it manifests itself in a broken pex tool if you 'pip install pex' into an empty virtualenv:

mba=pex=; virtualenv dist/pex_ve
New python executable in dist/pex_ve/bin/python2.7
Also creating executable in dist/pex_ve/bin/python
Installing setuptools, pip...done.

mba=pex=; source dist/pex_ve/bin/activate

(pex_ve)mba=pex=; pip install dist/pex-0.8.7-dev0.tar.gz 
Unpacking ./dist/pex-0.8.7-dev0.tar.gz
  Running setup.py (path:/var/folders/rd/_tjz8zts3g14md1kmf38z6w80000gn/T/pip-hYkm9F-build/setup.py) egg_info for package from file:///Users/wickman/clients/pex/dist/pex-0.8.7-dev0.tar.gz

    warning: no files found matching 'docs/*.md'
Requirement already satisfied (use --upgrade to upgrade): setuptools>=2.2,<8 in ./dist/pex_ve/lib/python2.7/site-packages (from pex==0.8.7-dev0)
Installing collected packages: pex
  Running setup.py install for pex

    warning: no files found matching 'docs/*.md'
    Installing pex script to /Users/wickman/clients/pex/dist/pex_ve/bin
Successfully installed pex
Cleaning up...

(pex_ve)mba=pex=; pip list
pex (0.8.7-dev0)
pip (1.5.6)
setuptools (3.6)
wsgiref (0.1.2)

(pex_ve)mba=pex=; pex --help
Traceback (most recent call last):
  File "/Users/wickman/clients/pex/dist/pex_ve/bin/pex", line 9, in <module>
    load_entry_point('pex==0.8.7-dev0', 'console_scripts', 'pex')()
  File "/Users/wickman/clients/pex/dist/pex_ve/lib/python2.7/site-packages/pkg_resources.py", line 356, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Users/wickman/clients/pex/dist/pex_ve/lib/python2.7/site-packages/pkg_resources.py", line 2431, in load_entry_point
    return ep.load()
  File "/Users/wickman/clients/pex/dist/pex_ve/lib/python2.7/site-packages/pkg_resources.py", line 2145, in load
    if require: self.require(env, installer)
  File "/Users/wickman/clients/pex/dist/pex_ve/lib/python2.7/site-packages/pkg_resources.py", line 2159, in require
    items = working_set.resolve(reqs, env, installer)
  File "/Users/wickman/clients/pex/dist/pex_ve/lib/python2.7/site-packages/pkg_resources.py", line 639, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: wheel>=0.24.0,<0.25.0

In the meantime we should remove the extras_requires from pex. This means in order to build pexes with wheels (the default option) we will need an interpreter cache #47.

1.0 release backwards incompatibilities

This is a proposed list of backwards incompatible changes that will be introduced in 1.0 and is subject to change.

For the pex tool:

pex -r will take requirements.txt instead of requirement specifiers (e.g. "Flask>=0.9")
requirement specifiers will be arguments to pex
Further pip-compatible flag aliases such as -f/--find-links will be added
--python will be used to specify an exact python shebang

For pex invocation:

PEX_INTERPRETER changed to accept interpreter class, e.g. 'CPython==2.7.9' or 'PyPy'
new variable PEX_REPL will be used to drop into a REPL
new variable PEX_PYTHON will be used to specify the exact interpreter to invoke and will override PEX_INTERPRETER

AttributeError: 'UUID' object has no attribute 'get_hex'

Was trying to follow the pex introduction, and after the $ pex -r flask -p flask.pex I had an exception:

Traceback (most recent call last):
  File "/home/nsukami/envs/pex/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/home/nsukami/envs/pex/lib/python3.4/site-packages/pex/bin/pex.py", line 310, in main
    pex_builder = build_pex(args, options)
  File "/home/nsukami/envs/pex/lib/python3.4/site-packages/pex/bin/pex.py", line 277, in build_pex
    platform=options.platform)
  File "/home/nsukami/envs/pex/lib/python3.4/site-packages/pex/resolver.py", line 110, in resolve
    new_requirements.update(requires(highest_package, requirement))
  File "/home/nsukami/envs/pex/lib/python3.4/site-packages/pex/resolver.py", line 82, in requires
    dist = obtainer.obtain(package)
  File "/home/nsukami/envs/pex/lib/python3.4/site-packages/pex/obtainer.py", line 156, in obtain
    dist = super(CachingObtainer, self).obtain(req_or_package)
  File "/home/nsukami/envs/pex/lib/python3.4/site-packages/pex/obtainer.py", line 106, in obtain
    return self._translate_from([req_or_package])
  File "/home/nsukami/envs/pex/lib/python3.4/site-packages/pex/obtainer.py", line 94, in _translate_from
    dist = self._translator.translate(package)
  File "/home/nsukami/envs/pex/lib/python3.4/site-packages/pex/translator.py", line 43, in translate
    dist = tx.translate(package)
  File "/home/nsukami/envs/pex/lib/python3.4/site-packages/pex/translator.py", line 112, in translate
    target_path_tmp = target_path + uuid4().get_hex()
AttributeError: 'UUID' object has no attribute 'get_hex'

This is my virtualenv:

 > pip list
pex (0.7.0)
pip (1.5.6)
setuptools (3.6)
twitter.common.python (0.7.0)
wheel (0.24.0)

Thanks for any suggestions on how to fix this.

Failure in running setup.py for a dependency leads to confusing error

Traceback (most recent call last):
  File "/ernestd/src/pex/dist/pex/.bootstrap/_pex/pex.py", line 271, in execute
  File "/ernestd/src/pex/dist/pex/.bootstrap/_pex/pex.py", line 320, in execute_entry
  File "/ernestd/src/pex/dist/pex/.bootstrap/_pex/pex.py", line 350, in execute_pkg_resources
  File "/Users/edurbin/.pex/install/pex-1.0.0.dev0-py2.py3-none-any.whl.26c70bed807e428f3c64ac665ab11a241a8c0d08/pex-1.0.0.dev0-py2.py3-none-any.whl/pex/bin/pex.py", line 467, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/Users/edurbin/.pex/install/pex-1.0.0.dev0-py2.py3-none-any.whl.26c70bed807e428f3c64ac665ab11a241a8c0d08/pex-1.0.0.dev0-py2.py3-none-any.whl/pex/bin/pex.py", line 436, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/Users/edurbin/.pex/install/pex-1.0.0.dev0-py2.py3-none-any.whl.26c70bed807e428f3c64ac665ab11a241a8c0d08/pex-1.0.0.dev0-py2.py3-none-any.whl/pex/resolver.py", line 154, in resolve
    distributions[package] = self.build(package, resolvable.options)
  File "/Users/edurbin/.pex/install/pex-1.0.0.dev0-py2.py3-none-any.whl.26c70bed807e428f3c64ac665ab11a241a8c0d08/pex-1.0.0.dev0-py2.py3-none-any.whl/pex/resolver.py", line 203, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/Users/edurbin/.pex/install/pex-1.0.0.dev0-py2.py3-none-any.whl.26c70bed807e428f3c64ac665ab11a241a8c0d08/pex-1.0.0.dev0-py2.py3-none-any.whl/pex/resolver.py", line 124, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
  File "/Users/edurbin/.pex/install/pex-1.0.0.dev0-py2.py3-none-any.whl.26c70bed807e428f3c64ac665ab11a241a8c0d08/pex-1.0.0.dev0-py2.py3-none-any.whl/pex/translator.py", line 50, in __str__
    return 'ChainedTranslator(%s)' % (', '.join(map(repr, tx) for tx in self._translators))
  File "/Users/edurbin/.pex/install/pex-1.0.0.dev0-py2.py3-none-any.whl.26c70bed807e428f3c64ac665ab11a241a8c0d08/pex-1.0.0.dev0-py2.py3-none-any.whl/pex/translator.py", line 50, in <genexpr>
    return 'ChainedTranslator(%s)' % (', '.join(map(repr, tx) for tx in self._translators))
TypeError: argument 2 to map() must support iteration

index urls without trailing / are improperly handled

given https://user:[email protected]/simple pex 1.0.0dev0 fails to construct the appropriate url for packages in the index.

Expected:

--extra-index-url https://user:[email protected]/simple
my-private-dep==1.0.0

should search https://user:[email protected]/simple/my-private-dep for version matching 1.0.0

instead, it searches https://user:[email protected]/my-private-dep

Release pex 0.8.6

A patch is in with a fix for handling of custom sys.excepthook's and this is needed to resolve: pantsbuild/pants#1026

There is no rush on this, the issue is just for help tracking pantsbuild/pants work.

Make --platform/--interpreter multi-options

This is to make it easier to build multi-platform pex files a la pants.

pex --version provides an incorrect version

In the cli, the displayed version is hard-coded to version 0.2.

See: https://github.com/pantsbuild/pex/blob/master/pex/bin/pex.py#L54

implement an interpreter cache

Pants implements an interpreter cache: a set of known-working interpreters along with cached and built dependencies necessary for certain features of pex to work (e.g. setuptools, wheel, etc.) Right now you can't build pexes with wheels if your interpreter does not have wheel. Pants is smart enough to fetch and build this for you -- this functionality should be within pex instead.

Parse 'Requires-Dist' markers

A current bug with pex is that it will resolve dependencies for the packages in the python environment of the pex program, not the target python environment of the output pex.

For example if pex is running under Python 2.7, it will end up not adding argparse to the final pex even if the user specifies a target of Python 2.6.

To remedy this we need to extract the Requires-Dist metadata out of every distribution we are going to add. This metadata (described in PEP 345) contains entries like argparse; python_version == '2.6'. If we take this metadata, pass it to the target interpreter, let the target interpreter evaluate the metadata and let the target interpreter decide if we need to add the requirement for the distribution or not.

The easiest way to implement this is to first change the resolution code to read the Requires-Dist metadata and then pass that data to the target interpreter, similar to how we evaluate the target interpreter version.

wheel `safer_name` for locating scripts

fe895b2#diff-3ffb3bc5eb80cbb36374158a2dbbc433R251 uses dist.key but should use pkg_resources.to_filename(dist.key) or similar for listing directories within the wheel.

add a PEX-INFO metadata versioning scheme

We should add a metadata versioning so that we can at least know whether we expect to be able to read/parse metadata from a certain generation.

pex can pull in a bad pkg_resources

There are two non-namespace packages on pypi that share the 'pants' namespace: pants and pantsbuild.pants. If they're both installed in your environment or virtualenv, importing pkg_resources will stacktrace. This means that pex won't work in that situation.

I'm not sure which version of pkg_resources is broken in particular, but I know at least the one bundled in setuptools 12.2 does not suffer the following problem, so we should pin pex to something sufficiently new to avoid this:

# vendored pkg_resources
(ve)mba=ve=; PYTHONPATH=/Users/wickman/.tools-cache/home/mesos/tools/client/libexec/aurora-client/.bootstrap python
Python 2.7.8 (default, Nov  3 2014, 10:09:47) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/wickman/.tools-cache/home/mesos/tools/client/libexec/aurora-client/.bootstrap/pkg_resources.py", line 2736, in <module>
    add_activation_listener(lambda dist: dist.activate())
  File "/Users/wickman/.tools-cache/home/mesos/tools/client/libexec/aurora-client/.bootstrap/pkg_resources.py", line 698, in subscribe
    callback(dist)
  File "/Users/wickman/.tools-cache/home/mesos/tools/client/libexec/aurora-client/.bootstrap/pkg_resources.py", line 2736, in <lambda>
    add_activation_listener(lambda dist: dist.activate())
  File "/Users/wickman/.tools-cache/home/mesos/tools/client/libexec/aurora-client/.bootstrap/pkg_resources.py", line 2274, in activate
    list(map(declare_namespace, self._get_metadata('namespace_packages.txt')))
  File "/Users/wickman/.tools-cache/home/mesos/tools/client/libexec/aurora-client/.bootstrap/pkg_resources.py", line 1882, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/Users/wickman/.tools-cache/home/mesos/tools/client/libexec/aurora-client/.bootstrap/pkg_resources.py", line 1849, in _handle_ns
    loader.load_module(packageName)
  File "/Users/wickman/Python/CPython-2.7.8/lib/python2.7/pkgutil.py", line 246, in load_module
    mod = imp.load_module(fullname, self.file, self.filename, self.etc)
  File "/private/tmp/ve/lib/python2.7/site-packages/pants/__init__.py", line 31, in <module>
    from pants._channel import HAS_IPV6, HAS_UNIX
  File "/private/tmp/ve/lib/python2.7/site-packages/pants/_channel.py", line 34, in <module>
    from pants.engine import Engine
ImportError: cannot import name Engine
>>> 
(ve)mba=ve=; 


# new pkg_resources
(ve)mba=ve=; python
Python 2.7.8 (default, Nov  3 2014, 10:09:47) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>>

Package data disappears from pex file.

Hey all,

Sorry if this is the wrong venue to report this but I couldn't find any contact information for the group. I'm happy to report elsewhere if this is the wrong venue.

I'm finding that my package data is disappearing from my pex file even though the wheels that I'm using build my pex file do have the package data included in them. In case it matters, I'm just trying to package up a Flask application with a few Jinja2 templates. I'm using Python 2.6. Here is the process I'm following to produce this problem.

First, I build a wheel for my project.

$ pip --version
pip 6.0.8 from /example-webapp_trunk/example-webapp/build/venv/lib/python2.6/site-packages (python 2.6)
$ pip wheel --no-deps .
Processing /example-webapp_trunk/example-webapp
Building wheels for collected packages: example-webapp
  Running setup.py bdist_wheel for example-webapp
  Destination directory: /example-webapp_trunk/example-webapp/wheelhouse
Successfully built example-webapp

Second, I verified that the wheel I built has my package data using unzip.

(example-webapp)[sholsapp@sholsapp-ld example-webapp]$ unzip -l dist/wheel-cache/example_webapp-0.0.12-py2-none-any.whl
Archive:  dist/wheel-cache/example_webapp-0.0.12-py2-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
     4138  04-03-2015 14:55   examplewebapp/__init__.py
     1629  04-03-2015 14:55   examplewebapp/view_source.py
       67  04-03-2015 14:55   examplewebapp/gunicorn_conf.py
     6430  04-03-2015 14:55   examplewebapp/templates/rickshaw_example.html
     3648  04-03-2015 14:55   examplewebapp/templates/form_example.html
      105  04-03-2015 14:55   examplewebapp/templates/logged_in.html
      307  04-03-2015 14:55   examplewebapp/templates/form_success.html
     2985  04-03-2015 14:55   examplewebapp/templates/layout.html
    15235  04-03-2015 14:55   examplewebapp/templates/index.html
       10  04-06-2015 14:17   example_webapp-0.0.12.dist-info/DESCRIPTION.rst
      129  04-06-2015 14:17   example_webapp-0.0.12.dist-info/entry_points.txt
      614  04-06-2015 14:17   example_webapp-0.0.12.dist-info/metadata.json
       22  04-06-2015 14:17   example_webapp-0.0.12.dist-info/top_level.txt
       92  04-06-2015 14:17   example_webapp-0.0.12.dist-info/WHEEL
      225  04-06-2015 14:17   example_webapp-0.0.12.dist-info/METADATA
     3569  04-06-2015 14:17   example_webapp-0.0.12.dist-info/RECORD
---------                     -------
    92014                     36 files

I also checked the RECORD file to make sure it also captured my package data.

(example-webapp)[sholsapp@sholsapp-ld wheelhouse]$ cat example_webapp-0.0.12.dist-info/RECORD 
examplewebapp/__init__.py,sha256=Xu4qPwrcP1XdPdrAu34N_yvFZV_xWfq2hpPVh-W99wA,4138
examplewebapp/view_source.py,sha256=11OjrdUI4ArGVB3fer3aBVjN7zj-36EDz2zh-s3ZO0M,1629
examplewebapp/gunicorn_conf.py,sha256=oEl1dOYBHN6nPf1bfCfWIBCevdR4pWiIp-eaDmnGRGM,67
examplewebapp/templates/rickshaw_example.html,sha256=ZG64KMioGVHZ0J5n5QUmkMy5wL-1PjKd3h1EOhxoS4g,6430
examplewebapp/templates/form_example.html,sha256=vMMxNfuzdH9v1TF8UgG-Z-cUfOfmC5cTlSCxAC9JR9M,3648
examplewebapp/templates/logged_in.html,sha256=tXKsNhiuphHfdTZFAJYuzTobMin573pMBAsDeuTqTHg,105
examplewebapp/templates/form_success.html,sha256=wKk0lj_66QXm9QXiPrmajIT84V1ecb6yakLnwT197tk,307
examplewebapp/templates/layout.html,sha256=tU3MPyQrBO8n_HFIq06sL9okcz6YYv8kMETxZs33EhA,2985
examplewebapp/templates/index.html,sha256=7z2wIztX7-YManTSMQt_6oa7cLK5LWEZWvDpMo4CW0A,15235
example_webapp-0.0.12.dist-info/metadata.json,sha256=DZomPXYYEFCdhfsQj1cF8Bz7jdPrm3s91BBEhtQsg1Q,614
example_webapp-0.0.12.dist-info/entry_points.txt,sha256=OetkMrLMgTyjklJrKQABLAEA84xGcnnOMlTSVUAc2qI,129
example_webapp-0.0.12.dist-info/DESCRIPTION.rst,sha256=OCTuuN6LcWulhHS3d5rfjdsQtW22n7HENFRh6jC6ego,10
example_webapp-0.0.12.dist-info/WHEEL,sha256=54bVun1KfEBTJ68SHUmbxNPj80VxlQ0sHi4gZdGZXEY,92
example_webapp-0.0.12.dist-info/METADATA,sha256=IJsnJ4DSiLUdiKY8NiJR3QnykYUjk2HSsymLzXVtuc8,225
example_webapp-0.0.12.dist-info/RECORD,,
example_webapp-0.0.12.dist-info/top_level.txt,sha256=aIxHYhOnMuCpK4efLE7qdosLNdg1ibmx5y9AxNn-Vxs,22

So, I have a wheel that has my package data, now it's time to create a pex file. I've built my wheel cache locally and don't use PyPI. I removed all the steps that create those packages.

$ pex --version
pex 0.8.5
(example-webapp)[sholsapp@sholsapp-ld example-webapp]$ pex -r example-webapp --no-pypi --repo=dist/wheel-cache/ -o test.zip

I inspect the output test.zip pex file using unzip.

$ unzip -l test.zip 
Archive:  test.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
      552  04-06-2015 14:20   .bootstrap/_markerlib/__init__.py
     1571  04-06-2015 14:20   .bootstrap/_markerlib/__init__.pyc
     3979  04-06-2015 14:20   .bootstrap/_markerlib/markers.py
     6303  04-06-2015 14:20   .bootstrap/_markerlib/markers.pyc
      131  04-06-2015 14:20   .bootstrap/_pex/__init__.py
     1428  04-06-2015 14:20   .bootstrap/_pex/archiver.py
     1128  04-06-2015 14:20   .bootstrap/_pex/base.py
     8996  04-06-2015 14:20   .bootstrap/_pex/common.py
     2248  04-06-2015 14:20   .bootstrap/_pex/compatibility.py
     4288  04-06-2015 14:20   .bootstrap/_pex/crawler.py
     6708  04-06-2015 14:20   .bootstrap/_pex/environment.py
     1259  04-06-2015 14:20   .bootstrap/_pex/fetcher.py
     7712  04-06-2015 14:20   .bootstrap/_pex/finders.py
     7553  04-06-2015 14:20   .bootstrap/_pex/http.py
     7616  04-06-2015 14:20   .bootstrap/_pex/installer.py
    13015  04-06-2015 14:20   .bootstrap/_pex/interpreter.py
     2323  04-06-2015 14:20   .bootstrap/_pex/iterator.py
     3425  04-06-2015 14:20   .bootstrap/_pex/link.py
     2329  04-06-2015 14:20   .bootstrap/_pex/marshaller.py
     2121  04-06-2015 14:20   .bootstrap/_pex/orderedset.py
     8210  04-06-2015 14:20   .bootstrap/_pex/package.py
     5155  04-06-2015 14:20   .bootstrap/_pex/pep425.py
    13526  04-06-2015 14:20   .bootstrap/_pex/pex.py
     2313  04-06-2015 14:20   .bootstrap/_pex/pex_bootstrapper.py
    14087  04-06-2015 14:20   .bootstrap/_pex/pex_builder.py
     8285  04-06-2015 14:20   .bootstrap/_pex/pex_info.py
     2387  04-06-2015 14:20   .bootstrap/_pex/platforms.py
    10386  04-06-2015 14:20   .bootstrap/_pex/resolver.py
     5363  04-06-2015 14:20   .bootstrap/_pex/testing.py
     4128  04-06-2015 14:20   .bootstrap/_pex/tracer.py
     6205  04-06-2015 14:20   .bootstrap/_pex/translator.py
     5813  04-06-2015 14:20   .bootstrap/_pex/util.py
       22  04-06-2015 14:20   .bootstrap/_pex/version.py
    99996  04-06-2015 14:20   .bootstrap/pkg_resources.py
       10  04-06-2015 14:20   .deps/example_webapp-0.0.12-py2-none-any.whl/example_webapp-0.0.12.dist-info/DESCRIPTION.rst
      225  04-06-2015 14:20   .deps/example_webapp-0.0.12-py2-none-any.whl/example_webapp-0.0.12.dist-info/METADATA
      985  04-06-2015 14:20   .deps/example_webapp-0.0.12-py2-none-any.whl/example_webapp-0.0.12.dist-info/RECORD
       92  04-06-2015 14:20   .deps/example_webapp-0.0.12-py2-none-any.whl/example_webapp-0.0.12.dist-info/WHEEL
      129  04-06-2015 14:20   .deps/example_webapp-0.0.12-py2-none-any.whl/example_webapp-0.0.12.dist-info/entry_points.txt
      614  04-06-2015 14:20   .deps/example_webapp-0.0.12-py2-none-any.whl/example_webapp-0.0.12.dist-info/metadata.json
       22  04-06-2015 14:20   .deps/example_webapp-0.0.12-py2-none-any.whl/example_webapp-0.0.12.dist-info/top_level.txt
     4138  04-06-2015 14:20   .deps/example_webapp-0.0.12-py2-none-any.whl/examplewebapp/__init__.py
       67  04-06-2015 14:20   .deps/example_webapp-0.0.12-py2-none-any.whl/examplewebapp/gunicorn_conf.py
     1630  04-06-2015 14:20   .deps/example_webapp-0.0.12-py2-none-any.whl/examplewebapp/view_source.py
      410  04-06-2015 14:20   PEX-INFO
      737  04-06-2015 14:20   __main__.py
---------                     -------
   279619                     46 files

Blast!

Somewhere along the way I lost my package data (i.e., the templates directory). I can't seem to figure out what I'm doing wrong since I have the sdists, eggs and wheel distributions all packaging up my package data seemingly correctly. It seems only when I create the pex file are my data files lost.

Any ideas?

The `setup_requires` are not searched in cache

I'm trying to build a pex archive and I have all the package dependencies local but it looks like setup_requires are not checked in cache, but rather installed from pypi.

I'm trying to build a pex with pygit2 like this: pex -v --no-pypi --cache-dir=/tmp/build/packages/ -r 'pygit2', but I'm getting this error. The build machine has no internet connection:

pex.http: Crawling /tmp/build/packages/: 0.5ms
**** Failed to install pygit2-0.21.3. stdout:OKgW2

**** Failed to install pygit2-0.21.3. stderr:
Download error on https://pypi.python.org/simple/cffi/: [Errno 111] Connection refused -- Some packages may not be found!
Couldn't find index page for 'cffi' (maybe misspelled?)
Download error on https://pypi.python.org/simple/: [Errno 111] Connection refused -- Some packages may not be found!
No local packages or download links found for cffi
Traceback (most recent call last):
  File "<stdin>", line 6, in <module>
  File "setup.py", line 215, in <module>
    cmdclass=cmdclass)
  File "/usr/lib/python2.7/distutils/core.py", line 112, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 239, in __init__
    self.fetch_build_eggs(attrs.pop('setup_requires'))
  File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 264, in fetch_build_eggs
    replace_conflicting=True
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in best_match
    return self.obtain(req, installer) # try and download/install
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 870, in obtain
    return installer(requirement)
  File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 314, in fetch_build_egg
    return cmd.easy_install(req)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 610, in easy_install
    raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('cffi')

pex: Packaging pygit2: 88.0ms
pex:   Installing /tmp/tmp7OKgW2: 86.8ms
Traceback (most recent call last):
  File "/usr/bin/pex", line 9, in <module>
    load_entry_point('pex==0.7.0', 'console_scripts', 'pex')()
  File "/usr/lib/python2.7/dist-packages/pex/bin/pex.py", line 310, in main
    pex_builder = build_pex(args, options)
  File "/usr/lib/python2.7/dist-packages/pex/bin/pex.py", line 277, in build_pex
    platform=options.platform)
  File "/usr/lib/python2.7/dist-packages/pex/resolver.py", line 101, in resolve
    raise Unsatisfiable('Cannot satisfy requirements: %s' % requirement_set[requirement.key])
pex.resolver.Unsatisfiable: Cannot satisfy requirements: [Requirement.parse('pygit2==0.21.3')]

The contents of /tmp/build/packages/ is:

/tmp/build/packages/cffi-0.8.6.tar.gz
/tmp/build/packages/pycparser-2.10.tar.gz
/tmp/build/packages/pygit2-0.21.3.tar.gz

"can't use a string pattern on a bytes-like object" on python 3.4

Running pex on python 3.4 produces the following error, without further details:

pex: Unknown exception encountered: can't use a string pattern on a bytes-like object

Upon hitting C-c, pex emits the following traceback:

Traceback (most recent call last):
  File "ve/bin/pex", line 9, in <module>
    load_entry_point('pex==0.8.0-rc0', 'console_scripts', 'pex')()
  File "/home/vagrant/ve/lib/python3.4/site-packages/pex/bin/pex.py", line 308, in main
    pex_builder = build_pex(args, options)
  File "/home/vagrant/ve/lib/python3.4/site-packages/pex/bin/pex.py", line 279, in build_pex
    cache_ttl=options.cache_ttl)
  File "/home/vagrant/ve/lib/python3.4/site-packages/pex/resolver.py", line 215, in resolve
    existing=distribution_set.get(requirement.key))
  File "/home/vagrant/ve/lib/python3.4/site-packages/pex/resolver.py", line 101, in packages_from_requirement_cached
    return packages_from_requirement(iterator, requirement, *args, **kw)
  File "/home/vagrant/ve/lib/python3.4/site-packages/pex/resolver.py", line 69, in packages_from_requirement
    return [package for package in existing
  File "/home/vagrant/ve/lib/python3.4/site-packages/pex/resolver.py", line 69, in <listcomp>
    return [package for package in existing
  File "/home/vagrant/ve/lib/python3.4/site-packages/pex/iterator.py", line 57, in iter
    for package in self._sort(self._iter_unordered(req, follow_links)):
  File "/home/vagrant/ve/lib/python3.4/site-packages/pex/iterator.py", line 53, in _sort
    return sorted(package_list, key=key, reverse=True)
  File "/home/vagrant/ve/lib/python3.4/site-packages/pex/iterator.py", line 46, in _iter_unordered
    crawled_url_iterator = self._crawler.crawl(url_iterator, follow_links=follow_links)
  File "/home/vagrant/ve/lib/python3.4/site-packages/pex/crawler.py", line 132, in crawl
    queue.join()
  File "/home/vagrant/.local/lib/python3.4/queue.py", line 86, in join
    self.all_tasks_done.wait()
  File "/home/vagrant/.local/lib/python3.4/threading.py", line 290, in wait
    waiter.acquire()
KeyboardInterrupt

upgrade pex to a modern version of setuptools

newer versions of setuptools have changed pretty dramatically. pkg_resources is now a package and version identifiers adhere to PEP440. this should probably coincide with pex 1.0 as it's a major backwards incompatible change.

allow specifying a script to run in lieu of an entry point

If you build a simple flask app but don't want to go through the ceremony of defining a setup.py and the like, you should be able to just do 'pex -r flask_requirements.txt --entry_script=my_flask.py -o my_flask.pex' rather than have to keep the two separate.

doc PEX_* env vars

A question came up on a pants-devel thread for which the answer was "use PEX_ROOT". https://groups.google.com/forum/#!topic/pants-devel/HKgKqUIUUIk . To explain it, JSirois had to link to two places: a docstring that mentions the pex_root property and the pex_root-property code that checks the env for PEX_ROOT. I dunno how often questions come up around these; if the answer is "often", then it probably makes sense to describe them in a doc somewhere.

'PEX_ALWAYS_CACHE'
'PEX_COVERAGE'
'PEX_FORCE_LOCAL'
'PEX_INHERIT_PATH'
'PEX_INTERPRETER'
'PEX_MODULE'
'PEX_PROFILE'
'PEX_PROFILE_ENTRIES'
'PEX_PROFILE_SORT'
'PEX_ROOT'
'PEX_TEARDOWN_VERBOSE'
'PEX_VERBOSE'

ensure .pyc files are always written into the .pex

Right now .pyc files are not always written for every .py (e.g. some __init__.py and files within .egg and .whl directories.) In order to speed up the invocation of directory PEXes owned by superuser (and hence where .pyc cannot be locally cached) we should always make sure that we're building these. An example: the aurora client as directory PEX speeds up from 1.4s to 600ms when all .pyc's are present.

We used to build .pyc using the CodeMarshaller in twitter.common.python but stopped once multi-interpreter support was added, as we could no longer rely upon the current interpreter to produce correct .pyc.

The solution will involve bolting a compile() method onto PythonInterpreter that forks out a stub compile script. We'd also need to consider doing this for multiple interpreters in case we want to support multi-interpreter pex files. I believe starting in py3 they will build pep-425 tagged pyc, e.g. '.py33.pyc' so we could potentially compile multiply tagged pycs.

– @wickman on our internal issue-tracker

raised pex.resolver.Untranslateable when trying to use redis library

Tested with --python set to python 2.7.9 and 3.4.2
$ pex -r redis==2.10.3 # Also it happens with -r redis
Traceback (most recent call last):
File "/usr/bin/pex", line 11, in
sys.exit(main())
File "/usr/lib/python2.7/site-packages/pex/bin/pex.py", line 325, in main
pex_builder = build_pex(args, options)
File "/usr/lib/python2.7/site-packages/pex/bin/pex.py", line 303, in build_pex
cache_ttl=options.cache_ttl)
File "/usr/lib/python2.7/site-packages/pex/resolver.py", line 229, in resolve
new_requirements.update(requires(highest_package, requirement))
File "/usr/lib/python2.7/site-packages/pex/resolver.py", line 201, in requires
raise Untranslateable('Package %s is not translateable.' % package)
pex.resolver.Untranslateable: Package SourcePackage('file:///home/user/.pex/build/redis-2.10.3.tar.gz') is not translateable.

pex -e should support console_scripts pointers

pex -e should be changed to use console_scripts entry points. this will allow for pex -e pex -r pex, where pex = pex.bin.pex:main [whl], hence picking up the whl extra_requires.

pex -m should be added to supplant the prior -e functionality
(do this on a 1.0 since this is a major user-facing compatibility break)

Tag releases

All public releases should be tagged so they show up here:
https://github.com/pantsbuild/pex/releases

Error module object has no attribute unsetenv

This is installing pex into a clean Python 3.4 virtualenv on Windows 7 64-bit.

To create the test environment:

> virtualenv ve
> ve\scripts\activate
> cd ve
> pip install wheel
> pip wheel pex
> pip install -f wheelhouse pex

The error is:

>pex
Traceback (most recent call last):
  File "C:\Apps\Python34\Lib\runpy.py", line 171, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Apps\Python34\Lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Work\Scratch\ve\Scripts\pex.exe\__main__.py", line 9, in <module>
  File "C:\Work\Scratch\ve\lib\site-packages\pex\bin\pex.py", line 327, in main
    return pex.run(args=list(args))
  File "C:\Work\Scratch\ve\lib\site-packages\pex\pex.py", line 322, in run
    self.clean_environment(forking=True)
  File "C:\Work\Scratch\ve\lib\site-packages\pex\pex.py", line 53, in clean_environment
    os.unsetenv('MACOSX_DEPLOYMENT_TARGET')
AttributeError: 'module' object has no attribute 'unsetenv'

It looks like os.unsetenv is not available on Windows since Python 3. Deleting values from os.environ via del os.environ[key] is probably more portable.

Filter prerelease versions in package iterators by default

Pip has the --pre flag -- we should allow similar behavior.

PEX doesn't scrub "local" from virtualenv site-packages

In recent versions of virtualenv there's a "local" directory that just contains symlinks

% ./build-support/virtualenv --version
1.11.6

% ls -l build-support/pants.venv/local
total 12
lrwxrwxrwx 1 ksweeney ksweeney 60 Oct 22 16:03 bin -> /home/ksweeney/workspace/aurora/build-support/pants.venv/bin
lrwxrwxrwx 1 ksweeney ksweeney 64 Oct 22 16:03 include -> /home/ksweeney/workspace/aurora/build-support/pants.venv/include
lrwxrwxrwx 1 ksweeney ksweeney 60 Oct 22 16:03 lib -> /home/ksweeney/workspace/aurora/build-support/pants.venv/lib

PEX doesn't scrub this local site-packages from sys.path, causing pants dependencies to shadow pex dependencies:

Minimal test case:

def test_psutil():
  import sys
  print('\n'.join(sys.path))
  import psutil
  assert psutil.__version__ == '2.1.1'

Test output:

/home/ksweeney/workspace/aurora/src/test/python/apache/aurora/executor
/tmp/tmpV7InVa/.bootstrap
/tmp/tmpV7InVa
/home/ksweeney/workspace/aurora/build-support/pants.venv/lib/python2.7
/home/ksweeney/workspace/aurora/build-support/pants.venv/lib/python2.7/plat-x86_64-linux-gnu
/home/ksweeney/workspace/aurora/build-support/pants.venv/lib/python2.7/lib-tk
/home/ksweeney/workspace/aurora/build-support/pants.venv/lib/python2.7/lib-old
/home/ksweeney/workspace/aurora/build-support/pants.venv/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/home/ksweeney/workspace/aurora/build-support/pants.venv/local/lib/python2.7/site-packages
/tmp/tmpV7InVa/.deps/unittest2-0.5.1-py2-none-any.whl
/tmp/tmpV7InVa/.deps/pytest_cov-1.8.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/pytest_timeout-0.4-py2.py3-none-any.whl
/tmp/tmpV7InVa/.deps/pytest-2.6.3-py2-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.string-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.quantity-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.python-0.5.4-py2-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.metrics-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.exceptions-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/psutil-2.1.1-cp27-none-linux_x86_64.whl
/tmp/tmpV7InVa/.deps/twitter.common.contextutil-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/py-1.4.25-py2.py3-none-any.whl
/tmp/tmpV7InVa/.deps/cov_core-1.14.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.lang-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/wheel-0.24.0-py2.py3-none-any.whl
/tmp/tmpV7InVa/.deps/setuptools-7.0-py2.py3-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.decorators-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.dirutil-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/coverage-4.0a1-cp27-none-linux_x86_64.whl
F

============================================== FAILURES ==============================================
____________________________________________ test_psutil _____________________________________________

    def test_psutil():
      import sys
      print('\n'.join(sys.path))
      import psutil
>     assert psutil.__version__ == '2.1.1'
E     assert '1.1.3' == '2.1.1'
E       - 1.1.3
E       + 2.1.1

As a workaround, removing the "local" directory makes isolation work correctly:

~aurora git aurora/. kts/upgrade-virtualenv  U  
% rm -fr build-support/pants.venv/local                                     
~aurora git aurora/. kts/upgrade-virtualenv  U  
% ./pants src/test/python/apache/aurora/executor:executor_vars -s -k test_psutil
src.test.python.apache.aurora.executor.executor_vars                            .....   SUCCESS

pex.resolver.Untranslateable when using setup from setuptools

Hello,

I am trying to build a pex file with several requirements and my own package that has a setup.py.

From reading pex --help, I understand that I need to use the -s to provide the directory that contains my setup.py file.

I get this error:

pex -s . -e greatpackage.script -o output.pex
**** Failed to install greatpackage-0.1. stdout:

**** Failed to install greatpackage-0.1. stderr:
Traceback (most recent call last):
  File "<stdin>", line 6, in <module>
IOError: [Errno 2] No such file or directory: 'setup.py'

Traceback (most recent call last):
  File "/Users/bqm/garbage/reproducible/env/bin/pex", line 9, in <module>
    load_entry_point('pex==0.8.4', 'console_scripts', 'pex')()
  File "/Users/bqm/garbage/reproducible/env/lib/python2.7/site-packages/pex/bin/pex.py", line 325, in main
    pex_builder = build_pex(args, options)
  File "/Users/bqm/garbage/reproducible/env/lib/python2.7/site-packages/pex/bin/pex.py", line 303, in build_pex
    cache_ttl=options.cache_ttl)
  File "/Users/bqm/garbage/reproducible/env/lib/python2.7/site-packages/pex/resolver.py", line 229, in resolve
    new_requirements.update(requires(highest_package, requirement))
  File "/Users/bqm/garbage/reproducible/env/lib/python2.7/site-packages/pex/resolver.py", line 201, in requires
    raise Untranslateable('Package %s is not translateable.' % package)
pex.resolver.Untranslateable: Package SourcePackage('file:///Users/bqm/.pex/build/greatpackage-0.1.tar.gz') is not translateable.

Here is a way to reproduce my error:

mkdir reproducible
cd reproducible
virtualenv env
. env/bin/activate
pip install wheel
pip install pex
mkdir greatpackage
echo '
from setuptools import setup, find_packages

setup(
    name="greatpackage",
    version="0.1",
    author="bqm",
    packages=find_packages(),
    scripts=["greatpackage/script.py"]
)' > setup.py

touch greatpackage/__init__.py

echo '
if __name__ =="__main__":
  print "Hello world!"
' > greatpackage/script.py

pex -s . -e greatpackage.script -o output.pex

Am I missing something? What am I doing wrong?
Thanks

pex -s option does not pull in transitive requirements

Drop support for urllib

This would mean either making requests an explicit dependency, or vendoring it. There's a bunch of stuff that requests gives us that is just plain smart (verify=True, more sensible retry policies, etc.) Right now we abstract away the smarts which makes it hard to implement features like #26.

doc += a pex can contain artifacts from mult platforms, can work on mult platforms

At the risk of revealing how dense I can be: I didn't know you could build a pex that contained both foo-macosx.egg and foo-linux.egg that would work on both osx and linux. I assumed you built a pex per platform.

The excellent @Yasumoto on the thread https://groups.google.com/forum/#!topic/pants-devel/9yeI4h9xu7k says "Ideally you'll have one pex, which includes dependencies for each of the platforms you need to support..." and I was enlightened.