deepwalker / pundler Goto Github PK

View Code? Open in Web Editor NEW

183.0 183.0 9.0 253 KB

Python bundler-alike alternative to virtualenv

License: BSD 2-Clause "Simplified" License

Python 100.00%

pundler's People

Contributors

Stargazers

Watchers

Forkers

pombredanne weaksauce netpastor discypher jjlorenzo tailhook jayvdb ksimmi alexdoesh

pundler's Issues

`pundle exec` should be similar to `bundle exec`

If I have a requirements.txt with requests in it and try to do

$ python -m pundle exec python
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests

it doesn't work. I see that pundle env does this, but why use a different command? Since pundler's meant to be a bundler clone, shouldn't pundle exec work similarly to bundle exec?

In bundler, if I have a Gemfile with sinatra in it, I can do

$ bundle exec ruby hello.rb

To run the app

Let me know if you think this makes sense, I may be able to contribute a PR

Get requirements from setup.py for packages development.

Parse requirements from setup.py.

thoughts on vendoring

...so that you can vendor packages listed in requirements.txt in the repo itself (~ /vendor) similar to what Bundler, govendor, and many other dependency managers can do.

problem with importing `pip._internal.main` as `pip_exec`

Hello!
Today I started new project with pyenv and python-3.8.0 and I had problems with installation packages via pundle. For example I putted one line pyramid into requirements.txt and then I had an error:

pundle install
Install some packages
try <pip._vendor.distlib.locators.SimpleScrapingLocator object at 0x7fb64044cc10> for pyramid
try <pip._vendor.distlib.locators.JSONLocator object at 0x7fb6404af970> for pyramid
Use temp dir /tmp/tmpdizbrgtx
pip install --no-deps -t /tmp/tmpdizbrgtx https://pypi.python.org/packages/12/e5/8cffc593b70ee6392ea08e8562843166895b96cf33ace892229bcfca410e/pyramid-1.8.3.tar.gz
Traceback (most recent call last):
  File "/home/ksimmi/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pundle.py", line 242, in locate_and_install
    pip_exec([
TypeError: 'module' object is not callable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ksimmi/.pyenv/versions/3.8.0/bin/pundle", line 10, in <module>
    sys.exit(CmdRegister.main())
  File "/home/ksimmi/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pundle.py", line 1027, in main
    cls.commands[alias]()
  File "/home/ksimmi/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pundle.py", line 1033, in cmd_install
    install_all(**create_parser_or_exit())
  File "/home/ksimmi/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pundle.py", line 922, in install_all
    suite.install()
  File "/home/ksimmi/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pundle.py", line 477, in install
    state.reveal_requirements(self, install=install)
  File "/home/ksimmi/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pundle.py", line 334, in reveal_requirements
    dist = self.check_installed_version(suite, install=install)
  File "/home/ksimmi/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pundle.py", line 302, in check_installed_version
    dist = self.requirement.locate_and_install(suite, installed=self.get_installed())
  File "/home/ksimmi/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pundle.py", line 252, in locate_and_install
    raise PundleException('%s was not installed due error %s' % (self.egg or loc_dist.name, exc))
pundle.PundleException: pyramid was not installed due error 'module' object is not callable

When I replaced pyramid line in requirements.txt with django I had the same error. I repeated it with lots of packages with the same result.

I don't know why but It behaves because of how Python imports the main method from pip._internal. The imported method is not a method! The method's type is a module. I fixed this behavior by simple check of type of the imported method's type and if it module I fetching method from it.

Version of pip package by github ref

I need to use the specific version of package referred by github reference id, I used the following record in requirements.txt"

https://github.com/mitsuhiko/jinja2/archive/85820fceb83569df62fa5e6b9b0f2f76b7c6a3cf.zip

But when I use pundler I get exception:

ValueError: ('Expected version spec in', 'https://github.com/mitsuhiko/jinja2/archive/85820fceb83569df62fa5e6b9b0f2f76b7c6a3cf.zip', 'at', '://github.com/mitsuhiko/jinja2/archive/85820fceb83569df62fa5e6b9b0f2f76b7c6a3cf.zip')

The pip's way:

-e git+https://github.com/mitsuhiko/jinja2.git@85820fceb83569df62fa5e6b9b0f2f76b7c6a3cf#egg=jinja2-2.8.0

also won't work.
How can I do it with pundler?

Does pandler handler nested VCS references?

I have three different projects in my private VCS(bitbucket), which have nested dependencies like the following. Also, Let's say project A, B and C depend on the same pypi lib but different versions.

A => B => C

I'd like to have whole dependencies of all the projects, listed in frozen.txt of project A, and find any conflicts among dependent pypi libs of them. Would it be possible to handle this with the current version of Pundler? thanks

`pundle install` generates invalid filename on Windows

It seems that pundler is using a filename containing : that is invalid on Windows:

OSError: [WinError 123] 檔案名稱、目錄名稱或磁碟區標籤語法錯誤。: 'C:\\Users\\Ow
ner\\AppData\\Local\\Temp\\tmp7gjjfpuk\\bin' -> 'C:\\Users\\Owner\\.pundledir\\C
Python-3.7.1-v3.7.1:260ec2c36a\\livereload-2.5.2\\bin'

Windows 7 x64
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit (AMD64)]
pundle 0.9.1

Full log

D:\Dev\comiccrawler>pundle install
Install some packages
try <pip._vendor.distlib.locators.SimpleScrapingLocator object at 0x000000000308
3CF8> for livereload<3,>=2.5.2
Collecting livereload==2.5.2 from https://files.pythonhosted.org/packages/dd/b4/
213daced3ff1b4e02a1f700748e20e9a7481f5bfef57d11ae9babfd4aa2f/livereload-2.5.2-py
2.py3-none-any.whl
  Using cached https://files.pythonhosted.org/packages/dd/b4/213daced3ff1b4e02a1
f700748e20e9a7481f5bfef57d11ae9babfd4aa2f/livereload-2.5.2-py2.py3-none-any.whl
Installing collected packages: livereload
Successfully installed livereload-2.5.2
Traceback (most recent call last):
  File "c:\users\owner\appdata\local\programs\python\python37\lib\shutil.py", li
ne 557, in move
    os.rename(src, real_dst)
OSError: [WinError 123] 檔案名稱、目錄名稱或磁碟區標籤語法錯誤。: 'C:\\Users\\Ow
ner\\AppData\\Local\\Temp\\tmp7gjjfpuk\\bin' -> 'C:\\Users\\Owner\\.pundledir\\C
Python-3.7.1-v3.7.1:260ec2c36a\\livereload-2.5.2\\bin'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\owner\appdata\local\programs\python\python37\lib\site-packages\
pundle.py", line 246, in locate_and_install
    shutil.move(op.join(tmp_dir, item), op.join(target_dir, item))
  File "c:\users\owner\appdata\local\programs\python\python37\lib\shutil.py", li
ne 568, in move
    symlinks=True)
  File "c:\users\owner\appdata\local\programs\python\python37\lib\shutil.py", li
ne 315, in copytree
    os.makedirs(dst)
  File "c:\users\owner\appdata\local\programs\python\python37\lib\os.py", line 2
11, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "c:\users\owner\appdata\local\programs\python\python37\lib\os.py", line 2
11, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "c:\users\owner\appdata\local\programs\python\python37\lib\os.py", line 2
21, in makedirs
    mkdir(name, mode)
NotADirectoryError: [WinError 267] 目錄名稱無效。: 'C:\\Users\\Owner\\.pundledir
\\CPython-3.7.1-v3.7.1:260ec2c36a'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\owner\appdata\local\programs\python\python37\lib\runpy.py", lin
e 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\owner\appdata\local\programs\python\python37\lib\runpy.py", lin
e 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Owner\AppData\Local\Programs\Python\Python37\Scripts\pundle.exe
\__main__.py", line 9, in <module>
  File "c:\users\owner\appdata\local\programs\python\python37\lib\site-packages\
pundle.py", line 1025, in main
    cls.commands[alias]()
  File "c:\users\owner\appdata\local\programs\python\python37\lib\site-packages\
pundle.py", line 1031, in cmd_install
    install_all(**create_parser_or_exit())
  File "c:\users\owner\appdata\local\programs\python\python37\lib\site-packages\
pundle.py", line 908, in install_all
    suite.install()
  File "c:\users\owner\appdata\local\programs\python\python37\lib\site-packages\
pundle.py", line 467, in install
    state.reveal_requirements(self, install=install)
  File "c:\users\owner\appdata\local\programs\python\python37\lib\site-packages\
pundle.py", line 327, in reveal_requirements
    dist = self.check_installed_version(suite, install=install)
  File "c:\users\owner\appdata\local\programs\python\python37\lib\site-packages\
pundle.py", line 298, in check_installed_version
    dist = self.requirement.locate_and_install(suite, installed=self.get_install
ed())
  File "c:\users\owner\appdata\local\programs\python\python37\lib\site-packages\
pundle.py", line 248, in locate_and_install
    raise PundleException('%s was not installed due error %s' % (self.egg or loc
_dist.name, exc))
pundle.PundleException: livereload was not installed due error [WinError 267] 目
錄名稱無效。: 'C:\\Users\\Owner\\.pundledir\\CPython-3.7.1-v3.7.1:260ec2c36a'

Pundler doesn't support pip 10

File "/usr/local/lib/python3.5/dist-packages/pundle.py", line 235, in locate_and_install
    raise PundleException('%s was not installed due error %s' % (self.egg or loc_dist.name, exc))
__main__.PundleException: Jinja2 was not installed due error module 'pip' has no attribute 'main'

In pip 10 function main has been moved into pip._internal.

ipython[notebook]

How to add ipython[notebook] into requirements.txt?

Describe workflow

Would be nice to see how an usual development workflow would look with pundle (as well as some explanations about each pundle command's intended purposes)

Support for hash checks

https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode

Package %s was not installed due some error' % self.key

Hi,
I'm a Ruby developer and I was so excited when I saw the python's Bundler but it is not stable!
I mean when I trying to run pundle install it fails each time in different packages. One of my logs:

ERROR: Could not install packages due to an EnvironmentError.
Traceback (most recent call last):
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 60, in add
    with open(entry_path) as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-tracker-umaz932d/3b45732363e9583efd55b602b7f827c3b46e770223a2f00f06f4d23e'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 345, in run
    resolver.resolve(requirement_set)
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 196, in resolve
    self._resolve_one(requirement_set, req)
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 359, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 307, in _get_abstract_dist_for
    self.require_hashes
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 213, in prepare_linked_requirement
    with self.req_tracker.track(req):
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 94, in track
    self.add(req)
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 68, in add
    with open(entry_path, 'w') as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-tracker-umaz932d/3b45732363e9583efd55b602b7f827c3b46e770223a2f00f06f4d23e'
Cleaning up...
  Removing source in /tmp/pip-req-build-gzs78k8t
Cleaned build tracker '/tmp/pip-req-tracker-umaz932d'
Package markupsafe was not installed due some error
Traceback (most recent call last):
  File "/home/ksimmi/.pyenv/versions/3.7.4/bin/pundle", line 10, in <module>
    sys.exit(CmdRegister.main())
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pundle.py", line 1027, in main
    cls.commands[alias]()
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pundle.py", line 1033, in cmd_install
    install_all(**create_parser_or_exit())
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pundle.py", line 922, in install_all
    suite.install()
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pundle.py", line 477, in install
    state.reveal_requirements(self, install=install)
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pundle.py", line 343, in reveal_requirements
    already_revealed=already_revealed,
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pundle.py", line 473, in adjust_with_req
    state.reveal_requirements(self, install=install, upgrade=upgrade, already_revealed=already_revealed or set())
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pundle.py", line 343, in reveal_requirements
    already_revealed=already_revealed,
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pundle.py", line 473, in adjust_with_req
    state.reveal_requirements(self, install=install, upgrade=upgrade, already_revealed=already_revealed or set())
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pundle.py", line 334, in reveal_requirements
    dist = self.check_installed_version(suite, install=install)
  File "/home/ksimmi/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pundle.py", line 305, in check_installed_version
    raise PundleException('Package %s was not installed due some error' % self.key)
pundle.PundleException: Package markupsafe was not installed due some error

After about 25 reruning of pundle install dependencies were installed fully but before it was fails on differen packages.

My requiremets.txt contains only 4 packages:

sqlalchemy
alembic
nameko
nameko-sqlalchemy

Support for .pth files?

Briefly looking over the code, looks like you're using pkg_resources.Distribution.activate to add packages to sys.path. Wouldn't that break packages that rely on .pth files to be run when site.py run run (excluding the .pth that are generated by setuptools for namespace packages)?

Few examples of such packages:

Of course it could be argued that those cases are not worth supporting

Crash when using VCS URLs

Pundler seems to crash when using a VCS URL. See:

$ pip freeze | grep pundle
You are using pip version 6.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
pundle==0.5.3
$ cat requirements.txt 
Django~=1.4.0
git+https://github.com/GoodCloud/django-longer-username.git@e4e85d7d29b2ba4fdd7937aa76c02e0098516406#egg=longerusername-dev
$ python -m pundle install
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/dpassaro/.local/lib/python2.7/site-packages/pundle.py", line 744, in <module>
    CmdRegister.main()
  File "/home/dpassaro/.local/lib/python2.7/site-packages/pundle.py", line 612, in main
    cls.commands[alias]()
  File "/home/dpassaro/.local/lib/python2.7/site-packages/pundle.py", line 618, in cmd_install
    install_all(**create_parser_or_exit())
  File "/home/dpassaro/.local/lib/python2.7/site-packages/pundle.py", line 494, in install_all
    suite = Parser(*a, **kw).create_suite()
  File "/home/dpassaro/.local/lib/python2.7/site-packages/pundle.py", line 398, in create_suite
    reqs, freezy, diry = self.parse_requirements(), self.parse_frozen(), self.parse_directory()
  File "/home/dpassaro/.local/lib/python2.7/site-packages/pundle.py", line 434, in parse_requirements
    return dict((req.key, req) for req in (CustomReq(line, 'requirements file') for line in requirements))
  File "/home/dpassaro/.local/lib/python2.7/site-packages/pundle.py", line 434, in <genexpr>
    return dict((req.key, req) for req in (CustomReq(line, 'requirements file') for line in requirements))
  File "/home/dpassaro/.local/lib/python2.7/site-packages/pundle.py", line 434, in <genexpr>
    return dict((req.key, req) for req in (CustomReq(line, 'requirements file') for line in requirements))
  File "/home/dpassaro/.local/lib/python2.7/site-packages/pundle.py", line 112, in __init__
    res = parse_vcs_requirement(line)
  File "/home/dpassaro/.local/lib/python2.7/site-packages/pundle.py", line 68, in parse_vcs_requirement
    parsed_url = urlparse(url)
TypeError: 'module' object is not callable

Commenting out the VCS entry (2nd line) of requirements.txt allows python -m pundle install to execute.

Add support to deployment

Discussable option. Actually frozen.txt is a correct way to create virtualenv with actual packages.

Virtualenv is not garbage at all

Virtualenv is sweet. It's so sweet I have no idea why people are even bothering with poetry and pipenv and the like.

I think you should try and qualify your assertion in the readme. If you don't, the only people who will take your package seriously are those who already agree with you.

Disable site-packages?

Do we need to disable site-packages to prevent pollution of our environment by system installed packages? With pyenv this is not actual, but do we need to support debian/ubuntu/centos/whatever system pythons? Unlikely, but need to discuss.

distutils.errors.DistutilsOptionError: can't combine user with prefix, exec_prefix/home, or install_(plat)base

Hello,

I tried to create simple requirements.txt file where there is only one line matrix_client.

~/tmpfs/test  
❯ cat requirements.txt 
matrix_client

~/tmpfs/test  
❯ pundle
Install some packages
try <pip._vendor.distlib.locators.SimpleScrapingLocator object at 0x7fe1c9230610> for matrix_client
Collecting matrix-client==0.3.2 from https://files.pythonhosted.org/packages/6b/0b/65dc841fd3d14e7ebc6081bbfce23365a6b2f68cc6ae2ae2d1d7d59570cd/matrix_client-0.3.2-py2.py3-none-any.whl
  Using cached https://files.pythonhosted.org/packages/6b/0b/65dc841fd3d14e7ebc6081bbfce23365a6b2f68cc6ae2ae2d1d7d59570cd/matrix_client-0.3.2-py2.py3-none-any.whl
Installing collected packages: matrix-client
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 360, in run
    prefix=options.prefix_path,
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 247, in move_wheel_files
    prefix=prefix,
  File "/usr/lib/python2.7/dist-packages/pip/locations.py", line 153, in distutils_scheme
    i.finalize_options()
  File "/usr/lib/python2.7/distutils/command/install.py", line 289, in finalize_options
    raise DistutilsOptionError("can't combine user with prefix, "
DistutilsOptionError: can't combine user with prefix, exec_prefix/home, or install_(plat)base
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/kepi/.local/lib/python2.7/site-packages/pundle.py", line 1250, in <module>
    CmdRegister.main()
  File "/home/kepi/.local/lib/python2.7/site-packages/pundle.py", line 1025, in main
    cls.commands[alias]()
  File "/home/kepi/.local/lib/python2.7/site-packages/pundle.py", line 1031, in cmd_install
    install_all(**create_parser_or_exit())
  File "/home/kepi/.local/lib/python2.7/site-packages/pundle.py", line 908, in install_all
    suite.install()
  File "/home/kepi/.local/lib/python2.7/site-packages/pundle.py", line 467, in install
    state.reveal_requirements(self, install=install)
  File "/home/kepi/.local/lib/python2.7/site-packages/pundle.py", line 327, in reveal_requirements
    dist = self.check_installed_version(suite, install=install)
  File "/home/kepi/.local/lib/python2.7/site-packages/pundle.py", line 299, in check_installed_version
    self.frozen = dist.version
AttributeError: 'NoneType' object has no attribute 'version'

System is Debian Linux (Unstable), python is installed from distribution. Same problem is with python3.

It is probably some problem with distutils configuration or something like that. I tried to search if I had some user configuration changes but I didn't find any.

I found this: https://stackoverflow.com/questions/4495120/combine-user-with-prefix-error-with-setup-py-install but creating that config file didn't help and more, I'm not sure what will be other sideeffects.

Is it dead?

This project is dead?

build_prefix and src_prefix not in pip.locations anymore

It seems that build_prefix and src_prefix have disappeared from pip.locations (pip 8.1.2)

File ".../3.5/bin/pundler", line 9, in <module>
    from pip.locations import build_prefix, src_prefix
ImportError: cannot import name 'build_prefix'

As a quick fix, I copy and pasted the part of the locations.py that was defining those variables

Support for standalone python executables

I would like to be able to have standalone python executable files relying on pundle. The idea would be to be able to start them directly - pundle would take care of downloading/installing dependencies and set the Python path accordingly.

I am unsure how this could be achieved, but if we agree on a specification for this can be found I can start to work on it.

A first solution would be to create a new command, e.g. standalone and add embed the requirements in the standalone file

#!/usr/bin/env pundle standalone

# requirements
# icolor
# ...
# /requirements

Pundle could then use a special cache to store the frozen state instead of storing it in the same directory to avoid clashes.