pip issue in ghdl/vunit:gcc-master (and others?) about docker HOT 6 CLOSED

from docker.

So, the issue seems to be the following warning:

Using legacy 'setup.py install' for vunit-hdl, since package 'wheel' is not installed.

I now added wheel to the VUnit images (afb90b6). It seems that the problem was solved for "stable" images, but not for *-master:

# docker run --rm -it ghdl/vunit:mcode pip3 list
Package    Version
---------- -------
colorama   0.4.3
pip        20.2.2
setuptools 50.0.0
vunit-hdl  4.4.0
wheel      0.35.1

# docker run --rm -it ghdl/vunit:mcode-master pip3 list
Package    Version
---------- -------
pip        20.2.2
setuptools 50.0.0
wheel      0.35.1

# docker run --rm -it ghdl/vunit:llvm pip3 list
Package    Version
---------- -------
colorama   0.4.3
pip        20.2.2
setuptools 50.0.0
vunit-hdl  4.4.0
wheel      0.35.1

# docker run --rm -it ghdl/vunit:llvm-master pip3 list
Package    Version
---------- -------
pip        20.2.2
setuptools 50.0.0
wheel      0.35.1

# docker run --rm -it ghdl/vunit:gcc pip3 list
Package    Version
---------- -------
colorama   0.4.3
gcovr      4.2
Jinja2     2.11.2
lxml       4.5.2
MarkupSafe 1.1.1
pip        20.2.2
setuptools 50.0.0
vunit-hdl  4.4.0
wheel      0.35.1

# docker run --rm -it ghdl/vunit:gcc-master pip3 list
Package    Version
---------- -------
gcovr      4.2
Jinja2     2.11.2
lxml       4.5.2
MarkupSafe 1.1.1
pip        20.2.2
setuptools 50.0.0
wheel      0.35.1

That is, pip3 install vunit_hdl works now, but python3 setup.py install is still failing.

Unfortunately, I really don't understand the details of pip packages and the several mechanisms to install Python packages. Any guess?

I tried the same thing in the python:3-slim-buster (upon which ghdl/vunit:gcc-master is based?

No, ghdl/vunit:* images are based on ghdl/ghdl:buster-* images, which are based on debian:buster-slim.

OTOH, image python:3-slim-buster is used as a base for ghdl/ext:ls-debian. So, ghdl/ext:ls-vunit is based indirectly on python:3-slim-buster. It seems that wheel is/was installed in python:3-slim-buster, by default. Surprisingly, although python3 setup.py install is used (https://github.com/ghdl/docker/blob/master/gui.dockerfile#L25), it works:

# docker run --rm -it ghdl/ext:ls-vunit pip3 list
Package    Version
---------- --------
attrs      20.1.0
colorama   0.4.3
pip        20.2.2
pyghdl     1.0.dev0
setuptools 49.3.1
vunit-hdl  4.4.1rc0
wheel      0.34.2

from docker.

Some more info: when installed through python3 setup.py install, VUnit and colorama are indeed installed in /usr/lib/python3.7/site-packages/:

# ls /usr/lib/python3.7/site-packages/
colorama-0.4.3-py3.7.egg/     easy-install.pth              vunit_hdl-4.4.1rc0-py3.7.egg/

# cat /usr/lib/python3.7/site-packages/easy-install.pth
./vunit_hdl-4.4.1rc0-py3.7.egg
./colorama-0.4.3-py3.7.egg

But pip seems not to find those. Moreover, /usr/lib/python3/dist-packages/pip exists.

Conversely, in ghdl/ext:ls-vunit:

/# ls /usr/local/lib/python3.8/site-packages/
README.txt              colorama-0.4.3-py3.8.egg  pip                        setuptools-49.3.1.dist-info      
__pycache__             distutils-precedence.pth  pip-20.2.2.dist-info       vhdl_langserver
_distutils_hack         easy-install.pth          pkg_resources              vunit_hdl-4.4.1rc0-py3.8.egg     
attr                    easy_install.py           pyghdl-1.0.dev0.dist-info  wheel
attrs-20.1.0.dist-info  libghdl                   setuptools                 wheel-0.34.2.dist-info

from docker.

Now, pip3 install . is used instead of python3 setup.py install. That should fix all the images:

# docker run --rm -it ghdl/vunit:mcode pip3 list
Package    Version
---------- -------
colorama   0.4.3
pip        20.2.2
setuptools 50.0.0
vunit-hdl  4.4.0
wheel      0.35.1

# docker run --rm -it ghdl/vunit:mcode-master pip3 list
Package    Version
---------- --------
colorama   0.4.3
pip        20.2.2
setuptools 50.0.0
vunit-hdl  4.4.1rc0
wheel      0.35.1

# docker run --rm -it ghdl/vunit:llvm pip3 list
Package    Version
---------- -------
colorama   0.4.3
pip        20.2.2
setuptools 50.0.0
vunit-hdl  4.4.0
wheel      0.35.1

# docker run --rm -it ghdl/vunit:llvm-master pip3 list
Package    Version
---------- --------
colorama   0.4.3
pip        20.2.2
setuptools 50.0.0
vunit-hdl  4.4.1rc0
wheel      0.35.1

# docker run --rm -it ghdl/vunit:gcc pip3 list
Package    Version
---------- -------
colorama   0.4.3
gcovr      4.2
Jinja2     2.11.2
lxml       4.5.2
MarkupSafe 1.1.1
pip        20.2.2
setuptools 50.0.0
vunit-hdl  4.4.0
wheel      0.35.1

# docker run --rm -it ghdl/vunit:gcc-master pip3 list
Package    Version
---------- --------
colorama   0.4.3
gcovr      4.2
Jinja2     2.11.2
lxml       4.5.2
MarkupSafe 1.1.1
pip        20.2.2
setuptools 50.0.0
vunit-hdl  4.4.1rc0
wheel      0.35.1

@LukasVik, please, let me know whether it works for you. If so, I think we can close this issue 😃

from docker.

Good morning,

Indeed it does work well now. Thank you so much @eine!

Looking at the issue I still find it a little funny. I'm not an expert at pip, but there must be some mechanism that is broken in there. Where it supposedly installs a package, the files are there in the file tree, but it has not "registered" that it is installed. If pip truly needs wheel in order to install packages, it should signal an error instead of the warning we got.

I googled the issue yesterday and I could not find anyone else with the same issue. And it also does not appear to happen in all python environments. On my ubuntu box I could install vunit_hdl just fine without wheel, and also in the python:3-slim-buster image after doing pip uninstall wheel.

Just to note, I don't think it has anything to do with the vunit_hdl package specifically. I observed the same issue with a python package named wrapt and indeed the tsfpga package.

Anyway, the fix/workaround works and that is all that matters I guess. I feel no need to dig deeper into the issue. Something seems a little sketchy but it works now! Closing this issue. Thanks again @eine!

from docker.

@LukasVik, just for the sake of knowledge, let me drop my thoughts too:

• I think we can blame setuptools. Until two days ago, v49.6.0 was used (https://github.com/ghdl/docker/runs/1048161781?check_suite_focus=true#step:3:337). Yesterday, v50.0.0 was published (https://github.com/ghdl/docker/runs/1049962487?check_suite_focus=true#step:3:337). I don't see an explicit connection with the changes (https://github.com/pypa/setuptools/blob/master/CHANGES.rst), but that seems to be the case: pypa/setuptools#2350, pypa/setuptools#2232.

• My poor understanding is that some commands create packages, others upload them, and others install them.

  • When using tox, it takes care of creating a virtual environment, creating the package and installing it inside. I recently updated VUnit for using pyproject.toml, instead of tox.ini. However, that was two weeks ago.
  • When deploying, we generate an sdist package (which I assume that is NOT installed), and it is pushed with twine: https://github.com/VUnit/vunit/blob/master/.github/workflows/push.yml#L164-L176.

However, I don't understand what's the role of pip there. Also, I am not familiar with easy_install. Using python3 setup.py install seems to be "the easy_install/Egg way of doing things", while pip uses wheel. See https://packaging.python.org/discussions/pip-vs-easy-install/ and https://packaging.python.org/discussions/wheel-vs-egg/. Still, as you say, it is funny that this was working for months/years, and it broke yesterday without an explicit warning. I mean, the warning was shown by tox long before, not by setuptools.

On my ubuntu box I could install vunit_hdl just fine without wheel, and also in the python:3-slim-buster image after doing pip uninstall wheel.

Is it possible that both your Ubuntu box and python:3-slim-buster have Python >=3.8? I'm thinking that this might be an issue with 3.7, 3.6... But I did not carefully read pypa/setuptools issues yet.

from docker.