uwsgi fails to build/install about python-runtime HOT 20 CLOSED

Disabling LTO seems to fix it. So it looks like a bug in uwsgi, since configuring Python to build --with-lto is a valid option.

$ docker run -it --entrypoint=/bin/bash gcr.io/google-appengine/python:latest
$ virtualenv -p python3.5 /env
$ . /env/bin/activate
(env) $ LDFLAGS=-fno-lto pip install uwsgi
Successfully installed uwsgi-2.0.15
(env) $ /env/bin/uwsgi --python-version
3.5.2

from python-runtime.

This is in Python 3, FYI.

from python-runtime.

Same error on 1) Older versions of uwsgi (2.0.15, .14, .10, and .0), and 2) Older releases of the runtime image.

from python-runtime.

Possibly related to compiling the Python3.5 interpreter with link-time optimization and profile-guided optimization.

from python-runtime.

from python-runtime.

I can't think of a good fix or workaround on our end. There doesn't seem to be a way to pass LDFLAGS inside a requirements.txt, and honestly I was surprised that doing what I did worked.

The GCC command line doesn't match what I expect, so it's not clear if this is a bug in uwsgi, or setuptools, or Python (for passing the wrong ld flags).

I will look a bit more.

CC @gpshead

from python-runtime.

@duggelz I wonder if we should consider not compiling with LTO? Do the debian interpreters do so?

from python-runtime.

The problem here appears to be uwsgi. It has a very complicated build process and is much more than just an extension module.

Yes, Debian builds it's python3.5 package with LTO.

There are a couple other things its rules file does such as objcopy -p --remove-section=.gnu.lto_.* on the static library (.a) files as well as modifying CFLAGS to have -flto -fuse-linker-plugin and sometimes -ffat-lto-objects.

Caveat: I'm just making quick investigation comments, I haven't dug into how any of this actually works so take my comments as side notes to ponder, I don't know which if any of this is related to the build problems.

from python-runtime.

Data points: Installation works fine with Docker's python image, and also works if I start with Debian 9, "apt-get install python3.5" and "pip3 install uwsgi". So maybe it's on our end.

$ docker run -it --entrypoint=/bin/bash python:3.5
root@4e2d887d5c03:/# pip -V
pip 9.0.1 from /usr/local/lib/python3.5/site-packages (python 3.5)
root@4e2d887d5c03:/# pip install uwsgi
Successfully installed uwsgi-2.0.15

$ docker run -it --entrypoint=/bin/bash debian:9
root@f8faf8282ad1:/# apt-get -q update && apt-get -q install python3.5 python3-pip
root@f8faf8282ad1:/# pip3 -V
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.5)
root@f8faf8282ad1:/# pip3 install uwsgi
Successfully installed uwsgi-2.0.15

from python-runtime.

should we just drop lto?

from python-runtime.

from python-runtime.

I successfully got this all to work by updating to Python 3.5.3, which changed how the lto flags are handled.

from python-runtime.

@duggelz sounds like something we should do anyway. :)

from python-runtime.

Each of Python 3.5.2, 3.5.3, 3.6.0, and 3.6.1 seem to handle '--with-lto', '--enable-optimzations', and/or 'make profile-opt' differently. I'm only about 60% sure that my 3.5.3 binary was actually using PGO and LTO, since:

1. Bugs in 3.5.3's makefile cause things to be redundantly rebuilt (https://bugs.python.org/issue28643)
2. Based on python/cpython#129, I might not actually have been using PGO.
3. Based on a different change, I might not actually have been using LTO (for a while --enable-optimizations implied --with-lto, then it didn't).
4. You can't examine an executable file after the fact and determine if it was built with PGO or LTO.

At this point, my interest is turning to making a 3.5.X that actually works with uwsgi, possibly by turning off LTO, and revisit LTO in Python 3.6.

from python-runtime.

Fedora builds without PGO and without LTO: http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/python3.spec

Arch Linux started building with --enable-optimizations and --with-lto (but might not actually build with PGO because of the bug) at Python 3.6.0:
https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/python

from python-runtime.

Still working on the right combination of flags in #104

from python-runtime.

I can't seem to get this to compile properly for Python 3.5.3, but I do have it working for Python 3.6.1, so I'm calling that "qualified success".

from python-runtime.

I'd definitely put "have 3.6" above "uwsgi works on 3.5". Just because it gives 3.x users a "path" to use uwsgi.

from python-runtime.

Turns out my #117 was too aggressive, because uswgi requires the libpython-blah.a

from python-runtime.

This was fixed quite some time ago. We still don't use LTO, but that's a separate issue.

from python-runtime.