Giter Club home page Giter Club logo

Comments (32)

alexlenail avatar alexlenail commented on May 20, 2024 1

Hi @joerick. No worries, I'm happy to go through this. From my perspective, you're one of the most responsive open source developers I've ever had the pleasure to work with! I think this project will be super helpful to me as well as many others, and I'm happy to help out ironing the kinks =)

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024 1

Turns out it was a weird bug related to setup_requires. Which is now fixed. Let me know how you get on!

pip install git+https://github.com/joerick/cibuildwheel@c3565d5025566dcf4b795eef8d40f38a12872151

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024 1

@joerick Looks like it's all working now! Just need to figure out how to automagically put the wheels on pypi.

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024

Hey thanks for this!

Yeah it's a little confusing, but I think that pybind11 is needed for the build. Now, that is correctly listed as a dependency in your setup.py install_requires, but pip isn't preinstalling that before trying to build the wheel. 🤔

I note that there's a setup.py field called setup_requires - maybe try adding the pybind11 library to that?


btw, the reason that doing pip install pybind11 before cibuildwheel doesn't work is that they are installed into different python environments. The default python environment on Travis is a homebrew-supplied one I think, that lives at /usr/local/bin/python. Cibuildwheel uses MacPython distributions which are installed into /Library/Frameworks/Python.framework

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

@joerick thanks for the information! Adding setup_requires gets me a different error which I think comes later but is related. Before, After.

Before:

    File "/private/var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/pip-vBbxQd-build/setup.py", line 18, in __str__
      import pybind11
  ImportError: No module named pybind11
  
  ----------------------------------------
  Failed building wheel for pcst-fast

After

  gcc -pthread -fno-strict-aliasing -Wformat -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/_internal/cpython-2.6.9-ucs2/include/python2.6 -I/root/.local/include/python2.6 -I/opt/_internal/cpython-2.6.9-ucs2/include/python2.6 -c src/pcst_fast_pybind.cc -o build/temp.linux-x86_64-2.6/src/pcst_fast_pybind.o -DVERSION_INFO="1.0.2" -std=c++11 -fvisibility=hidden
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
  src/pcst_fast_pybind.cc:6:31: fatal error: pybind11/pybind11.h: No such file or directory
   #include <pybind11/pybind11.h>
                                 ^
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for pcst-fast

Any ideas?

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024

Could you post the whole 'after' output? I think gcc isn't being invoked with the right include paths... maybe pybind needs to be installed globally for that to work

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

Hi @joerick, here's the whole output:

The command "pip install cibuildwheel==0.1.3 pybind11==2.1.0" exited with 0.
48.13s$ cibuildwheel --output-dir wheelhouse
Unable to find image 'quay.io/pypa/manylinux1_x86_64:latest' locally
latest: Pulling from pypa/manylinux1_x86_64
Status: Downloaded newer image for quay.io/pypa/manylinux1_x86_64:latest
+ cd /project
+ for PYBIN in '/opt/python/*/bin'
+ /opt/python/cp26-cp26m/bin/pip wheel . -w /tmp/linux_wheels
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Processing /project
Collecting pybind11==2.1.0 (from pcst-fast==1.0.2)
/opt/_internal/cpython-2.6.9-ucs2/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/opt/_internal/cpython-2.6.9-ucs2/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading pybind11-2.1.0-py2.py3-none-any.whl (109kB)
  Saved /tmp/linux_wheels/pybind11-2.1.0-py2.py3-none-any.whl
Skipping pybind11, due to already being wheel.
Building wheels for collected packages: pcst-fast
  Running setup.py bdist_wheel for pcst-fast: started
  Running setup.py bdist_wheel for pcst-fast: finished with status 'error'
  Complete output from command /opt/_internal/cpython-2.6.9-ucs2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-9aHlBZ-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpaFXl8bpip-wheel-:
  running bdist_wheel
  running build
  running build_py
  package init file 'src/__init__.py' not found (or not a regular file)
  creating build
  creating build/lib.linux-x86_64-2.6
  creating build/lib.linux-x86_64-2.6/pcst_fast
  copying src/test_pcst_fast.py -> build/lib.linux-x86_64-2.6/pcst_fast
  running build_ext
  creating tmp
  gcc -pthread -fno-strict-aliasing -Wformat -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/_internal/cpython-2.6.9-ucs2/include/python2.6 -c /tmp/tmplasQ5j.cpp -o tmp/tmplasQ5j.o -std=c++11
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
  gcc -pthread -fno-strict-aliasing -Wformat -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/_internal/cpython-2.6.9-ucs2/include/python2.6 -c /tmp/tmptKJK0N.cpp -o tmp/tmptKJK0N.o -fvisibility=hidden
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
  building 'pcst_fast' extension
  creating build/temp.linux-x86_64-2.6
  creating build/temp.linux-x86_64-2.6/src
  gcc -pthread -fno-strict-aliasing -Wformat -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/_internal/cpython-2.6.9-ucs2/include/python2.6 -I/root/.local/include/python2.6 -I/opt/_internal/cpython-2.6.9-ucs2/include/python2.6 -c src/pcst_fast_pybind.cc -o build/temp.linux-x86_64-2.6/src/pcst_fast_pybind.o -DVERSION_INFO="1.0.2" -std=c++11 -fvisibility=hidden
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
  src/pcst_fast_pybind.cc:6:31: fatal error: pybind11/pybind11.h: No such file or directory
   #include <pybind11/pybind11.h>
                                 ^
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for pcst-fast

It seems to get one step further now. Before "import pybind11" failed. Now that's succeeding, but when it goes to creating build/temp.linux-x86_64-2.6/src it can't find pybind11.h which is called for at the top of my c++ file.

How would you install globally?

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

@joerick This is how the include is done in the demo version of setup.py provided with pybind11 (adapted slightly). Seems reasonable to me, but I'm not totally sure in what order all of these events occur. Kind of strange that I can build the wheel locally no problem...

class get_pybind_include(object):
    """Helper class to determine the pybind11 include path

    The purpose of this class is to postpone importing pybind11
    until it is actually installed, so that the ``get_include()``
    method can be invoked. """

    def __init__(self, user=False):
        self.user = user

    def __str__(self):
        import pybind11
        return pybind11.get_include(self.user)


ext_modules = [
    Extension(
        'pcst_fast',
        sources=['src/pcst_fast_pybind.cc', 'src/pcst_fast.cc'],
        include_dirs=[
            # Path to pybind11 headers
            get_pybind_include(),
            get_pybind_include(user=True)
        ],
        language='c++'
    ),
]

Interestingly, someone here seems to be having a similar issue: pybind/python_example#18

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024

I think that when the project is listed in setup_requires it's installed into an egg by easy_install, so when your setup script asks pip where to find headers it returns the wrong path. Maybe.

I'm going to look at adding an option CIBW_BEFORE_BUILD. I'm thinking that if you put CIBW_BEFORE_BUILD=pip install pybind11 that'll solve your problem. We'll see!

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

Hi @joerick

Thanks for getting back to me! That sounds like a great option to add for this use case. Keep me posted on how it goes!

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024

Hey. I've added that feature, it's available on a feature branch of this repo for now. You can install with pip install git+https://github.com/joerick/cibuildwheel@2c0d6e3db1590a9f6902c6d46a8ff27aeb4f1add.

Would be interested if this allows you to solve your problem!

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

@joerick One of two has run, and it didn't work strangely. Here's my travis file by the way. I'm still waiting for the second build. (oh, and here's my setup.py copied from here.

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

Hooray! It worked! Got a new error though, which seems to have to do with the compiler? We don't get those error messages when we compile locally...

The second build errored in a really interesting way because it seems like it was able to build the wheel but not install it. Strange...

Thanks for the help @joerick. What will you do with this new feature? Will you merge into master?

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

@joerick from this log it seems like the flag you added only works for some versions of python?

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024

Ah yeah, on Mac you need to use pip3 instead of pip so that the package installs into the right environment.

I added {pip} for this purpose. If you change your option to {pip} install ... as the option that should change to pip or pip3 as required.

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024

Btw, I wrote docs for the option here 2c0d6e3

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

@joerick Amazing! One of the travis builds succeeded! I'm not sure I understand how the two of them are split up? The other one seems to fail because it has the wrong compiler? Link to travis page.

Oh I see, it succeeded on mac but failed on linux.

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024

Looks like it's failing on Python 2.6 because of missing PyCapsule. I think this API isn't available in Python 2.6....

Looks like you need the CIBW_SKIP option too! #3 Alternatively you could use the thunk posted above

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

The thunk?

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024

http://py3c.readthedocs.io/en/latest/capsulethunk.html

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

@joerick So I would either need to include that .h file in my project or CIBW_SKIP 2.6? I'd much prefer the latter, but it sounds like CIBW_SKIP will be fairly tough to implement?

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

@joerick Looks like I'm building all the wheels now in linux with CIBW_SKIP - (thanks by the way!) but it seems like it's trying to repair the pybind11 wheel at the end, which is failing: see the log. Is there some way to skip that as well? Perhaps this might be a problem arising from CIBW_BEFORE_BUILD?

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024

I think this is because pip wheel is outputting your dependency wheel as well as the one you want. I've added a commit to restrict auditwheel and delocate to the package we're building.

pip install git+https://github.com/joerick/cibuildwheel@d5fa5475d4e8c58000d056f44a9968f68337cf95

p.s. thank you for your patience and help with this! we're ironing out a lot of kinks in this project that will hopefully save a lot of debugging time for future users! 👍

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024

Looks like your build failed again

https://travis-ci.org/fraenkel-lab/pcst_fast/jobs/221348812

I think I'm guessing the wheel name wrong 🤔

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

@joerick you're one step ahead of me.

It's worth noting that when you added CIBW_SKIP the mac builds started failing (whereas they had been working when you added CIBW_BEFORE.... I'm not entirely sure why. Check it out.

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024

🙌 do you mind if I add your repo to the readme as an example?

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

Of course! No Problem! I think that particular repo of ours will be around for a long time.

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

@joerick interestingly, we're having a bug with python 3.5 / macOS (which is resolved in the python 3.6 wheel):

>>> import pcst_fast
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pcst_fast.cpython-35m-darwin.so, 2): Symbol not found: __PyThreadState_UncheckedGet
  Referenced from: /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pcst_fast.cpython-35m-darwin.so
  Expected in: flat namespace
 in /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pcst_fast.cpython-35m-darwin.so

Seems like the relevant part is Symbol not found: __PyThreadState_UncheckedGet. Do you know anything about this Symbol? I'm not quite sure where to look for this bug...

from cibuildwheel.

joerick avatar joerick commented on May 20, 2024

Hmm, never seen that before. Anything suspicious in the build log for that wheel?

from cibuildwheel.

alexlenail avatar alexlenail commented on May 20, 2024

Not that I can tell. That's super weird... How is it possible that this successfully installs the wheel on travis but not on real Macs with python 3.5? Could this have to do with minor versions? (I think it fails on 3.5.1, the wheel was built for 3.5.3...) @joerick

from cibuildwheel.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.