libnvrtc not found, v13 about cupy HOT 6 OPEN

NVRTC is now dlopen'd

from cupy.

Setting LD_LIBRARY_PATH is a workaround but we generally avoid using it. It works, but is not a solution unfortunately as we cannot tell thousands of users to set LD_LIBRARY_PATH for cupy to works. In my opinion, it is best to fail early if cupy depends on cuda.

ctypes find_library has a bug since forever where it does not return the full path of the library, hence not founding said library when they are installed in non standard location, such as on HPC systems. https://bugs.python.org/issue21042
We hacked ctypes with our own version to palliate for this issue, but even that does not work:

[coulombc@ng10403 ~]$ module load gcc python/3.11 cuda/12 custom-ctypes
[coulombc@ng10403 ~]$ makevenv 
(15787) [coulombc@ng10403 ~]$ pip install --no-index cupy-13.0.0+computecanada-cp311-cp311-linux_x86_64.whl 
(15787) [coulombc@ng10403 ~]$ python -c "import cupy; cupy.show_config()"

RuntimeError: CuPy failed to load libcudart.so.12: OSError: libcudart.so.12: cannot open shared object file: No such file or directory

I'll dig a bit more into why it did not work with our custom-ctypes, but since cupy links on cuda, would it be possible to offer distributors the option to soft or hard link on cuda libraries? Soft linking does not work for many of us unfortunately.

I could also patch cupy to add needed libs, but I'd prefer to avoid doing that systematically for every release now on.

from cupy.

Thanks for the feedback @ccoulombe, this is a thing I overlooked when moving to ctypes-based loading. I think we can fix it on our side as follows:

1. Add a new extension module cupy_backends.cuda.libs._cuda_deps, which has libnvrtc.so.12 as NEEDED.
2. During import cupy, do import cupy_backends.cuda.libs._cuda_deps while ignoring any errors raised.

If libnvrtc.so.12 can be discovered in step 2, ctypes.CDLL("libnvrtc.so.12") will use that shared library. This should allow RPATH set during the build to be searched first.

from cupy.

I am still confused, though. @ccoulombe I agree no one wants users to set LD_LIBRARY_PATH, but isn't this HPC system admin's job to ensure there's a mechanism for putting DSOs in a linker-searchable path via some administrated mechanisms (ex: module load)? If the admins set up a proper mechanism and the user follows, ideally the DSOs would be discoverable at run time by the linker, and users don't have do anything weird. (And you don't have to build CuPy from source to hack RPATH.)

To make RPATH work, we still need the DSOs to be locatable by the linker, so I don't know what problem it'd solve?

from cupy.

@ccoulombe gentle ping, could you help us understand better?

from cupy.

@leofang Sorry for the delay, thanks for the ping! :)
I'm out of office and want to take the time to provide a detailed response, I'll add it shortly!

from cupy.