cupy.cuda.compiler.CompileException: nvrtc: error: failed to load builtins about cmr HOT 17 CLOSED

@charliememory did you use chainer 3.3.0 or the latest version? With chainer==3.3.0 I keep getting an error:
RuntimeError: CUDA environment is not correctly set up (see https://github.com/chainer/chainer#installation).cannot import name sqrt_fixed
I checked chainer.backends.cuda.available and it returned False. With the new version of chainer, I get the nvrtc error above even though I have set the CUDA_PATH and LD_LIBRARY_PATH as instructed in the cupy documentation.

from cmr.

The method suggested in https://groups.google.com/forum/#!topic/chainer-jp/GKNe5KY_fm0 worked. Had multiple versions of CUDA installed, need to figure out the right version.

from cmr.

Hi @akshaykjagadish I am facing the same problem. Can you suggest the exact steps you followed to solve the problem?
Thank you

from cmr.

Hi, it has been some time since I solved this issue but I figure you just need to find out the right CUDA version (I had multiple versions on my AWS instance) and set path accordingly.

from cmr.

Hi @akshaykjagadish , I am also facing the same problem. Do you still remember which CUDA version are you using? I currently only have CUDA 9.0 installed on my Linux, but still have the CompileException.

And hi @viralbthakar , did you solve the problem later?

Thank you all.

from cmr.

I use the new version of neural_render git clone [email protected]:charliememory/neural_renderer.git and install the pre-built cupy with corresponding cuda version number, e.g. pip install cupy-cuda92. Then, I can avoid this issue.

from cmr.

@charliememory I used neural renderer from the repo that you mentioned and have cuda 9 and hence used the same command but while installing neural renderer I get the below error:

Traceback (most recent call last):
File "setup.py", line 3, in
import neural_renderer
File "/home/nsajjan/WORK/Capstone/cmr/external/neural_renderer/neural_renderer/init.py", line 1, in
from cross import cross
File "/home/nsajjan/WORK/Capstone/cmr/external/neural_renderer/neural_renderer/cross.py", line 2, in
import cupy as cp
File "/home/nsajjan/WORK/Capstone/cmr/venv_cmr/lib/python2.7/site-packages/cupy/init.py", line 7, in
from cupy import _version
ImportError: cannot import name _version

Any idea why I am getting this?

from cmr.

@nikhilaravi I did not have such issue. Maybe you can check whether the CUDA is accessible for other packages first to make sure the problem comes from chainer.

from cmr.

@BornInWater In my case, I modified the setup.py file. I used cupy-cuda92, maybe you can try cupy-cuda9 instead.

from cmr.

@charliememory Tried that. Not working. Same error.

from cmr.

@charliememory did you use chainer 3.3.0 or the latest version? With chainer==3.3.0 I keep getting an error:
RuntimeError: CUDA environment is not correctly set up (see https://github.com/chainer/chainer#installation).cannot import name sqrt_fixed
I checked chainer.backends.cuda.available and it returned False. With the new version of chainer, I get the nvrtc error above even though I have set the CUDA_PATH and LD_LIBRARY_PATH as instructed in the cupy documentation.

Have you slove this problem？

from cmr.

@ZilinZhou1995 , No, I could not solve it.

from cmr.

@nikhilaravi , @ZilinZhou1995 Its working for me now.

I have CUDA 9. I first installed chainer 5.2.0 version and cupy 5.2.0[pip install cupy-cuda90] version separately. Then installed neural mesh renderer and Perceptual similarity in the external directory. But in the original script, branch v1.1.0 of neural mesh renderer gets installed. This was not working for me, so I installed the most recent one. That works.
After that, in the requirements.txt file, I removed chainer and cupy and just did pip install -r requirements.txt for the rest.
Also, I had to switch to Torch 1.0 version as I think torch 0.3 does not support CUDA 9.0.
Hope this helps!

from cmr.

@charliememory You are installing the newest version of NMR right? Were you able to train the model properly with the newest version of NMR? For me one of the losses was much higher than the cmr with the prescribed nmr. You can see the conversation here

from cmr.

@charliememory You are installing the newest version of NMR right? Were you able to train the model properly with the newest version of NMR? For me one of the losses was much higher than the cmr with the prescribed nmr. You can see the conversation here

Have you run the project successfully?

from cmr.

@nikhilaravi , @ZilinZhou1995 Its working for me now.

I have CUDA 9. I first installed chainer 5.2.0 version and cupy 5.2.0[pip install cupy-cuda90] version separately. Then installed neural mesh renderer and Perceptual similarity in the external directory. But in the original script, branch v1.1.0 of neural mesh renderer gets installed. This was not working for me, so I installed the most recent one. That works.
After that, in the requirements.txt file, I removed chainer and cupy and just did pip install -r requirements.txt for the rest.
Also, I had to switch to Torch 1.0 version as I think torch 0.3 does not support CUDA 9.0.
Hope this helps!

Have you run the project successfully? I have CUDA9.2 and use Torch1.0 but it has errors that says
a value of type "const float *" cannot be used to initialize an entity of type "float *"

from cmr.

python2.7, CUDA8.0, chainer 3.3.0, cupy 2.3.0, pytorch 0.3.1 works for me. Before running the demo command, I run: export PATH="$PATH:/usr/local/cuda-8.0/bin" ， export LD_LIBRARY_PATH="/usr/local/cuda-8.0/lib64" and reinstall the chainer and cupy.

from cmr.