RuntimeError: cuda runtime error (8) : invalid device function at /pytorch/torch/lib/THC/generic/THCTensorMath.cu:35 about factorizablenet HOT 14 CLOSED

Thank you for update the code version. However, unfortunatelly the CUDA runtime error(8) still remains! Maybe because pytorch 0.3.1+python2.7 does not fit GTX1080Ti architecture (cuda8.0).
I will try to run the code under pytorch 0.4.x

from factorizablenet.

It ends at the line 86 because we have an exception catch there. That is not the position where the error happened. I highly recommend you to use pdb to run the code step by step to check where it actually happens.

Now I will close the issue. Feel free to leave comments at this thread.

from factorizablenet.

So in my case, my graphics card uses CUDA compute capability 7.0 and PyTorch 3.x which this project requires I'm assuming isn't compatible with that level. If it helps to debug your situation, I moved to a computer with a Titan X (CUDA 8.0, capability 6.1) and it worked fine using the instructions in the readme.

from factorizablenet.

Can you provide more details?

from factorizablenet.

Maybe you can try training the model with pretrained RPN --rpn /path/to/rpn

from factorizablenet.

Hello!! author:

This error first occured when I try to evaluate the model with the pretrained model. I've add --rpn option, but it does not work!
I also tried to train the model but the same error occured! Right after start training, it reported THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCTensorCopy.cu line=204 error=8 : invalid device function and in engines_v1.py it jumps to Exception handle.
It seems to the roialign function does not work well.
Thank You!

from factorizablenet.

Yes, it is because that VG-DR-Net has some self-relations, e.g. A-relation-A. Please use git pull to update the latest version. Please check the Updates in README for more information.

If it works, please comment here and I will close the issue.

from factorizablenet.

Sorry, I haven't met this issues before. Maybe you can try the evalaution and training on different datasets to check whether it happens only the specific or any settings. Hope to hear more about the issue.

from factorizablenet.

Hello！Today, I have tried to run evaluation on DR dataset and tried to train RPN network, the same error exactly occurred! I think it's the cuda environment issue not the code having bugs.
LinXin

from factorizablenet.

I am so sorry to hear that.

I think you can use pdb to track there the bug happened. So we can check it is because of the settings of the code or the general configuration of your server.

Looking forward to any updates.

from factorizablenet.

Hello, I have debug the code beffore, it jumps to engines_v1.py/line86 exception handle.

from factorizablenet.

I had the same error. It might be related to a CUDA version mismatch (at least in my case) as the pytorch installed via pip isn't compiled using the latest version. Were you able to solve this?

from factorizablenet.

      I had the same error. It might be related to a CUDA version mismatch (at least in my case) as the pytorch installed via pip isn't compiled using the latest version. Were you able to solve this?

Not yet.

from factorizablenet.

Thank you for update the code version. However, unfortunatelly the CUDA runtime error(8) still remains! Maybe because pytorch 0.3.1+python2.7 does not fit GTX1080Ti architecture (cuda8.0).
I will try to run the code under pytorch 0.4.x

Imeet the same problem with pytorch 0.3.1+python3.6

from factorizablenet.