Issue with cuda lib? about gandlf HOT 11 CLOSED

What should I do if my the newest version of cuda my computer supports is 11.2?

If your hardware supports CUDA 11.2, it should support 11.6. Just follow the pip installation (it should install CUDA and all required libraries into your virtual environment).

from gandlf.

How did you install PyTorch? If you used Conda, can you try using pip [ref]?

pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116

from gandlf.

What should I do if my the newest version of cuda my computer supports is 11.2?

from gandlf.

So that fixed that, but now I'm getting the following:

Looping over training data for penalty calculation: 0%| | 0/2426 [00:00<?, ?it/s]/cbica/projects/DBT_AI/.conda/envs/venv_gandlf_new/lib/python3.8/site-packages/torchio/data/io.py:36: UserWarning: Error loading image with SimpleITK:
Exception thrown in SimpleITK ImageFileReader_Execute: /tmp/SimpleITK-build/ITK-prefix/include/ITK-5.2/itkImportImageContainer.hxx:192:
Failed to allocate memory for image.

Trying NiBabel...
warnings.warn(message)

Looping over training data for penalty calculation: 0%| | 0/2426 [00:00<?, ?it/s]
ERROR:

from gandlf.

Hmmmm, This seems like an interesting error. Can you please let us know your ITK version and the output of
head -n 2 data.csv for assisting you better?

from gandlf.

subjectID,channel_0,label
0,/cbica/projects/DBT_AI/Data/masks/75712684_PROC_LCC_RC_2/75712684_PROC_LCC_RC_mat.nii.gz,/cbica/projects/DBT_AI/Data/masks/75712684_PROC_LCC_RC_2/75712684_PROC_LCC_RC_mask.nii.gz

ITK verison: 3.8.0

from gandlf.

Can you mention the output of the following command:

# activate gandlf python environment
python -c "import SimpleITK as sitk;image=sitk.ReadImage('/cbica/projects/DBT_AI/Data/masks/75712684_PROC_LCC_RC_2/75712684_PROC_LCC_RC_mat.nii.gz');print(image.GetSize());mask=sitk.ReadImage('/cbica/projects/DBT_AI/Data/masks/75712684_PROC_LCC_RC_2/75712684_PROC_LCC_RC_mask.nii.gz');print(mask.GetSize())"

Also, it would be great if you can post at least the mask so that we can debug further.

from gandlf.

(1996, 2457, 73)
(1996, 2457, 73)

What do you mean by posting the mask?

from gandlf.

This is also at the top of the error file:
No NVIDIA kernel driver module found, skipping CUDA

from gandlf.

(1996, 2457, 73)
(1996, 2457, 73)

Hmm, if the piece of code I replied with is giving this output, it means that the IO is working as expected.

What do you mean by posting the mask?

I meant uploading it here for us to debug. But it doesn't matter, since the IO is working correctly (as seen from the output of the command I sent).

No NVIDIA kernel driver module found, skipping CUDA

This is not unrelated to GaNDLF, and is dependent on the host machine.

from gandlf.

I tried running the same job with 1/5 of the training data and it was able to run without an error, however, I'm getting this:
Epoch Final train loss : 1.0
Epoch Final train dice : 0.0
Epoch Final train dice_per_label : [0.0, 0.0]
Epoch Final train iou : 0.34576352043151853
Epoch Final train f1 : 0.5872102342128753

from gandlf.