Issues with gdr driver about gdrcopy HOT 9 CLOSED

Most probably the gdrdrv kernel module is either not loaded or not accessible (eg protection flags on the /dev/gdrdrv inode, container environment, etc) on the cluster nodes where your MPI job is run.

from gdrcopy.

Thanks. That sounds the right diagnosis. This is the first time I hear about protection flags (in this context) and couldn't find information about it. Is there any data or parameter that I should be reading to provide a more complete description of the system configuration?

from gdrcopy.

Can I ask you to run two commands and post the results?

1. ls -alh /dev/gdrdrv
2. lsmod | grep gdrdrv

You may want to try sudo ./insmod.sh to reload the driver. The script is in the top directory.

from gdrcopy.

After reloading the driver, I get:
ls -alh /dev/gdrdrv ---> crw-rw-rw-. 1 root root 235, 0 Jul 31 19:31 /dev/gdrdrv

lsmod | grep gdrdrv ---> 
gdrdrv                 17352  0
nvidia              17901608  3 nv_peer_mem,gdrdrv,nvidia_modeset

from gdrcopy.

Can you use gdrcopy now?

from gdrcopy.

Not really: I'm still getting the same error message. I suspect that the issue is some interference (maybe the protection flag that @drossetti mentioned) as the module has always been loaded during the tests. I could try two things:

1. Have a similar configuration but w/o loading the NVIDIA_peer_memory
2. Create a similar configuration but in a different environment (instance). However, this will take me a few hours as I have to build a few pieces from source

Thanks.

from gdrcopy.

I believe that @drossetti meant ls -alh /dev/gdrdrv ---> crw-rw-rw-. 1 root root 235, 0 Jul 31 19:31 /dev/gdrdrv. Your user needs read and write permissions to /dev/gdrdrv. In your case, the flags are set for all users, so that should not be your issue.

You mentioned that the error occurs when you ran MPI. Is it possible that gdrdrv is not loaded properly on some nodes?

If you have verified that and the error is still persisted, can I ask you to enable the debug mode and post the logs here?

1. Edit insmod.sh. Change line 28 sudo /sbin/insmod gdrdrv/gdrdrv.ko dbg_enabled=0 info_enabled=0 to sudo /sbin/insmod gdrdrv/gdrdrv.ko dbg_enabled=1 info_enabled=1.
2. On another terminal, run dmesg -w. You should see more lines show up after you run the following steps.
3. Do sudo ./insmod.sh.
4. export GDRCOPY_ENABLE_LOGGING=1
5. export GDRCOPY_LOG_LEVEL=1
6. Run your MPI. Remember to use mpirun -x GDRCOPY_ENABLE_LOGGING -x GDRCOPY_LOG_LEVEL <the rest of the command> so that the environment variables are properly passed to your mpi environment.
7. Post the log from gdrcopy.
8. Post the log from dmesg. Only what shows up after the step 2 is fine.

from gdrcopy.

Sorry about the delay as I'm troubleshooting several things. gdrcopy is running in a single node (1 vCPU + 1 GPU). I believe that the gdr error message can be avoided by including the flag MV2_GPUDIRECT_GDRCOPY_LIB=/home/ody/gdrcopy/libgdrapi.so (not sure why gdrcopy didn't install at /opt as specified but that is a different question). The dmesg output now reads:
[ 2058.859349] gdrdrv:minor=0
[ 2060.863156] gdrdrv:closing
I intend on building a new system tomorrow and will re-test everything. Accordingly, I'll post any update asap and will close the ticket if the issue was simply caused by not specifying the library at the time of initializing the MPI job. Thanks.

from gdrcopy.

It seems that declaring the flag MV2_GPUDIRECT_GDRCOPY_LIB as the specific subdirectory & library solves the problem ($LD_LIBRARY_PATH had the subsirectory so I'm unsure what was originating the error). Another issue came up while building UCX but I'm closing this ticket and will open a different one if necessary. Thank you for the help.

from gdrcopy.