call ibv_reg_mr failed using mapped memory about gdrcopy HOT 7 OPEN

@tangrc99 this expected as the implementation of ibv_reg_mr in the Linux kernel requires the virtual address range to be backed by CPU memory pages.

More exactly, pin_user_pages does not work on CPU mappings of PCIe resources created via io_remap_pfn_range.

The official way of enabling RDMA on GPU memory is:

• querying a dma-buf file descriptor from a GPU device pointer (https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__MEM.html#group__CUDA__MEM_1g51e719462c04ee90a6b0f8b2a75fe031) and
• passing that to ibv_reg_dmabuf_mr.

For a full deployment case, see for example https://github.com/openucx/ucx/blob/1308d2055ab0ba948eac213c8cfcd92776c34a53/src/uct/cuda/cuda_copy/cuda_copy_md.c#L410 and https://github.com/openucx/ucx/blob/1308d2055ab0ba948eac213c8cfcd92776c34a53/src/uct/ib/base/ib_md.c#L480.

from gdrcopy.

Thanks, cause A10 don't support dma-buf file descriptor. Can I use GDR on A10 with other methods ？

from gdrcopy.

It should. Are you using the openrm variant of the GPU kernel-mode driver, see https://developer.nvidia.com/blog/nvidia-releases-open-source-gpu-kernel-modules/ ?

from gdrcopy.

function cuMemGetHandleForAddressRange requires CU_MEM_RANGE_HANDLE_TYPE_DMA_BUF_FD which is 0 on A10. nv_peer_mem and nvidia-peermem is already loaded, is there any other requirements ?

from gdrcopy.

Hi @tangrc99,

Neither nvidia-peermem nor nv_peer_mem involves in dmabuf. A10 should support dmabuf. Could you check if your SW stack is new enough to support dmabuf?

• NVIDIA driver with the open variant version 515 or later.
• CUDA 11.7 or later.
• Linux kernel version 5.12 or later. This is for the NIC stack. The GPU stack does not have this requirement.

from gdrcopy.

Thanks, My Linux kernel 4.18.0 is too old.

from gdrcopy.

In that case you can use the legacy RDMA memory registration path, i.e. ibv_reg_mr, which involves the peer-direct kernel infrastructure (for example provided by MLNX_OFED) and nvidia-peermem.

from gdrcopy.