rdma_client_server's Introduction

RDMA based Client Server Example (`RDMAClient`, `RDMAServer`)

Supported features

Support control-plane primitive setup_server, setup_client, connect_server, disconnect_server
Support multi-iterations pair of following datapath commands
- Pairs of IBV_WR_SEND|IBV_WC_RECV using ibv_post_send/ibv_post_recv/ibv_poll_cq pairs from RDMAClient to RDMAServer
- Pairs of IBV_WR_RDMA_WRITE|IBV_WR_RDMA_READ using ibv_post_send/ibv_poll_cq from RDMAClient to RDMAServer
Profile RTT latency of the above datapath commands

Tutorial

To compile from source

mkdir build && cd build && cmake .. && make

To run client on host1 with RDMA compliant NICs

host1 $ ./RDMAClient <client ip> <server ip:port> <opcode> <iterations> <message size>
host1 $ ./RDMAClient 192.168.10.41 192.168.10.43:50053 SEND 1000 256

To run server on host2 with RDMA compliant NICs connected directly or via switch to host1

host1 $ ./RDMAServer <server ip:port>
host1 $ ./RDMAServer 192.168.10.43:50053

Here is example of the results on client host1,

[SEND-RECV] Round Trip Latency: 23770 nsec, Size: 256 bytes
...
[SEND-RECV] Round Trip Latency: 23800 nsec, Size: 256 bytes

rdma_client_server's People

Contributors

Watchers

rdma_client_server's Issues

Network Address to Plain Text Conversion is inaccurate in `RDMAClient`

In this example, RDMAClient peer and my IP address are showing the same value on the console, but when inspected using gdb, they are as expected. This means that the presentation layer has a potential bug, that needs to be addressed

Console Output

$ ./RDMAServer 192.168.60.43:50053
Server IP: 192.168.60.43, Port: 50053, Rank: 15403
Got 4 RDMA devices
Got RDMA CM Event: RDMA_CM_EVENT_CONNECT_REQUEST
Got RDMA CM Event: RDMA_CM_EVENT_ESTABLISHED
Got RDMA CM Event: RDMA_CM_EVENT_DISCONNECTED
...

$ ./RDMAClient 192.168.60.41 192.168.60.43:50053 10
PEER: 192.168.60.43
MY: 192.168.60.41
Client IP: 192.168.60.41, Iterations: 10, Rank: 15401 => Target Server: 192.168.60.43:50053
Client [192.168.60.41][192.168.60.41] => Server [192.168.60.41][192.168.60.41]
Got 4 RDMA devices
Attempting to connect between src: 192.168.60.41 dst: 192.168.60.41
Got RDMA CM Event: RDMA_CM_EVENT_ADDR_RESOLVED
Got RDMA CM Event: RDMA_CM_EVENT_ROUTE_RESOLVED
Got RDMA CM Event: RDMA_CM_EVENT_ESTABLISHED
Connected RDMA_RC between src: 192.168.60.41 dst: 192.168.60.41

GDB output

(gdb) n
Client [192.168.60.41][192.168.60.41] => Server [192.168.60.41][192.168.60.41]
28	in /opt/software/runtime/src/runtime/rdma_client_server/rdma_client.c
(gdb) p sv
$3 = <optimized out>
(gdb) p sip
$4 = 0x7ffff7fd7700 "192.168.60.41"
(gdb) p dip
$5 = 0x7ffff7fd7700 "192.168.60.41"
(gdb) p sv->my_addr
value has been optimized out
(gdb) b setup_client
Breakpoint 2 at 0x555555556ab0: file /opt/software/runtime/src/runtime/rdma_client_server/rdma_client_lib.c, line 70.
(gdb) c
Continuing.

Breakpoint 2, setup_client (src_addr=0x55555555b8e0, dst_addr=0x55555555b900) at /opt/software/runtime/src/runtime/rdma_client_server/rdma_client_lib.c:70
70	/opt/software/runtime/src/runtime/rdma_client_server/rdma_client_lib.c: No such file or directory.
(gdb) p src_addr
$6 = (struct sockaddr *) 0x55555555b8e0
(gdb) p ((struct sockaddr_in *)src_addr)->sin_addr
$7 = {s_addr = 691841216}
(gdb) p/x ((struct sockaddr_in *)src_addr)->sin_addr
$8 = {s_addr = 0x293ca8c0}
(gdb) p/x ((struct sockaddr_in *)dst_addr)->sin_addr
$9 = {s_addr = 0x2b3ca8c0}

Recommend Projects