This is a Github repository for the SIGCOMM'23 paper "Network Load Balancing with In-network Reordering Support for RDMA".

We describe how to run this repository either on docker or using your local machine with ubuntu:20.04.

For Ubuntu, following the installation guide here and make sure to apply the necessary post-install steps. Eventually, you should be able to launch the hello-world Docker container without the sudo command: docker run hello-world.

First, you do all these:

wget https://www.nsnam.org/releases/ns-allinone-3.19.tar.bz2
tar -xvf ns-allinone-3.19.tar.bz2
cd ns-allinone-3.19
rm -rf ns-3.19
git clone https://github.com/conweave-project/conweave-ns3.git ns-3.19

Here, ns-allinone-3.19 will be your root directory.

Create a Dockerfile at the root directory with the following:

FROM ubuntu:20.04
ARG DEBIAN_FRONTEND=noninteractive

RUN apt update && apt install -y gnuplot python python3 python3-pip build-essential libgtk-3-0 bzip2 wget git && rm -rf /var/lib/apt/lists/* && pip3 install install numpy matplotlib cycler
WORKDIR /root

Then, you do this:

docker build -t cw-sim:sigcomm23ae .

Once the container is built, do this from the root directory:

docker run -it -v $(pwd):/root cw-sim:sigcomm23ae bash -c "cd ns-3.19; ./waf configure --build-profile=optimized; ./waf"

This should build everything necessary for the simulator.

One can always just run the container:

docker run -it --name cw-sim -v $(pwd):/root cw-sim:sigcomm23ae 
cd ns-3.19;
./autorun.sh

That will run 0.1 second simulation of 8 experiments which are a part of Figure 12 and 13 in the paper. In the script, you can easily change the network load (e.g., 50%), runtime (e.g., 0.1s), or topology (e.g., leaf-spine). To plot the FCT graph, see below or refer to the script ./analysis/plot_fct.py.

❗ To run processes in background, use ./autorun.sh > 2>&1 & instead of ./autorun.sh.

We tested the simulator on Ubuntu 20.04, but latest versions of Ubuntu should also work.

sudo apt install build-essential python3 libgtk-3-0 bzip2

For plotting, we use numpy, matplotlib, and cycler for python3:

python3 -m pip install numpy matplotlib cycler

wget https://www.nsnam.org/releases/ns-allinone-3.19.tar.bz2
tar -xvf ns-allinone-3.19.tar.bz2
cd ns-allinone-3.19
rm -rf ns-3.19
git clone https://github.com/conweave-project/conweave-ns3.git ns-3.19
cd ns-3.19
./waf configure --build-profile=optimized
./waf

You can reproduce the simulation results of Figure 12 and 13 (FCT slowdown) by running the script:

./autorun.sh

In the script, you can easily change the network load (e.g., 50%), runtime (e.g., 0.1s), or topology (e.g., leaf-spine). This takes a few hours, and requires 8 CPU cores and 10G RAM. Note that we do not run DRILL since it takes too much time due to many out-of-order packets.

If you want to run the simulation individually, try this command:

python3 ./run.py --h

It first calls a traffic generator ./traffic_gen/traffic_gen.py to create an input trace. Then, it runs NS-3 simulation script ./scratch/network-load-balance.cc. Lastly, it runs FCT analyzer ./fctAnalysis.py and switch resource analyzer ./queueAnalysis.py.

You can easily plot the results using the following command:

python3 ./analysis/plot_fct.py

The result figures are located at ./analysis/figures. The script requires input parameters such as -sT and -fT which indicate the time window to analyze the fct result. By default, it assuems to use 0.1 second runtime.

To clean all data of previous simulation results, you can run the command:

./cleanup.sh

• At ./mix/output, several raw data is stored such as

  • Flow Completion Time (XXX_out_fct.txt),
  • PFC generation (XXX_out_pfc.txt),
  • Uplink's utility (XXX_out_uplink.txt),
  • Number of connections (XXX_out_conn.txt),
  • Congestion Notification Packet (XXX_out_cnp.txt).
  • CDF of number of queues usage per egress port (XXX_out_voq_per_dst_cdf.txt).
  • CDF of total queue memory overhead per switch (XXX_out_voq_cdf.txt).

• Each run of simulation creates a repository in ./mix/output with simulation ID (10-digit number).

• Inside the folder, you can check the simulation config config.txt and output log config.log.

• The output files include post-processed files such as CDF results.

• The history of simulations will be recorded in ./mix/.history.

We include ConWeave's parameter values into ./run.py based on flow control model and topology.

Most implementations of network load balancing are located in the directory ./src/point-to-point/model.

• switch-node.h/cc: Switching logic that includes a default multi-path routing protocol (e.g., ECMP) and DRILL.
• switch-mmu.h/cc: Ingress/egress admission control and PFC.
• conga-routing.h/cc: Conga routing protocol.
• letflow-routing.h/cc: Letflow routing protocol.
• conweave-routing.h/cc: ConWeave routing protocol.
• conweave-voq.h/cc: ConWeave in-network reordering buffer.
• settings.h/cc: Global variables for logging and debugging.
• rdma-hw.h/cc: RDMA-enable NIC behavior model.

RNIC behavior model to out-of-order packet arrival As disussed in the paper, we observe that RNIC reacts to even a single out-of-order packet sensitively by sending CNP packet. However, existing RDMA-NS3 simulator (HPCC, DCQCN, TLT-RDMA, etc) did not account for this. In this simulator, we implemented that behavior in rdma-hw.cc.

If you find this repository useful in your research, please consider citing:

@inproceedings{song2023conweave,
  title={Network Load Balancing with In-network Reordering Support for RDMA},
  author={Song, Cha Hwan and Khooi, Xin Zhe and Joshi, Raj and Choi, Inho and Li, Jialin and Chan, Mun Choon},
  booktitle={Proceedings of SIGCOMM},
  year={2023}
}

This code repository is based on https://github.com/alibaba-edu/High-Precision-Congestion-Control for Mellanox Connect-X based RDMA-enabled NIC implementation, and https://github.com/kaist-ina/ns3-tlt-rdma-public.git for Broadcom switch's shared buffer model and IRN implementation.

MIT License

Copyright (c) 2023 National University of Singapore

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.