Investigate SR-IOV with Mellanox ConnectX-4 NICs about cnf-testbed HOT 3 CLOSED

Working notes for enabling and using VFs from Mellanox ConnectX-4 NIC

Update kernel parameters:

• Edit /etc/default/grub
  • Add: "iommu=pt intel_iommu=on hugepagesz=2M hugepages=10240 isolcpus=2,4,6 nohz_full=2,4,6 rcu_nocbs=2,4,6"
• $ update-grub2
• Reboot machine
• $ echo 5120 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
• $ echo 5120 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages

Installed Mellanox OFED drivers:

• http://content.mellanox.com/ofed/MLNX_OFED-4.4-1.0.0.0/MLNX_OFED_LINUX-4.4-1.0.0.0-ubuntu18.04-x86_64.tgz
• ./mlnxofedinstall ((--dpdk --upstream-libs --force))
  • Might be necessary to install without the extra options (( )) to have tools available to enable VFs on the hardware (see DOC-2386 - Link below)
  • Installs drivers and necessary additions for use with DPDK

Enable VFs

• echo 8 > /sys/class/net/enp94s0f0/device/sriov_numvfs
  • Check the interface name before calling command
  • Not persistent through reboots

Refer to Mellanox documentation for useful commands and configuration notes:

• https://community.mellanox.com/docs/DOC-2386
• https://community.mellanox.com/docs/DOC-2729
  • Building and installing VPP with MLX5 support

Build VPP:

• Install VPP v18.07 (Refer to mellanox document above)
  • git clone https://gerrit.fd.io/r/vpp
  • cd vpp/
  • git checkout tags/v18.07
  • sed -i '/vpp_uses_dpdk_mlx5_pmd/s/^# //g' build-data/platforms/vpp.mk
  • make install-dep
  • make dpdk-install-dev DPDK_MLX5_PMD=y
  • make build-release
  • make pkg-deb vpp_uses_dpdk_mlx5_pmd=yes
  • dpkg -i build-root/vpp-lib*
  • dpkg -i build-root/vpp_18*
  • dpkg -i build-root/vpp-plugins*
• ( Set the number of hugepages to 5120 again (see above) )

Enable VFs

• echo 8 > /sys/class/net/enp94s0f0/device/sriov_numvfs
  • Check the interface name before calling command

from cnf-testbed.

Behavior has been confirmed on two nodes, both hanging when trying to "up" VF interfaces in VPP

from cnf-testbed.

Working solution (VPP v18.04, DPDK 18.02.1)

Start by getting the correct version of VPP:

• git clone https://gerrit.fd.io/r/vpp
• cd vpp/
• git checkout tags/v18.04
  Add the following patches to the code:

diff --git a/dpdk/deb/debian/rules b/dpdk/deb/debian/rules
index 98b1048b..b9a58636 100755
--- a/dpdk/deb/debian/rules
+++ b/dpdk/deb/debian/rules
@@ -8,7 +8,7 @@ BASE_VER = $(word 1, $(subst -, ,$(VERSION)))
 export DPDK_BUILD_DIR=$(CURDIR)/_build
 export DPDK_INSTALL_DIR=$(CURDIR)/debian/tmp/usr

-SRC=_build/dpdk-$(BASE_VER)
+SRC=$(wildcard _build/dpdk*$(BASE_VER))

 MAKE_ARGS=-C ..

diff --git a/src/vpp-api/java/Makefile.am b/src/vpp-api/java/Makefile.am
index 21a18575..5e9e53a4 100644
--- a/src/vpp-api/java/Makefile.am
+++ b/src/vpp-api/java/Makefile.am
@@ -71,9 +71,7 @@ jvpp-registry/io_fd_vpp_jvpp_VppJNIConnection.h: $(jvpp_registry_src_files)
        @echo "  JVPP GEN  $@"
        @rm -rf jvpp-registry/target
        @mkdir -p jvpp-registry/target
-       @$(JAVAC) -d jvpp-registry/target $^
-       @$(JAVAH) -force -classpath jvpp-registry/target -d jvpp-registry io.fd.vpp.jvpp.VppJNIConnection
-       @$(JAVAH) -force -classpath jvpp-registry/target -d jvpp-registry io.fd.vpp.jvpp.JVppRegistryImpl
+       @$(JAVAC) -source 8 -target 8 -d jvpp-registry/target -h jvpp-registry $^
        @touch jvpp-registry.ok

 define japigen
@@ -83,11 +81,8 @@ define japigen
          -i $(jvpp_$(1)_json_files)
        @find jvpp-$(1)/target -name \*.java > jvpp-$(1).generated.files
        @find @srcdir@/jvpp-$(1) -name \*.java > jvpp-$(1).static.files
-       @$(JAVAC) -classpath jvpp-registry/target \
-         -d jvpp-$(1)/target @jvpp-$(1).generated.files @jvpp-$(1).static.files
-       @$(JAVAH) -force \
-         -classpath jvpp-registry/target:jvpp-$(1)/target \
-         -d jvpp-$(1) io.fd.vpp.jvpp.$(1).$(2)
+        @$(JAVAC) -source 8 -target 8 -classpath jvpp-registry/target \
+         -d jvpp-$(1)/target -h jvpp-$(1) @jvpp-$(1).generated.files @jvpp-$(1).static.files
 endef

 #

Continue following the steps above, starting at the following:

• sed -i '/vpp_uses_dpdk_mlx5_pmd/s/^# //g' build-data/platforms/vpp.mk
• ...

from cnf-testbed.