Comments (5)
Google doc with Summary of networking issues found with NF testing
from cnf-testbed.
Network configuration requirements for Packet
For the initial box-by-box benchmark and comparison we are only interested in the performance of individual VNFs and CNFs, with focus on the data plane performance (throughput) and memory usage (resident set size, RSS).
For these tests the data plane network should be as simple as possible, which can be realized by attaching VFs (Virtual Functions) directly to the VNF or CNF being tested. The traffic generator (Pktgen) runs on a separate instance and can be connected via either PFs (Physical Functions) or VFs depending on the network configuration provided by Packet. Given that the current configuration runs both data plane and management / external networks through the same NIC, the connections will likely be based on VFs created from single port / PF, as the other port will be handling management and external network.
Below is a small diagram showing how this implementation can be realized using two Packet instances. Note that the data plane network will need to be configured with a VLAN to act as an L2 connection between instances.
The main requirement for this to work is that the necessary flags for SR-IOV are set in the BIOS (it should be possible to configure this via the Packet.net customer portal)
There are other configurations for the “System Under Test” instance that involves the use of VPP between the NIC and the VNFs and CNFs, but this only changes the software requirements, and should not change the requirements for Packet.
from cnf-testbed.
Network configuration requirements for fd.io
The requirements for fd.io are very similar to those for Packet. The biggest difference is seen in the connections between instances, as the fd.io CSIT testbeds have NICs dedicated for data plane traffic using point-to-point connections that removes the need for configuring the data plane network.
The diagram below shows the configuration that has been used for benchmarks.
By default the testbeds don’t fully support IOMMU. This can be fixed by enabling Intel. VT for Directed I/O (VT-d) in the BIOS (listed under Chipset -> North Bridge -> IIO Configuration).
Details about testbeds and network connections are available through CSIT
from cnf-testbed.
Networking requirements for VPP/DPDK NFs
Focusing on the “System Under Test” instance, there are several ways that this can be configured to support multiple VNFs and CNFs. Examples of how this can be done is shown in the diagram below.
- Starting at the top, the connections between NIC (VF) and VPP (PMD) do have a dependency on the hardware being used, more specifically the type of NIC. The PMD (Poll Mode Driver) varies depending on the NIC, but PMDs for both Intel and Mellanox (ConnectX-4) have been installed and should work (not yet tested with traffic on instances provided by Packet)
- There is a high probability that PMDs will have to be compiled using DPDK before they can be used by VPP. For Mellanox in particular this involves a few extra steps
- Going to the VNF, it is possible to create a connection using a Vhost interface in VPP, which is passed to the VM where the guest will see it as a “Virtio PCI” device.
- For the CNF there are two ways of creating network connections to VPP. Both rely on having resources shared between the host and the guest, where packets can be shared and accessed by both VPP and the CNF.
- Vhost - Vhost: Similar to the method used for VNFs. The difference here is that the CNF will have to use the vhost interface directly, where the VNF has it mapped to a “Virtio PCI” device
- Memif - Memif: A relatively new implementation that is quite similar to Vhost, but with the potential for higher performance.
Most of these connections have been partially tested on Packet hardware.
from cnf-testbed.
Issues seen during deployment and testing
Initial deployments were done on a single “all-in-one” instance, meaning both traffic generator and NF was running side by side. The data plane network was implemented using the default bridge implementations available in the frameworks used for virtualization, Vagrant (libvirt) for VMs and Docker for containers. Both of these work in similar fashion, as can be seen in the diagram below.
While both of these deployments did work, the amount of traffic that can be handled by these host bridges is very limited, to the point where the VNFs/CNFs would only be utilizing a few percent of their available resources. Variations based on these configurations were also tested, e.g. using TCP tunnels between the traffic generator and NF, but the results were similar to what was observed using the host bridges.
A different approach using an “all-in-one” instance was also tested, this one using VPP as the data plane network inside a single instance. The diagram below shows the configuration differences when testing either VNFs or CNFs.
The traffic generator is deployed as a VNF in both scenarios, as it currently only supports attaching to PCI devices, which is done through the Vhost to “Virtio PCI” mapping that happens in the VM. This solution removes the bottleneck that was seen previously with host bridges.
This solution is however also not ideal, as the traffic generator only supports a single queue per “Virtio PCI” interface, which limits the number of CPU cores that can be used to one per interface, or two in total with the configuration used. While the throughput is several times higher compared to the bridge configuration, it is still too low to fully utilize the resources available for the VNFs/CNFs.
List of issues with references
-
Throughput bottleneck using host bridge (and TCP tunnel) for data plane network
- Only used for initial vDNS benchmarks (not part of the primary data plane)
- Low throughput was expected and verified using mockup NFs (running VPP)
- #27
-
Throughput bottleneck using “All-in-one” instance with VPP
- Used for initial vBNG benchmarks
- Bottleneck caused by limitations in Pktgen VNF
- Only 1 queue per port, limiting the number of cores to 2 total
- #35
-
VPP doesn’t work with Mellanox ConnectX-4 (Workaround found)
- VPP hangs when Mellanox interfaces are set in “up” state
- Issue was observed with VPP v18.07
- Workaround: Downgrade VPP to v18.04
- No benchmark has been done, but packets were observed across VFs on the NIC
- #38
-
SSH broken in newer version of Ubuntu box provided by Vagrant
- Solved by forcing the use of an older version
- #41
from cnf-testbed.
Related Issues (20)
- Two issues seen with current vSwitch (VPP) installation HOT 1
- K8s provisioning ansible task fails - dict object' has no attribute 'v1.18' HOT 11
- Update k8s-infra to dynamically use latest kubespray release HOT 1
- Create stand-alone Example VPP IP Forwarder CNF
- Update remaining images and use-cases (VPP and GoGTP) HOT 1
- NSM install fails after fresh cnf testbed setup HOT 5
- Cluster starts misbehaving after a while HOT 3
- Move/cleanup old and outdated "baseline" directories
- Packetgen provisioning through Ansible fails
- Incorrect configuration for VPP in 3c2n based use-cases HOT 1
- VLAN creation issues deploying cnf-testbed to Packet HOT 5
- [Documentation] update Install Guide HOT 1
- [Documentation] update README
- Add diagram to snake and pipeline use cases HOT 2
- [Maintenance] Review Packet resources in use HOT 13
- Cleanup of unused content (scripts, playbooks, roles)
- Issue with Kubespray v2.14.0 (likely related to Multus configuration) HOT 4
- Update use of Packet to Equinix Metal HOT 1
- How to control the behavior of enb? HOT 23
- New Equinix Metal server type "n3.xlarge"
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cnf-testbed.