sNAT via conntrack ACLs about faucet HOT 5 CLOSED

So in conntrack mode faucet shouldn't touch your packets at all, everything should be processed inside the Open vSwitch dataplane.

I haven't benchmarked OVS NATing myself so not sure what performance you can expect, though stateful connection tracking is inherently more work than regular routing so if you have a slow CPU this could be expected.

You can rule out faucet as the culprit of the slow performance by setting up NAT outside of faucet using this OVS tutorial: https://docs.openvswitch.org/en/latest/tutorials/ovs-conntrack/

Also if you post your hardware specs and OVS version, we might be able to help you tune things to get better performance.

from faucet.

Thanks for the quick response.

It turns out the performance problem was actually a regression in OVS code in the 5.15 LTS kernel that currently ships with Ubuntu 22.04 LTS (5.15.0-39-generic). It was generating a kernel WARN stack trace for every packet passed through conntrack. The warning did not break functionality but really hampered performance. Upgrading the kernel to latest mainline fixed the problem. Throughput is line rate 1Gb/s again for a basic NAT flow.

Here's the relevant kernel issue and commit discussion for anyone that's curious:

WARNING: CPU: 4 PID: 0 at include/net/netfilter/nf_conntrack.h:175 __ovs_ct_lookup+0x36c/0x3e0 [openvswitch]

https://lore.kernel.org/netdev/[email protected]/T/

I would still be interested in any general performance tuning suggestions if you have any advice or relevant links. I'm on OVS 2.17.0 with plans to run on a variety of hardware in campus environments, but the following is the lab system in question.

Machine:
  Type: Kvm System: Supermicro product: SYS-E300-8D
  Mobo: Supermicro model: X10SDV-TP8F
Memory:
  RAM: total: 15.53 GiB used: 1.22 GiB (7.8%)
Network:
  Device-1: Intel Ethernet X552 10 GbE SFP+ driver: ixgbe
  Device-2: Intel Ethernet X552 10 GbE SFP+ driver: ixgbe
  Device-3: Intel I210 Gigabit Network driver: igb
  Device-4: Intel I210 Gigabit Network driver: igb
  Device-5: Intel I350 Gigabit Network driver: igb
  Device-6: Intel I350 Gigabit Network driver: igb
  Device-7: Intel I350 Gigabit Network driver: igb
  Device-8: Intel I350 Gigabit Network driver: igb

I'd also like to say that Faucet is a really great project. I've worked in the "SDN" space for a long time and Faucet so far seems to have a great balance between flexibility and approachability compared to the other popular "open" SDN controllers out there. I hope the community can keep the momentum going.

Are you open to doc PRs? I would like to add the NAT example to your "Recipe Book", but I noticed so far you seem to have only full-blown tutorials.

from faucet.

Great detective work, I'm not running any Ubuntu 22.04 machines yet so hadn't come across this issue.

We of course are happy to take doc only PRs, especially in the area of conntrack support for faucet since it's such a new feature. I'm happy to work with you to get this landed.

I'm working on writing some stateful firewall rulesets for faucet, but I haven't got them to the stage where I can publish them yet, but happy to share my working ruleset if you're interested in doing more than NAT.

from faucet.

I am definitely interested in doing more than NAT, so any stateful rules you're willing to share would be very helpful! That was actually going to be my next on my list of things to try...

I will try and put together a PR with some conntrack docs. Hopefully sometime next week. Since you're working on stateful rules, do you think it makes more sense to try and put together a new Tutorial about conntrack, or just begin to populate the Recipe Book?

from faucet.

It turns out the performance problem was actually a regression in OVS code in the 5.15 LTS kernel that currently ships with Ubuntu 22.04 LTS (5.15.0-39-generic). It was generating a kernel WARN stack trace for every packet passed through conntrack. The warning did not break functionality but really hampered performance. Upgrading the kernel to latest mainline fixed the problem.

FWIW here's a link to the Ubuntu bug for the kernel issue along with a link to the fix.

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1978865

from faucet.