[Bug]: latency regression about amzn-drivers HOT 13 CLOSED

Thank you for raising this issue @wrfeewrtgaqwfrwaq

While we look into this further, can you please share more information?

1. Have you observed this in previous versions of the ENA driver? Your comparison is between the base driver (in-tree) which comes with the kernel vs the github driver (out-of-tree), which differ.
2. Which zero-copy patch have you applied?
3. Can you please share more info on which round-trip tests you've done? There are different UDP-based tools.

from amzn-drivers.

1. I didn't do bisection so I can only compare old vs new driver. I'll try doing some more testing today
2. Latest in this thread #221 (link)[https://github.com/amzn/amzn-drivers/files/14994782/0001-linux-ena-Fix-some-bugs-in-AF-XDP-support.patch] But as I said, they patch doesn't influence results, the UDP is slower both with and without the patch on new driver
3. It's in house made test bench that just sends a bunch of packets with tcs timestamp, "server" just mirrors them back and client calculates average round trip time. If you have any recommendations of more popular testbench I'm more than happy to run it.

from amzn-drivers.

Thank you for sharing this @wrfeewrtgaqwfrwaq
The reason I asked about the patch is the fact that it implies AF_XDP utilization and testing, while the discussion is more generic (per my understanding of the issue) and is unrelated to XDP and/or AF_XDP. Trying to reduce the number of variables here :)

from amzn-drivers.

So the plan was to try running onload on machines but after seeing much, much worse latency with onload than without I dug dipper only to find that latency spike is actually a driver issue.

from amzn-drivers.

Thank you for the information @wrfeewrtgaqwfrwaq

There are many factors that may impact latency, and getting stable latency results require configurations. Additional information about this topic can be found in https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ena-improve-network-latency-linux.html.

It is interesting that you point out that TCP is not affected but UDP is, the distinction between TCP and UDP is not in the ENA driver but rather in the upper layers of the stack.

I've tried running uperf on the same instance configuration and haven't observed any noticeable differences in TCP or UDP.

One of the differences between the in-tree driver in 6.8.9 and the github driver is the fact that adaptive interrupt moderation is not enabled by default in the in-tree driver (is enabled from kernel 6.9). Having said that, this configuration alone shouldn't cause an increase to the numbers that you're observing.

In the experiments that you've conducted, can you please share some percentiles?

from amzn-drivers.

I rerun the experiments using uperf to unify the information and not leak any internal information. The following configuration was used for testing:

<?xml version="1.0"?>
<profile name="netperf">
  <group nthreads="1">
        <transaction iterations="1">
            <flowop type="connect" options="remotehost=$h protocol=tcp
            wndsz=50k tcp_nodelay"/>
        </transaction>
        <transaction iterations="100000">
            <flowop type="write" options="size=90"/>
            <flowop type="read" options="size=90"/>
        </transaction>
        <transaction iterations="1">
            <flowop type="disconnect" />
        </transaction>
  </group>
</profile>

I got results suggesting the performance is exactly the same, I'll need to further investigate if our test bench generates different traffic or is broken.

from amzn-drivers.

Thank you for the information @wrfeewrtgaqwfrwaq

There are many factors that may impact latency, and getting stable latency results require configurations. Additional information about this topic can be found in https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ena-improve-network-latency-linux.html.

It is interesting that you point out that TCP is not affected but UDP is, the distinction between TCP and UDP is not in the ENA driver but rather in the upper layers of the stack.

I've tried running uperf on the same instance configuration and haven't observed any noticeable differences in TCP or UDP.

One of the differences between the in-tree driver in 6.8.9 and the github driver is the fact that adaptive interrupt moderation is not enabled by default in the in-tree driver (is enabled from kernel 6.9). Having said that, this configuration alone shouldn't cause an increase to the numbers that you're observing.

In the experiments that you've conducted, can you please share some percentiles?

You're saying that difference between TCP and UDP is not made in the driver but in TCP/IP stack. What happens then when I enable ena express acceleration for udp? Where exactly the translation happens?

from amzn-drivers.

You're saying that difference between TCP and UDP is not made in the driver but in TCP/IP stack. What happens then when I enable ena express acceleration for udp? Where exactly the translation happens?

Hi @wrfeewrtgaqwfrwaq,
I was referring to the changes in how the network stack handles TCP and UDP traffic, but I am not sure that's relevant here and wouldn't be the reason for such a delta.

Can you share where and/or how ENA express is relevant here?

In addition, you're welcome to contact me via Email in [email protected].

from amzn-drivers.

Hi @wrfeewrtgaqwfrwaq

Do you have any further queries?

from amzn-drivers.

Hi @davidarinzon ,
sorry, I've been on holidays. I'm still working on bisecting the driver and differentiating the benchmarks. I'll get back with the results soon!

from amzn-drivers.

I tried compiling all driver versions to bisect where performance changes but I'm stuck around 2.8.0 (going backwards from the newest release) with the following error:

/home/mpietrzak/amzn-drivers/kernel/linux/ena/ena_ethtool.c:1220:20: error: initialization of ‘void (*)(struct net_device *, struct ethtool_ringparam *, struct kernel_ethtool_ringparam *, struct netlink_ext_ack *)’ from incompatible pointer type ‘void (*)(struct net_device *, struct ethtool_ringparam *)’ [-Werror=incompatible-pointer-types]
  .get_ringparam  = ena_get_ringparam,
                    ^~~~~~~~~~~~~~~~~
compilation terminated due to -Wfatal-errors.
cc1: some warnings being treated as errors

Do you have any ideas how to fix it?

In case it matters I'm not compiling using native system headers but different ones by patching BUILD_KERNEL in Makefile.

from amzn-drivers.

Hi @wrfeewrtgaqwfrwaq
Given our build system and how it works, I am not sure your method is going to work properly.
But, if you wish, you can contact me via Email at [email protected] and we can look at the technical details and howto overcome the obstacles.
My assumption is that the issue is that the signature of this function has changed in RHEL >= 8.7 and RHEL >= 9.1.
We expect the correct distribution versions when compiling.
(https://github.com/amzn/amzn-drivers/blame/master/kernel/linux/ena/kcompat.h#L914)

from amzn-drivers.

Thanks for the tip! I set up small EC2 instance with all tooling required and compiled the drivers there. I run some more tests, this time on x7i machines and all latency discrepancies seem to disappear. ena_linux_2.8.5 , ena_linux_2.9.0, ena_linux_2.10.0, ena_linux_2.12.1 and in-tree modules show latency within one sigma.

from amzn-drivers.