xilinx-cns / onload Goto Github PK

the project builder attempts to do the right thing for systemd-based builds, but falls short when building in a containerized environment because process 1 isn't a good signal for "are we building for systemd". container environments often have a different process 1.

Hi, first of all thank you for sharing this code.

I'm trying to run on a virtual machine for some testing but not able to load onload driver to the kernel.

I use stock Ubuntu 20.10 distribution and I followed the steps in DEVELOPING.md. Namely these commands:

export PATH="$PWD/scripts:$PATH"

mmakebuildtree --driver
mmakebuildtree --user

make -C "$(mmaketool --toppath)/build/$(mmaketool --driverbuild)"
make -C "$(mmaketool --toppath)/build/$(mmaketool --userbuild)"

"$(mmaketool --toppath)/build/$(mmaketool --driverbuild)/driver/linux/load.sh" onload

The output I keep getting is

unload.sh: /sbin/rmmod virtual_bus
unload.sh: /sbin/rmmod sfc_driverlink
NET_OPT is
CHAR_OPT is
modprobe: FATAL: Module mtdchar not found in directory /lib/modules/5.8.0-38-generic
ERROR: Did not find sfc_control in /proc/devices
load.sh: no Solarflare NICs detected in this machine

While it makes sense that no Solarflare NIC is detected, a module called mtdchar doesn't seem to exist based on what I see in the kernel source.

Same behaviour was observed on Ubuntu 20.04. Any guesses to what am I doing wrong ?

Given that now AF_XDP support has been enabled, please consider the following knowledge tasks:

1. Revise and release Openonload User Guide.
2. Revise and release TCP_DIRECT User Programming Guide.
3. Update instructions for Linux OS distro specific instructions for building, installing and enabling onload.
4. Please provide instructions for building, installing and enabling onload for sample Cloud service provider like Azure for SR-IOV or virtIO via hardware offload. A sketch is also enough.

Thanks!

I followed the following steps:

1. cloned this repository
2. used the steps in DEVELOPING.md to build from repository
3. ran ./onload_tool reload
4. wrote the name of the interface in /sys/module/sfc_resource/afxdp/register with sudo privileges
5. ran my application using the following syntax: "sudo onload ./bts_sut" (it is a udp to tcp application. It receives udp packets from a certain interface and sends tcp replies)

On running this application I get the following output:

onload: WARNING: EF_TCP_SYNRECV_MAX=16384 and EF_MAX_ENDPOINTS=8192 are inconsistent.
onload: EF_TCP_SYNRECV_MAX is set to 16384 based on /proc/sys/net/ipv4/tcp_max_syn_backlog value and assuming up to 4 listening sockets in the Onload stack
onload: Too few endpoints requested: ~4 syn-receive states consume one endpoint. 
oo:bts_sut[13180]: Using Onload 0d46412079ceb6c162e6cb5ceabf26d610c5a5d4 [6]
oo:bts_sut[13180]: Copyright 2019-2021 Xilinx, 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks

TCP Connection Failed 

When I check syslog I get the following information:

Feb 10 14:28:34 bts-s1 kernel: [12726.892817] [onload] [6]: WARNING: huge pages are incompatible with AF_XDP. Disabling hugepage support.
Feb 10 14:28:34 bts-s1 kernel: [12726.916274] [onload] oof_socket_add_full_hw: 6:3 ERROR: FILTER TCP 10.0.0.17:57471 10.0.0.21:29999 failed (-95)
Feb 10 14:28:34 bts-s1 kernel: [12726.916966] [sfc efhw] af_xdp_flush_rx_dma_channel: FIXME AF_XDP
Feb 10 14:28:34 bts-s1 kernel: [12726.917166] [sfc efhw] af_xdp_flush_tx_dma_channel: FIXME AF_XDP

Machine specification are as follows::

1. 82599 NIC
2. Operating System: Ubuntu 18.04.4 LTS
3. Kernel: Linux 5.7.0+
4. Machine: Intel Xeon D2166

I use this application often, it works perfectly fine without onload af_xdp solution. I was looking into this in order to accelerate my application without changing the source code
Is there anything that I am doing wrong? Or is there anything I can do to make this work?

Hello @maciejj-xilinx

I have been trying to get Onload working on a:

40GBe NiC (X710, 4 x 10 bonded)
i40e driver, version: 2.8.20-k (its a stock version, same issue with 2.13 i40e driver update as well)
64 core Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
Centos 8, 4.18.0-240.1.1.el8_3.x86_64

So onload compiles well, but when I go to reload it, I see this error:

Please see dmesg as well below:

Any help is appreciated.

Could you issue a new supported/official OpenOnload release for Linux kernel version 5.10.x? It's the latest longterm kernel and a few distributions are now using it.

I have a program which captures SIGINT and does not close the application.
The application itself is a server which blocks on accept() call.

While the application is blocking on accept() if it receives SIGINT:

On kernel:
accept does not return

On Onload:
accept returns and the next statement is executed

Machine specification are as follows::

Solarflare Communications XtremeScale SFC9250 NIC
Operating System: Ubuntu 20.04.1
Kernel: Linux 5.8.0-55-generic

make -C "$(mmaketool --toppath)/build/$(mmaketool --driverbuild)"
.
.
.
/home/paolini/onload/build/x86_64_linux-4.15.0-135-generic/driver/linux_net/drivers/net/ethernet/sfc/kernel_compat.c: In function ‘flow_rule_alloc’:
/home/paolini/onload/build/x86_64_linux-4.15.0-135-generic/driver/linux_net/drivers/net/ethernet/sfc/kernel_compat.c:642:17: error: implicit declaration of function ‘struct_size’; did you mean ‘bd_set_size’? [-Werror=implicit-function-declaration]
rule = kzalloc(struct_size(rule, action.entries, num_actions),
^~~~~~~~~~~
bd_set_size
/home/paolini/onload/build/x86_64_linux-4.15.0-135-generic/driver/linux_net/drivers/net/ethernet/sfc/kernel_compat.c:642:35: error: ‘action’ undeclared (first use in this function); did you mean ‘tc_action’?
rule = kzalloc(struct_size(rule, action.entries, num_actions),
^~~~~~
tc_action
/home/paolini/onload/build/x86_64_linux-4.15.0-135-generic/driver/linux_net/drivers/net/ethernet/sfc/kernel_compat.c:642:35: note: each undeclared identifier is reported only once for each function it appears in
cc1: all warnings being treated as errors
scripts/Makefile.build:330: recipe for target '/home/paolini/onload/build/x86_64_linux-4.15.0-135-generic/driver/linux_net/drivers/net/ethernet/sfc/kernel_compat.o' failed
make[8]: *** [/home/paolini/onload/build/x86_64_linux-4.15.0-135-generic/driver/linux_net/drivers/net/ethernet/sfc/kernel_compat.o] Error 1

👋🏽

Our Setup

Tech stack:

• Solarflare Card
• Kubernetes
• Multus
• VNIC/MACVLAN

We are using the multus CNI plugin to:

• create a VNIC backed by the Solarflare card, and,
• attaching the VNIC as a secondary network interface to each pod

Our Concern

Pods are able to view traffic associated with other pods via onload_tcpdump.

Additionally, onload_stackdump shows Onload Stacks associated with card regardless of which pod the stack was created in.

Question

Is there a way to achieve isolation at the device level, such that pods are restricted to viewing traffic and stacks associated with the VNIC they have access to?

Thanks in advance

Hello, onload team.

I am reading SF-104474-CD-33_Onload_User_Guide.pdf，I see the latency test results as follow:
Let us focus on the last acceleration mode, Kernel, the latency is very lower in this mode.
1、Is this mode uses linux kernek net stack ?
2、As far as I know, the latency of Kernel is bigger than 12usec，I want to know waht lead to this lower latency ? Solarflare NICS ?

Hello Onload Team,

I have been testing Onload with AF_XDP support on 10 GBe Intel 82599. I have updated the driver of the NiC to 5.9.4. This version has the support for Zero Copy primitive of AF_XDP as referenced here:

But when I start onload by editing the latency profile (made a new copy to latency-af-xdp.opf), please see below:

and start the application, I see no stacks being created.

Also, as a result, the config var is not set either.

Requesting your team to look at this.

Please see: When EF_AF_XDP_ZEROCOPY is not enabled, the stacks are created fine and with ixgbe ZC enabled, for payloads under 3KB, I do see a marginal increase of 5% in throughput.

When i want to use AF_XDP, i follow the next instruction:
"For each interface on which Onload is to use AF_XDP, execute the following:
echo ens2f0 > /sys/module/sfc_resource/afxdp/register“

But i cannot write anything into "/sys/module/sfc_resource/afxdp/register“. I try echo and vim. They have failed.
The file is 777, and i am root. I have "setenforce 0". But I also cannot write anything into "/sys/module/sfc_resource/afxdp/register“.
[centos 8, linux 5.14.21]
Thanks

Hi,

I'm trying to run onload on a linux veth interface pair on Ubuntu 20.10:
Linux ubuntu 5.8.0-44-generic #50-Ubuntu SMP Tue Feb 9 06:29:41 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

I create a veth pair and register the interfaces as follows:

ip addr add 10.1.0.1/24 dev veth0
ip addr add 10.1.0.2/24 dev veth1
ip link set veth0 up
ip link set veth1 up
ip addr add 10.1.0.2/24 dev veth1^C
echo veth0 > /sys/module/sfc_resource/afxdp/register
echo veth1 > /sys/module/sfc_resource/afxdp/register

When trying a simple command:
/home/adrien/onload/scripts/onload netcat -l 10.1.0.1 10000

I get the following kernel panic:

[  636.641583] sfc_driverlink: module verification failed: signature and/or required key missing - tainting kernel
[  636.642225] Solarflare driverlink driver v5.3.8.1011 API v32.0
[  636.675918] Solarflare NET driver v5.3.8.1011
[  637.294107] [sfc efrm] init_sfc_resource: RESOURCE driver starting
[  637.294108] [sfc efrm] efrm_driver_ctor: driver created
[  637.294237] [sfc efrm] efrm_driverlink_register:
[  637.294238] Efx driverlink registering resource driver
[  637.368676] [onload] Onload <dev-snapshot>
[  637.368680] [onload] Copyright 2019-present Xilinx, 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks
[  637.476205] onload_cp_server[31076]: Spawned daemon process 31080
[  716.488784] IPv6: ADDRCONF(NETDEV_CHANGE): veth1: link becomes ready
[  716.488927] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[  742.890243] [sfc efrm] nondl_register_store: cannot detect number of channels for device veth0 assuming 1
[  742.890245] [sfc efrm] efrm_nondl_register_device: register veth0
[  742.890249] [sfc efrm] Using VI range 0+(0-1)<<0
[  742.890250] [sfc efrm] veth0 type=4:\x000 ifindex=4
[  742.890277] [sfc efrm] efrm_driverlink_resume:
[  742.890971] [sfc efrm] veth0 index=0 ifindex=4
[  742.890977] [onload] oo_nic_add: ifindex=4 oo_index=0
[  745.858300] [sfc efrm] nondl_register_store: cannot detect number of channels for device veth1 assuming 1
[  745.858302] [sfc efrm] efrm_nondl_register_device: register veth1
[  745.858308] [sfc efrm] Using VI range 0+(0-1)<<0
[  745.858309] [sfc efrm] veth1 type=4:\x000 ifindex=3
[  745.858340] [sfc efrm] efrm_driverlink_resume:
[  745.858935] [sfc efrm] veth1 index=1 ifindex=3
[  745.858942] [onload] oo_nic_add: ifindex=3 oo_index=1
[  791.998897] [sfc efrm] efrm_vi_rm_delayed_free: 0000000058301cf0
[  791.998907] [sfc efrm] efrm_vi_rm_delayed_free: 0000000058301cf0
[  792.000153] [onload] [0]: WARNING: huge pages are incompatible with AF_XDP. Disabling hugepage support.
[  792.049776] BUG: kernel NULL pointer dereference, address: 0000000000000050
[  792.049783] #PF: supervisor read access in kernel mode
[  792.049784] #PF: error_code(0x0000) - not-present page
[  792.049786] PGD 0 P4D 0 
[  792.049790] Oops: 0000 [#1] SMP PTI
[  792.049794] CPU: 0 PID: 31163 Comm: netcat Tainted: G           OE     5.8.0-44-generic #50-Ubuntu
[  792.049796] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
[  792.049823] RIP: 0010:efrm_filter_insert+0x55/0x360 [sfc_resource]
[  792.049826] Code: 81 16 01 00 48 89 c7 49 89 c4 e8 36 35 00 00 48 85 c0 0f 84 ec 02 00 00 49 89 c5 48 8b 80 50 05 00 00 48 c7 45 c0 00 00 00 00 <4c> 8b 70 50 4d 85 f6 75 03 4c 8b 30 48 c7 c7 48 aa 9f c0 e8 93 64
[  792.049828] RSP: 0018:ffffadf5c1d038d0 EFLAGS: 00010286
[  792.049830] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000001027
[  792.049832] RDX: 0000000000000001 RSI: 00000000fffffe01 RDI: ffffffffc09dd3f9
[  792.049833] RBP: ffffadf5c1d03910 R08: 0000000000000008 R09: 0000000000000262
[  792.049834] R10: ffffadf5c1d0393c R11: 0000000000000000 R12: ffffa013909c6c00
[  792.049835] R13: ffffa013b4cbf000 R14: 0000000000000000 R15: ffffadf5c1d0393c
[  792.049838] FS:  00007f85a4017300(0000) GS:ffffa013b9e00000(0000) knlGS:0000000000000000
[  792.049839] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  792.049841] CR2: 0000000000000050 CR3: 0000000082626002 CR4: 00000000001606f0
[  792.049842] Call Trace:
[  792.049882]  oo_hw_filter_set_hwport+0x2dd/0x860 [onload]
[  792.049899]  oo_hw_filter_add_hwports+0x1a0/0x240 [onload]
[  792.049916]  oo_hw_filter_set+0x1f/0x40 [onload]
[  792.049931]  __oof_hw_filter_set.constprop.0+0x18b/0x770 [onload]
[  792.049948]  __oof_socket_add_wild+0xfa/0x180 [onload]
[  792.049964]  oof_socket_add_wild.constprop.0+0x174/0x640 [onload]
[  792.049969]  ? kmalloc_order+0x63/0x80
[  792.049972]  ? kmalloc_order_trace+0x24/0xa0
[  792.049998]  oof_socket_add+0x1654/0x1bf0 [onload]
[  792.050002]  ? ima_file_check+0x5b/0x80
[  792.050016]  tcp_helper_endpoint_set_filters+0x338/0x1190 [onload]
[  792.050031]  efab_ep_filter_set+0x38/0x80 [onload]
[  792.050041]  oo_fop_unlocked_ioctl+0x213/0x3b0 [onload]
[  792.050045]  ? _cond_resched+0x1a/0x50
[  792.050050]  ksys_ioctl+0x8e/0xc0
[  792.050052]  __x64_sys_ioctl+0x1a/0x20
[  792.050055]  do_syscall_64+0x49/0xc0
[  792.050058]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  792.050060] RIP: 0033:0x7f85a415231b
[  792.050063] Code: 89 d8 49 8d 3c 1c 48 f7 d8 49 39 c4 72 b5 e8 1c ff ff ff 85 c0 78 ba 4c 89 e0 5b 5d 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1d 3b 0d 00 f7 d8 64 89 01 48
[  792.050065] RSP: 002b:00007fff444bbbd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  792.050067] RAX: ffffffffffffffda RBX: 00007f85a4016698 RCX: 00007f85a415231b
[  792.050068] RDX: 00007fff444bbbec RSI: 00000000400c5a23 RDI: 0000000000000005
[  792.050070] RBP: 0000000000000000 R08: 00000000005ffb40 R09: 0000000000000007
[  792.050071] R10: 00007f85a31dbc00 R11: 0000000000000246 R12: 0000000000000005
[  792.050072] R13: 00000000000007ff R14: 00007f85a43d27f8 R15: 00000000001ffc00
[  792.050074] Modules linked in: veth onload(OE) sfc_char(OE) sfc_resource(OE) sfc(OE) virtual_bus(OE) sfc_driverlink(OE) nf_flow_table nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c vdpa mtd 8021q garp mrp stp llc mdio hwmon_vid i2c_algo_bit crc32_generic mii rfcomm bnep vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock nls_iso8859_1 intel_rapl_msr intel_rapl_common crct10dif_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper rapl vmw_balloon snd_ens1371 snd_ac97_codec gameport ac97_bus snd_pcm snd_seq_midi joydev input_leds snd_seq_midi_event serio_raw btusb btrtl snd_rawmidi btbcm btintel bluetooth snd_seq uvcvideo ecdh_generic videobuf2_vmalloc ecc videobuf2_memops videobuf2_v4l2 snd_seq_device videobuf2_common snd_timer videodev mc snd soundcore vmw_vmci mac_hid sch_fq_codel vmwgfx ttm drm_kms_helper cec rc_core fb_sys_fops syscopyarea sysfillrect sysimgblt parport_pc ppdev lp drm parport ip_tables x_tables autofs4 hid_generic
[  792.050119]  usbhid hid crc32_pclmul psmouse mptspi mptscsih mptbase e1000 scsi_transport_spi ahci libahci i2c_piix4 pata_acpi
[  792.050130] CR2: 0000000000000050
[  792.050132] ---[ end trace 5b17494d5fe46b2e ]---
[  792.050142] RIP: 0010:efrm_filter_insert+0x55/0x360 [sfc_resource]
[  792.050144] Code: 81 16 01 00 48 89 c7 49 89 c4 e8 36 35 00 00 48 85 c0 0f 84 ec 02 00 00 49 89 c5 48 8b 80 50 05 00 00 48 c7 45 c0 00 00 00 00 <4c> 8b 70 50 4d 85 f6 75 03 4c 8b 30 48 c7 c7 48 aa 9f c0 e8 93 64
[  792.050146] RSP: 0018:ffffadf5c1d038d0 EFLAGS: 00010286
[  792.050148] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000001027
[  792.050149] RDX: 0000000000000001 RSI: 00000000fffffe01 RDI: ffffffffc09dd3f9
[  792.050150] RBP: ffffadf5c1d03910 R08: 0000000000000008 R09: 0000000000000262
[  792.050151] R10: ffffadf5c1d0393c R11: 0000000000000000 R12: ffffa013909c6c00
[  792.050152] R13: ffffa013b4cbf000 R14: 0000000000000000 R15: ffffadf5c1d0393c
[  792.050154] FS:  00007f85a4017300(0000) GS:ffffa013b9e00000(0000) knlGS:0000000000000000
[  792.050156] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  792.050157] CR2: 0000000000000050 CR3: 0000000082626002 CR4: 00000000001606f0
[  792.050812] [onload] Stack [0] released with lock stuck (0x4000000020000000)```

I've also tried on an Ubuntu 20.04 VM running on AWS and I am seeing the same issue.

Thanks for your help

Adrien

Hello,

I am trying to get the Onload to offload memcached on an Azure VM, but it does not work.

Below is the error message:

root@sriov-onload1 ~]# onload -p latency memcached -m 24576 -c 1024 -t 4 -u root -l <ip-address>:11211
oo:memcached[68979]: netif_tcp_helper_alloc_u: ENODEV.
This error can occur if:
 - no Solarflare network interfaces are active/UP, or they are running
   packed stream firmware or are disabled, and
 - there are no AF_XDP interfaces registered with sfc_resource

Environment details of Azure:
VM Type: D8ds_v4
OS: Red Hat Enterprise Linux release 8.4 (Ootpa)
Kernel version: 4.18.0-305.7.1.el8_4.x86_64
CPU Make: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
vCPUs: 8

VM has access to underlying NiC via SR-IOV, with mirror pass-through mode.

NiC Model: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80)
Kernel Driver : mlx5_core, version: 5.4-1.0.3
Kernel Driver Loaded ? YES
SFC driver and Onload compiled and loaded: YES

sfc_char              110592  1 onload
sfc_resource          184320  2 onload,sfc_char
sfc                   655360  0
virtual_bus            16384  1 sfc
sfc_driverlink         16384  2 sfc,sfc_resource
mtd                    69632  1 sfc
mdio                   16384  1 sfc
vdpa                   16384  1 sfc

SR-IOV interface registered : YES, the below message comes only when we try to re-register the NiC interface.

[root@sriov-onload1 ~]# echo enP28349s1 > /sys/module/sfc_resource/afxdp/register
-bash: echo: write error: Operation already in progress.

dmesg o/p:

[    5.371350] mlx5_core 6ebd:00:02.0 enP28349s1: Link up
[    5.374995] hv_netvsc 000d3a7d-9c68-000d-3a7d-9c68000d3a7d eth0: Data path switched to VF: enP28349s1
[    5.377992] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    5.378043] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    6.454942] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
[    8.021253] fbcon: Taking over console
[    8.021576] Console: switching to colour frame buffer device 128x48
[   51.343609] hv_balloon: Max. dynamic memory size: 32768 MB
[  737.254449] Solarflare driverlink driver v5.3.9.1001 API v32.0
[  737.260284] Solarflare NET driver v5.3.9.1001
[  737.285720] Efx driverlink registering resource driver
[  737.334475] [onload] Onload <dev-snapshot>
[  737.334476] [onload] Copyright 2019-present Xilinx, 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks
[  737.439749] onload_cp_server[67516]: Spawned daemon process 67536
[  840.879660] [sfc efrm] efrm_nondl_register_device: register enP28349s1
[  840.879662] [sfc efrm] enP28349s1 type=4:
[  841.879298] [sfc efrm] enP28349s1 index=0 ifindex=3
[  841.879310] [onload] oo_nic_add: ifindex=3 oo_index=0
[  950.500153] [sfc efrm] efrm_nondl_register_device: register eth0
[  950.500156] [sfc efrm] eth0 type=4:
[  950.500796] hv_netvsc 000d3a7d-9c68-000d-3a7d-9c68000d3a7d eth0: XDP: not support LRO
[  950.500799] [sfc efrm] ?: ERROR: hardware init failed rc=-95
[  950.500800] [sfc efrm] eth0 index=1 ifindex=2
[  950.500810] [onload] oo_nic_add: ifindex=2 oo_index=1
[ 1601.579528] [sfc efhw] af_xdp_flush_rx_dma_channel: FIXME AF_XDP
[ 1601.579531] [sfc efhw] af_xdp_flush_tx_dma_channel: FIXME AF_XDP
[ 1749.414471] [sfc efhw] af_xdp_flush_rx_dma_channel: FIXME AF_XDP
[ 1749.414473] [sfc efhw] af_xdp_flush_tx_dma_channel: FIXME AF_XDP
[ 1789.819187] [sfc efhw] af_xdp_flush_rx_dma_channel: FIXME AF_XDP
[ 1789.819188] [sfc efhw] af_xdp_flush_tx_dma_channel: FIXME AF_XDP

Request your attention.

Is there a way to run an application but disable onload from taking over TCP connections? I still want to use onload to accelerate UDP. I didn't see in the documentation a way to build/disable certain aspects of onload.

vis[] is declared in include/ci/internal/ip_types.h as
"ef_vi vis[CI_MAX_VIS_PER_INTF]" where CI_MAX_VIS_PER_INTF is 1

But q_id is 1, as the assert verifies in
void ci_netif_dmaq_shove_plugin(ci_netif* ni, int intf_i, int q_id)
{
ef_vi* vi = &ni->nic_hw[intf_i].vis[q_id];
ci_assert_ge(q_id, 1);

Hence, vis[1] is out of bounds.

CC /root/src/onload-master/build/x86_64_linux-3.10.0-1160.31.1.el7.x86_64/lib/transport/ip/netif_tx.o
/root/src/onload-master/build/x86_64_linux-3.10.0-1160.31.1.el7.x86_64/lib/transport/ip/netif_tx.c: In function ‘ci_netif_dmaq_shove_plugin’:
/root/src/onload-master/build/x86_64_linux-3.10.0-1160.31.1.el7.x86_64/lib/transport/ip/netif_tx.c:203:38: error: array subscript 1 is above array bounds of ‘ef_vi[1]’ [-Werror=array-bounds]
203 | ef_vi* vi = &ni->nic_hw[intf_i].vis[q_id];
| ~~~~~~~~~~~~~~~~~~~~~~^~~~~~
In file included from /root/src/onload-master/src/include/ci/internal/ip.h:58,
from /root/src/onload-master/build/x86_64_linux-3.10.0-1160.31.1.el7.x86_64/lib/transport/ip/ip_internal.h:19,
from /root/src/onload-master/build/x86_64_linux-3.10.0-1160.31.1.el7.x86_64/lib/transport/ip/netif_tx.c:16:
/root/src/onload-master/src/include/ci/internal/ip_types.h:40:30: note: while referencing ‘vis’
40 | ef_vi vis[CI_MAX_VIS_PER_INTF];
| ^~~
/root/src/onload-master/build/x86_64_linux-3.10.0-1160.31.1.el7.x86_64/lib/transport/ip/netif_tx.c:203:38: error: array subscript 1 is above array bounds of ‘ef_vi[1]’ [-Werror=array-bounds]
203 | ef_vi* vi = &ni->nic_hw[intf_i].vis[q_id];
| ~~~~~~~~~~~~~~~~~~~~~~^~~~~~
In file included from /root/src/onload-master/src/include/ci/internal/ip.h:58,
from /root/src/onload-master/build/x86_64_linux-3.10.0-1160.31.1.el7.x86_64/lib/transport/ip/ip_internal.h:19,
from /root/src/onload-master/build/x86_64_linux-3.10.0-1160.31.1.el7.x86_64/lib/transport/ip/netif_tx.c:16:
/root/src/onload-master/src/include/ci/internal/ip_types.h:40:30: note: while referencing ‘vis’
40 | ef_vi vis[CI_MAX_VIS_PER_INTF];
| ^~~
cc1: all warnings being treated as errors
make[5]: *** [scripts/Makefile.build:334: /root/src/onload-master/build/x86_64_linux-3.10.0-1160.31.1.el7.x86_64/lib/transport/ip/netif_tx.o] Error 1
make[4]: *** [Makefile:1316: module/root/src/onload-master/build/x86_64_linux-3.10.0-1160.31.1.el7.x86_64/lib/transport/ip] Error 2

Hello, i tried to build openonload on CentOS8(linux kernel 5.15, 12th intel). But it fail because of many errors like [-Werror]. How to fix it. Thank you.

I think I came pretty close to using onload exclusively from within docker.
I know this messes up security and is not standard practice when working with drivers but it offers some portability if you have many servers.

I managed to build, install (user and kernelmode binaries) and also call onload_tool reload from within docker without anything installed on the host. And I see the driver being loaded in the host's journal.

However using onload gives this:

$ onload ping localhost
oo:ping[104]: ci_netif_init: failed to get local control plane handle: -2
PING localhost(localhost (::1)) 56 data bytes

The culprit seems to be the onload_cp_server component, which when I start manually (also within docker), gives:

$ sudo onload_cp_server
onload_cp_server[121]: ERROR: failed to register MIB memory: Machine is not on the network
onload_cp_server[121]: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
onload_cp_server[121]: !!! Onload Control Plane server has FAILED TO START !!!
onload_cp_server[121]: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I am of course running docker with --net host --privileged and the error message above has led me to try adding --ipc host, but to no avail. By any chance does anyone know why is it saying "Machine is not on the network"? Or is this utterly impossible to do.

We use DKMS for delivery of Onload to our production machines. onload_mkdkms works fine, but produces an RPM that can't build on a default install.

libcap-devel should be added as a dependency for the DKMS RPM, to avoid the error:

sys/capability.h: No such file or directory

I feel like an idiot for asking this here, could not find a conclusive answer anywhere. Is it possible to run OpenOnload on non solarflare / Xilinx hardware. Aka if a setup my hosting from AWS, is it possible to configure OpenOnload to work on the ENA driver that automatically comes with that machine?

Firstly, onload's external headers don't parse correctly with clang, so even if you build onload with GCC and then everything else uses clang, you can't #include onload from clang without error. Here is our patch:

        # openonload headers included by XXX are not compatible with clang, so fix that now
        file(GLOB_RECURSE openonloadheaders "${XXX_ROOT_DIR}/thirdparty/openonload/*/debug.h")
        foreach(openonloadheader ${openonloadheaders})
          file(READ "${openonloadheader}" origcontents)
          string(REPLACE "[%s=\"IPX_FMT\"]\"" "[%s=\" IPX_FMT \"]\"" newcontents "${origcontents}")
          if(NOT origcontents STREQUAL newcontents)
            message(STATUS "NOTE: Fixed illegal C++ in '${openonloadheader}'")
            file(WRITE "${openonloadheader}" "${newcontents}")
          endif()
        endforeach()

Secondly, attempting to build onload with clang fails badly. You appear to be using quite a bit of GCC-specific stuff. It would be great if you fixed this, as we currently have to special case only onload when we are compiling with clang.

Thirdly, onload won't link when built with GCC 10 due to:

ci_bpf_oobpf_elf_sh.o:(.bss+0x0): multiple definition of `citp_signal_data'
ci_bpf_oobpf_sh.o:(.bss+0x0): first defined here
collect2: error: ld returned 1 exit status
make[3]: *** [oobpfintf0.so] Error 1

Also, your configure script can't handle GCC 10:

gcc_major_version() {
  gcc --version |head -1 |sed "s/^gcc.*) \([1-9]*\)\..*$/\1/"
}

This doesn't account for GCC 10.

I'm seeing a possible regression introduced with one of these 2 commits on the 9th of June.

4267b16
2f1ba59

All else being equal, reverting the 2 commits gives me clean journalctl and getting back ~5% of performance.
The regression is in the usermode files not kernel ones.

: Using Onload <dev-snapshot> [0]
: Copyright 2019-present Xilinx, 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks
: netif: ***** EVENT QUEUE OVERFLOW *****
: ... message limit reached

Jun 19 22:05:12 kkperf2 kernel: R10: 0000000000000002 R11: 0000000000000018 R12: ffff9ac8aac04348
Jun 19 22:05:12 kkperf2 kernel: R13: 0000000000000001 R14: ffffb95082208198 R15: ffff9ac8aac04028
Jun 19 22:05:12 kkperf2 kernel: FS:  0000000000000000(0000) GS:ffff9acaca280000(0000) knlGS:0000000000000000
Jun 19 22:05:12 kkperf2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 19 22:05:12 kkperf2 kernel: CR2: 000055c8ef8dd2e8 CR3: 0000000242810002 CR4: 00000000007706e0
Jun 19 22:05:12 kkperf2 kernel: PKRU: 55555554
Jun 19 22:05:12 kkperf2 kernel: Call Trace:
Jun 19 22:05:12 kkperf2 kernel:  ? ci_netif_rx_post+0x1fc/0x740 [onload]
Jun 19 22:05:12 kkperf2 kernel:  ? ci_netif_poll_n+0x7f9/0x19a0 [onload]
Jun 19 22:05:12 kkperf2 kernel:  ? newidle_balance+0x352/0x3b0
Jun 19 22:05:12 kkperf2 kernel:  ? __switch_to_asm+0x42/0x70
Jun 19 22:05:12 kkperf2 kernel:  ? linux_tcp_helper_periodic_timer+0x138/0x1e0 [onload]
Jun 19 22:05:12 kkperf2 kernel:  ? process_one_work+0x1df/0x370
Jun 19 22:05:12 kkperf2 kernel:  ? worker_thread+0x50/0x400
Jun 19 22:05:12 kkperf2 kernel:  ? process_one_work+0x370/0x370
Jun 19 22:05:12 kkperf2 kernel:  ? kthread+0x11b/0x140
Jun 19 22:05:12 kkperf2 kernel:  ? kthread_associate_blkcg+0xa0/0xa0
Jun 19 22:05:12 kkperf2 kernel:  ? ret_from_fork+0x1f/0x30
Jun 19 22:05:12 kkperf2 kernel: ---[ end trace f4b2f513f2d0c08a ]---
Jun 19 22:05:12 kkperf2 kernel: ------------[ cut here ]------------
Jun 19 22:05:12 kkperf2 kernel: WARNING: CPU: 10 PID: 1655 at /home/oo/onload/build/x86_64_linux-5.10.42-1-lts/lib/ciul/ef10_vi.c:686 ef10_ef_vi_receive_init+0xcf/0xf0 [onload]
Jun 19 22:05:12 kkperf2 kernel: Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c br_netfilter bridge stp llc overlay intel_rapl_msr intel_rapl_>
Jun 19 22:05:12 kkperf2 kernel:  ghash_clmulni_intel sfc(OE) snd_pcm_dmaengine aesni_intel snd_pcm cec r8169 crypto_simd cryptd virtual_bus(OE) intel_gtt snd_timer glue_helper syscopyarea mdio snd realtek sysfillrect vfat vdpa ipmi_devintf mdio_devres sysimgblt fat rapl>
Jun 19 22:05:12 kkperf2 kernel: CPU: 10 PID: 1655 Comm: kworker/u41:1 Tainted: G        W  OE     5.10.42-1-lts #1
Jun 19 22:05:12 kkperf2 kernel: Hardware name: ASUS System Product Name/Pro WS W480-ACE, BIOS 2004 02/08/2021
Jun 19 22:05:12 kkperf2 kernel: Workqueue: onload-wq:0 linux_tcp_helper_periodic_timer [onload]
Jun 19 22:05:12 kkperf2 kernel: RIP: 0010:ef10_ef_vi_receive_init+0xcf/0xf0 [onload]
Jun 19 22:05:12 kkperf2 kernel: Code: 31 c0 c3 48 63 51 10 48 03 81 c0 00 00 00 48 b9 00 00 00 00 00 00 ff ff 48 85 ca 75 14 48 c1 e2 30 48 09 f2 48 89 10 31 c0 c3 <0f> 0b e9 70 ff ff ff 0f 0b eb e8 0f 0b eb 91 0f 0b eb 9c 0f 0b eb
Jun 19 22:05:12 kkperf2 kernel: RSP: 0018:ffffb9508196bcf8 EFLAGS: 00010213
Jun 19 22:05:12 kkperf2 kernel: RAX: 0000000000000188 RBX: 0000000000000008 RCX: ffff9ac8aac04348
Jun 19 22:05:12 kkperf2 kernel: RDX: 0000000000000384 RSI: 00080000161c20c0 RDI: ffffb950822077b8
Jun 19 22:05:12 kkperf2 kernel: RBP: ffffb950822081ac R08: ffffb9508220714c R09: ffffb95082208440
Jun 19 22:05:12 kkperf2 kernel: R10: 0000000000000002 R11: 0000000000000018 R12: ffff9ac8aac04348
Jun 19 22:05:12 kkperf2 kernel: R13: 0000000000000001 R14: ffffb95082208198 R15: ffff9ac8aac04028
Jun 19 22:05:12 kkperf2 kernel: FS:  0000000000000000(0000) GS:ffff9acaca280000(0000) knlGS:0000000000000000
Jun 19 22:05:12 kkperf2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 19 22:05:12 kkperf2 kernel: CR2: 000055c8ef8dd2e8 CR3: 0000000242810002 CR4: 00000000007706e0

This data is tested by EF_AF_XDP_ZEROCOPY not set, when sen EF_AF_XDP_ZEROCOPY, the performance is very low.
This test is about TCP.
In most scenarios，there are TCP long pkts，but the performance of onload is obviously lower than linux kernel.

NIC:
[root@A03-R05-I139-66-FVP3HP2 onload]# ethtool -i eth0
driver: ixgbe
version: 5.1.0-k-rh8.2.0
firmware-version: 0x8000090c, 18.3.6
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

OS:
[root@A03-R05-I139-66-FVP3HP2 onload]# cat /etc/centos-release
CentOS Linux release 8.2.2004 (Core)
[root@A03-R05-I139-66-FVP3HP2 onload]# uname -a
Linux A03-R05-I139-66-FVP3HP2.JD.LOCAL 4.18.0-240.10.1.el8_3.x86_64 #1 SMP Mon Jan 18 17:05:51 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@A03-R05-I139-66-FVP3HP2 onload]#

I have trying to get OpenOnload to work on a bare-metal box from Vultr.io

Environment Details:
CPU: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
NIC: Intel Corporation 82599 10 Gigabit Network Connection
OS: CentOS Linux release 8.2.2004 (Core)
Kernel: 4.18.0-193.28.1.el8_2.x86_64

How to Build and Install Onload:
wget https://rpmfind.net/linux/centos/8.2.2004/BaseOS/x86_64/os/Packages/kernel-headers-4.18.0-193.28.1.el8_2.x86_64.rpm
wget https://rpmfind.net/linux/centos/8.2.2004/BaseOS/x86_64/os/Packages/kernel-devel-4.18.0-193.28.1.el8_2.x86_64.rpm
yum install binutils gettext gawk gcc sed make bash glibc-common libcap-devel libmnl-devel perl-Test-Harness hmaccalc zlib-devel binutils-devel elfutils-libelf-devel libevent-devel
git clone https://github.com/Xilinx-CNS/onload.git
cd onload
scripts/onload_mkdist --release
cd onload-20201211/scripts
./onload_install
./onload_tool reload

Verify Onload driver has been built/inserted

[root@shirbare06 scripts]# lsmod | grep sfc sfc_char 98304 1 onload sfc_resource 180224 2 onload,sfc_char sfc 581632 0 virtual_bus 16384 1 sfc sfc_driverlink 16384 2 sfc,sfc_resource mtd 69632 1 sfc mdio 16384 2 sfc,ixgbe

**Check if XDP module has been enabled for via xdptools **
Install XDP tools
yum install clang llvm
dnf --enablerepo=PowerTools install libpcap-devel
git clone https://github.com/xdp-project/xdp-tools.git
cd xdp-tools
./configure
make
sudo make install

Check if xdpsock() is enabled:
`[root@shirbare06 scripts]# xdpdump -D
if_index if_name XDP program entry function

1 lo <No XDP program loaded!>
2 enp1s0 xdpsock()`

Start a sample memcached
s.chakrabarti@shirbare06 ~]$ onload -p latency memcached -m 24576 -c 1024 -t 8 -u s.chakrabarti -l 95.179.137.242:11211 oo:memcached[96975]: netif_tcp_helper_alloc_u: ERROR: Failed to allocate stack (rc=-1) See kernel messages in dmesg or /var/log/syslog for more details of this failure

Also verified stack is not being created:
[root@shirbare06 scripts]# onload_stackdump stacks

Can someone please look at this issue and fix :)

I am trying to build the user space tools on Arch linux, with GLIBC 2.33. I am getting the following errors when building:

nice cc -I. -I./../../../include -I/home/lrm/c/other/onload/src/include -I/home/lrm/c/other/onload/src/lib/transport/ip    -fPIC  -m64  -Werror -Wall -Wundef -Wpointer-arith -Wstrict-prototypes -Wnested-externs -Wno-stringop-truncation -Wno-format-truncation -Wimplicit-fallthrough=2 -Wno-array-bounds -Wno-stringop-overflow -Wno-deprecated-declarations -DTRANSPORT_CONFIG_OPT_HDR='<ci/internal/transport_config_opt_extra.h>' -include /home/lrm/c/other/onload/build/gnu_x86_64/cp_intf_ver.h -O2 -g $cflags $cppflags -c /home/lrm/c/other/onload/src/lib/transport/ip/syscall.c -o ci_ip_syscall.o
/home/lrm/c/other/onload/src/include/onload/declare_syscalls.h.tmpl:115:28: error: ‘__fxstat’ undeclared here (not in a function); did you mean ‘fstat’?
  115 | CI_MK_DECL(int           , __fxstat   , (int, int, struct stat *));
      |                            ^~~~~~~~
/home/lrm/c/other/onload/src/lib/transport/ip/syscall.c:26:62: note: in definition of macro ‘CI_MK_DECL’
   26 | #define CI_MK_DECL(ret, fn, args)  ret (*ci_sys_##fn) args = fn
      |                                                              ^~
/home/lrm/c/other/onload/src/include/onload/declare_syscalls.h.tmpl:120:28: error: ‘__fxstat64’ undeclared here (not in a function); did you mean ‘fstat64’?
  120 | CI_MK_DECL(int           , __fxstat64 , (int, int, struct stat64 *));
      |                            ^~~~~~~~~~
/home/lrm/c/other/onload/src/lib/transport/ip/syscall.c:26:62: note: in definition of macro ‘CI_MK_DECL’
   26 | #define CI_MK_DECL(ret, fn, args)  ret (*ci_sys_##fn) args = fn
      |                                                              ^~
make[3]: *** [/home/lrm/c/other/onload/mk/after.mk:151: ci_ip_syscall.o] Error 1

I wonder if this is related to this commit in GLIBC: bminor/glibc@8ed005d

Someone got great results on intel NICs with openonload?

For example vpp needs only 2 CPU cores to reach the network cards limit.

Any Benchmarks available for openonload how efficient they works

Thanks and

Best regards

hey, I am trying to install driver into RedHat 8.4. and the kernel is 4.18
for some reason, i can't ask for mail support from Xilinx(because the account register failed and I ask but nobody replies me )

steps

openload7.1.2.141(`by onload_install command`)   and sfutils RPM 64 bit.

next
onload_tool reload

then
sfupdate --write
but got no adapter.

[somewheve@localhost ~]$ sfupdate
Solarflare firmware update utility [v8.2.2]
Copyright 2002-2020 Xilinx, Inc.
Loading firmware images from /usr/share/sfutils/sfupdate_images
No supported Solarflare adapters found

so i try find the devei and input the lspci -vvv | grep "Solar" get output below

[somewheve@localhost ~]$ lspci -vvv  | grep "Solar"
65:00.0 Ethernet controller: Solarflare Communications Device 0b13 (rev 01)
        Subsystem: Solarflare Communications Device 0000
65:00.1 Ethernet controller: Solarflare Communications Device 0b13 (rev 01)
        Subsystem: Solarflare Communications Device 0000

then i try find the infomation by modinfo sfc

filename:       /lib/modules/4.18.0-305.12.1.el8_4.x86_64/extra/sfc.ko
version:        4.15.12.1008
license:        GPL
description:    Solarflare network driver
author:         Solarflare Communications and Michael Brown <[email protected]>
rhelversion:    8.4
srcversion:     0D781ADFE2D6C82855F0815
alias:          pci:v00001924d00001B03sv*sd*bc*sc*i*
alias:          pci:v00001924d00000B03sv*sd*bc*sc*i*
alias:          pci:v00001924d00001A03sv*sd*bc*sc*i*
alias:          pci:v00001924d00000A03sv*sd*bc*sc*i*
alias:          pci:v00001924d00001923sv*sd*bc*sc*i*
alias:          pci:v00001924d00000923sv*sd*bc*sc*i*
alias:          pci:v00001924d00001903sv*sd*bc*sc*i*
alias:          pci:v00001924d00000903sv*sd*bc*sc*i*
alias:          pci:v00001924d00000813sv*sd*bc*sc*i*
alias:          pci:v00001924d00000803sv*sd*bc*sc*i*
depends:        mdio,mtd
name:           sfc
vermagic:       4.18.0-305.12.1.el8_4.x86_64 SMP mod_unload modversions
parm:           major:char device major number to use (int)
parm:           efx_allow_nvconfig_writes:Allow access to static config and backup firmware (bool)
parm:           enable_vswitch:Force allocation of a VEB vswitch on supported adapters (bool)
parm:           vfs_vlan_restrict:[SFC9100-family] Restrict VLANs usage on VFs. VF driver needs to use HW VLAN filtering to get VLAN tagged traffic; default=N (bool)
parm:           vf_tx_filter:[SFC9000-family] VF transmit filter mode 0=>never, 1=>auto(default), 2=>always (int)
parm:           vf_max_tx_channels:[SFC9000-family] Limit the number of TX channels VFs can use (uint)
parm:           max_vfs:Specify the number of VFs initialized by the driver (array of int)
parm:           vf_count:Duplicate of the max_vfs parameter (array of int)
parm:           mcdi_logging_default:Enable MCDI logging on newly-probed functions (bool)
parm:           rx_recycle_ring_size:Maximum number of RX buffers to recycle pages for (uint)
parm:           underreport_skb_truesize:Give false skb truesizes. Debug option to restore previous driver behaviour. (bool)
parm:           rx_copybreak:Maximum size of packet that may be copied directly to the network stack
parm:           rx_refill_threshold:RX descriptor ring refill threshold (%) (uint)
parm:           lro_table_size:Size of the LRO hash table.  Must be a power of 2 (uint)
parm:           lro_chain_max:Maximum length of chains in the LRO hash table (uint)
parm:           lro_idle_jiffies:Time (in jiffies) after which an idle connection's LRO state is discarded (uint)
parm:           lro_slow_start_packets:Number of packets that must pass in-order before starting LRO. (uint)
parm:           lro_loss_packets:Number of packets that must pass in-order following loss before restarting LRO. (uint)
parm:           tx_copybreak:Maximum size of packet that may be copied to a new buffer on transmit, minimum is 16 bytes or 0 to disable (uint)
parm:           piobuf_size:[SFC9100-family] Maximum size of packet that may be copied to a PIO buffer on transmit (uint)
parm:           tx_push_max_fill:[SFC9100-family] Only use Tx push when the queue is below this fill level; 0=>never push 1=>push when empty; default always to push (uint)
parm:           tx_coalesce_doorbell:[SFC9100-family] Coalesce notification to NIC of pending TXdata when set this option sets tx_push_max_fill=0::default=N (bool)
parm:           multicast_chaining:[SFC9100-family] Enabled multicast filter chaining in firmware; default=Y (bool)
parm:           monitor_hw_available:[SFC9100-family] Check hardware availability during periodic monitor; default=N (bool)
parm:           tx_non_csum_queue:[SFC9100-family] Allocate dedicated TX queues for traffic not requiring checksum offload; default=N (bool)
parm:           rx_hash_insert:[SFC9000-family] Enable insertion of flow hash in RX buffers (bool)
parm:           rx_desc_cache_size:[SFC4000/SFC9000-family] Set RX descriptor cache size (int)
parm:           tx_desc_cache_size:[SFC4000/SFC9000-family] Set TX descriptor cache size (int)
parm:           efx_disable_port_on_mpe:[SFC4000/SFC9000-family] Disable interface on Memory Parity Error (bool)
parm:           lro:Large receive offload acceleration (bool)
parm:           separate_tx_channels:Use separate channels for TX and RX (bool)
parm:           xdp_alloc_tx_resources:[EXPERIMENTAL] Allocate resources for XDP TX (bool)
parm:           monitor_interval_ms:Bus state test interval in ms (uint)
parm:           rss_cpus:Number of CPUs to use for Receive-Side Scaling, or 'packages', 'cores', 'hyperthreads', 'numa_local_cores' or 'numa_local_hyperthreads' (charp)
parm:           rss_numa_local:Restrict RSS to CPUs on the local NUMA node (bool)
parm:           rss_use_fixed_key:Use a fixed RSS hash key, tested for reliable spreading across channels (bool)
parm:           irq_adapt_enable:Enable adaptive interrupt moderation (bool)
parm:           irq_adapt_low_thresh:Threshold score for reducing IRQ moderation (uint)
parm:           irq_adapt_high_thresh:Threshold score for increasing IRQ moderation (uint)
parm:           irq_adapt_irqs:Number of IRQs per IRQ moderation adaptation (uint)
parm:           debug:Bitmapped debugging message enable value (uint)
parm:           rx_ring:Maximum number of descriptors in a receive ring (uint)
parm:           tx_ring:Maximum number of descriptors in a transmit ring (uint)
parm:           num_vis:Set number of VIs (int)
parm:           phy_power_follows_link:Power down phy when interface is administratively down (bool)
parm:           performance_profile:Tune settings for different performance profiles: 'throughput', 'latency' or (default) 'auto' (charp)
parm:           auto_config_xps:Toggle automatic XPS configuration (default is enabled). (bool)
parm:           irq_set_affinity:Set SMP affinity of IRQs to support RSS (N=>disabled Y=>enabled (default)) (bool)
parm:           napi_weight:NAPI weighting (int)
parm:           rx_irq_mod_usec:Receive interrupt moderation (microseconds) (uint)
parm:           tx_irq_mod_usec:Transmit interrupt moderation (microseconds) (uint)
parm:           interrupt_mode:Interrupt mode (0=>MSIX 1=>MSI 2=>legacy) (uint)

and when i type ip addr got no sfc drive ,

[somewheve@localhost ~]$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f0:2f:74:95:5d:f6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.102/24 brd 192.168.1.255 scope global dynamic noprefixroute enp7s0
       valid_lft 5894sec preferred_lft 5894sec
    inet6 fe80::f22f:74ff:fe95:5df6/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f0:2f:74:95:5d:f5 brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:70:ff:a9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:70:ff:a9 brd ff:ff:ff:ff:ff:ff

can you help with this? I got no idea about this.

I notice that there are two repositories. and Xilinx-CNS/onload remove <zf/zf.h> which is used by tcp_direct. and <zf/zf.h> can be found at majek/openonload.

I am confused about this.

and https://support-nic.xilinx.com/wp/onload?sd=SF-116303-CD-9&pe=361 chapter6.1 told me to use <zf/zf.h>

See https://www.spinics.net/lists/kvm/msg236692.html

I believe the calls to remap_vmalloc_range_partial should be replaced with remap_vmalloc_range.

Hi!

I noticed that the default RSS CPU afffinity rss_cpus=cores - doesn't seem to ignore isolated cores, i.e. those marked with the isolcpus kernel command-line option.
Which is a bit annoying since usually one doesn't want to handle IRQs/softirqs on such core - for example irqbalanced ignores those cores and the kernel also does its best to avoid touching those.

Is it something which would make sense to change/add an option for, or maybe add support to https://github.com/Xilinx-CNS/onload/blob/master/scripts/sfcirqaffinity ?

Cheers,

Hello ,
I have been testing Onload with AF_XDP support on 10 GBe Intel 82599.
my Environment Details:
CPU: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
NIC: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
OS: Ubuntu 20.04.2 LTS
Kernel: 5.4.0-65-generic

root@xcsc:# ethtool -i enp4s0f1
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x000161ae
expansion-rom-version:
bus-info: 0000:04:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

before I enable the EF_AF_XDP_ZEROCOPY to 1 in the latency.opf, my application(onload -p ./conf/latency.opf ./zxTrade) works fine:

Recv tickData(15:29:05): { MkDataCounts: 23840 , TradeDataCounts: 0 , OrderDataCounts: 0 }
Recv tickData(15:30:05): { MkDataCounts: 26224 , TradeDataCounts: 0 , OrderDataCounts: 0 }

after I enable the EF_AF_XDP_ZEROCOPY to 1 in the latency.opf:

onload_set EF_AF_XDP_ZEROCOPY 1

stack can be created:
root@xcsc:# onload_stackdump stacks
#stack-id stack-name pids
1 - 9499

EF_AF_XDP_ZEROCOPY enabled:
root@xcsc:# onload_stackdump lots | grep XDP
EF_XDP_MODE: 0
EF_AF_XDP_ZEROCOPY: 1 (default: 0)
env: EF_AF_XDP_ZEROCOPY=1

but my application(onload -p ./conf/latency.opf ./zxTrade) can not work fine,application show no data packets received;

Recv tickData: { MkDataCounts:0, TradeDataCounts:0, OrderDataCounts:0 }
Recv tickData: { MkDataCounts:0, TradeDataCounts:0, OrderDataCounts:0 }

the thing below drived me crazy:
when I delete "the onload_set EF_AF_XDP_ZEROCOPY 1" in latency.opf, and reload my application, the system crashed, need to reboot.
After the Ubuntu system reboot, dmesg show these message:
root@xcsc:~# dmesg | grep onload
[ 14.229749] [onload] Onload
[ 14.229750] [onload] Copyright 2019-present Xilinx, 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks
[ 14.489442] onload_cp_server[461]: Spawned daemon process 481
[ 14.813890] synth uevent: /devices/virtual/onload/onload: failed to send uevent
[ 14.813893] onload onload: uevent: failed to send synthetic uevent
[ 14.832521] synth uevent: /devices/virtual/onload_epoll/onload_epoll: failed to send uevent
[ 14.832523] onload_epoll onload_epoll: uevent: failed to send synthetic uevent

Is ixgbe driver 5.1.0-K(come with Ubuntu20.04.2 LTS) is not support AF_XDP, ZEROCOPY or other reason ???

thanks.

I looked inside onload_build script and didn't find a "proper" way, so I tried doing the following before invoking onload_build:

export CFLAGS='-march=skylake'
export CXXFLAGS='-march=skylake'
also: replace '-mtune=native' with '-march=skylake' inside onload/mk/platform/gnu_x86_64.mk

Probably the wrong way as they are redundant, but I do see the arguments picked up in some of the compiler command lines (not all, sometimes I still see them being called with mtune=x86_64).

We would need the 5.11 support, but with TCPDirect. Would it be possible to create a patch for the released version for us to be able to compile it on newer kernels?

Python2 was retired on April 2020, but we still need python2-devel to build onload. Can we eliminate the need for python2-devel on onload ?

Hello @maciejj-xilinx ,

Onload is broken after this commit: e9d90b2

I am resetting this head to this commit and using it.

Steps to reprduce:

Install Onload on bare metal with Intel CPU on Centos 8, 8.3 kernel version and ixgbe driver. Start any app with onload -p and you will see the machine unreachable. I tried login via serial LAN of the BMC console to see whats going on and it looks like the XDP modules marks all the queues hidden of the NiC, irrespective of app being onloaded.

Please requesting to fix the master branch :)

Hello Onload Team,

I was able to get Onload to work on Mellanox NiC (SR-IOV mode) in Azure. Please see: #37 for details.

While the app is now being offloaded, we see no traffic being processed by the stack. Please see the following command update:

[root@sriov-onload1 ~]# onload_stackdump 
#stack-id stack-name      pids
6         -               -

[root@sriov-onload1 ~]# onload_stackdump stats | grep polls
k_polls: 3789944
u_polls: 0
ioctl_evq_polls: 0
periodic_polls: 2695
interrupt_polls: 3787248
deferred_polls: 0
timeout_interrupt_polls: 0

As can be seen, u_polls are ZERO and k_polls are incrementing, meaning userstack poll-mode driver is not being exercised.

Also, when we run the xdpdump -D command, I see no AF_XDP program loaded on the eBPF VM:

[root@sriov-onload1 ~]# xdpdump -D
Interface        Prio  Program name      Mode     ID   Tag               Chain actions
--------------------------------------------------------------------------------------
lo                     <No XDP program loaded!>
eth0                   <No XDP program loaded!>
enP28349s1             <No XDP program loaded!>

I have already registered the interface and enabled hugepages:

[root@sriov-onload1 ~]# ulimit -l unlimited
[root@sriov-onload1 ~]# echo 800 > /proc/sys/vm/nr_hugepages
[root@sriov-onload1 ~]# echo enP28349s1 > /sys/module/sfc_resource/afxdp/register

Help is appreciated.

Where can I get the latest version of onload user guide? Googling it seems to give some old results or paywalled docs (still for 2019 edition).

I am working with 2 servers with Solarflare NIC on the first one and Mellanox NIC on the second. I want to read hardware timestamps of received packets on both servers. On server with Mellanox NIC I can achieve this by setting appropriate options to setsockopt call and then read timestamps using cmsg mechanism. My application doesn't depend on libvma (Mellanox's alternative to onload) to enable and read hw timestamps. However, when I run the same application on first server with Solarflare NIC, application shows that hw timestamps are zero -- I assume that means that they are disabled. After reading some examles in this repo I added code from file exchange.c:

#ifdef USE_ONLOAD
  if (onload_is_present()) {
    onload_stack_opt_set_int("EF_RX_TIMESTAMPING", 3);
    std::cout << "Enabled hw ts for onload.\n";
  }
#endif

And with this snippet and when I compile my app with -lonload option I get the correct hw timestamps, when running on server with Solarflare adapter.

I want to avoid dependency on onload lib, as it seems that it is possible -- my app works perfectly fine on Mellanox NIC without any external dependencies on libvma or anything. So my question: is it possible to read hw timestamps from application, which uses Solarflare NIC without compiling it with onload lib? If so, what is the correct way to achieve it? I assumed that I should enable hw timestamping on network interface with sfptpd, so I ran it with these config options:

timestamping_interfaces enp1s0f0np0
timestamping_disable_on_exit off

but it didn't help.

Hi, I am developing a server application for RHEL server with Solarflare NIC and this application runs with openonload. I want to test it with my windows computer on WSL2 to improve developer experience. When I tried to call scripts/onload_install script it gives a clear error. I would like to ask if WSL2 support exists currently or is it in roadmap. Thanks in advance.

onload_install: Building Onload.
mmakebuildtree: No kernel modules at '/lib/modules/4.19.84-microsoft-standard+'
onload_build: FAILED: mmakebuildtree --driver -d x86_64_linux-4.19.84-microsoft-standard+
onload_install: ERROR: Build failed. Not installing

src/lib/transport/ip/signal.c

4 bytes integer instead of 8 bytes is getting shifted

hello, onload team.
i was running demo to learn onload delegated on my server
with single sfc 2522 10G card with interface p1p1(9.9.9.9) and p1p2(9.9.9.11)
they linked with optical fiber and works well on 'efvi ctpio send' and 'efvi event_poll'

server: onload nc -l 9527
client: onload ./trader_onload_ds_efvi -d -s 64 -p 9527 p1p1 9.9.9.11

using onload version: OpenOnload 201811-u1

ERROR: TEST(onload_delegated_send_prepare(s->tcp_sock, s->msg_len * 2, 0, &(s->ods)) == ONLOAD_DELEGATED_SEND_RC_OK) failed
ERROR: at /home/hakase/CLionProjects/efvi_delegated/trader_onload_ds_efvi.c:190
Aborted (core dumped)

i can not find answer on google or somewhere...

when run my app, i see error as below:
'''
This error can occur if no Solarflare network interfaces
are active/UP, or they are running packed stream
firmware, are disabled or lack Onload activation keys.
Please check your configuration. To obtain activation
keys, please contact your sales representative.
'''

LD_PRELOAD="$(mmaketool --toppath)/build/$(mmaketool --userbuild)/lib/transport/unix/libcitransport0.so" sockperf server -i 10.194.138.99 -p 1500 --tcp
oo:sockperf[62239]: netif_tcp_helper_alloc_u: ENODEV.
This error can occur if no Solarflare network interfaces
are active/UP, or they are running packed stream
firmware, are disabled or lack Onload activation keys.
Please check your configuration. To obtain activation
keys, please contact your sales representative.
sockperf: == version #3.7-1.gitb741ab3c60b1 ==
sockperf: [SERVER] listen on:
[ 0] IP = 10.194.139.66 PORT = 1500 # TCP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: [tid 62239] using recvfrom() to block on socket(s)
^Csockperf: Test end (interrupted by user)
sockperf: No messages were received on the server.
sockperf: cleanupAfterLoop() exit

I can not start onload in centos 8.3 and debian10(kernel 5.8), why onload need to find mtdchar?
errors follows:

[root@localhost onload]# "$(mmaketool --toppath)/build/$(mmaketool --driverbuild)/driver/linux/load.sh" onload
unload.sh: /sbin/rmmod onload
unload.sh: /sbin/rmmod sfc_char
unload.sh: /sbin/rmmod sfc_resource
unload.sh: /sbin/rmmod sfc
unload.sh: /sbin/rmmod virtual_bus
unload.sh: /sbin/rmmod sfc_driverlink
NET_OPT is
CHAR_OPT is
modprobe: FATAL: Module mtdchar not found in directory /lib/modules/4.18.0-240.10.1.el8_3.x86_64
ERROR: Did not find sfc_control in /proc/devices
sfc is a DEBUG driver
RESOURCE_OPT is
CHAR_OPT is
ONLOAD_OPT is

Hello!

System: CentOS 8.3.2011
To reproduce:
1 Install the latest CentOS
2 Clone this repo
3 Run scripts/onload_install

Please see this log:

/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c: In function ‘efx_tc_indr_setup_cb’:
/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c:4776:15: error: implicit declaration of function ‘flow_indr_block_cb_alloc’; did you mean ‘flow_indr_block_call’? [-Werror=implicit-function-declaration]
block_cb = flow_indr_block_cb_alloc(efx_tc_block_cb, binding,
^~~~~~~~~~~~~~~~~~~~~~~~
flow_indr_block_call
/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c:4776:13: warning: assignment to ‘struct flow_block_cb ’ from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
block_cb = flow_indr_block_cb_alloc(efx_tc_block_cb, binding,
^
/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c:4804:4: error: implicit declaration of function ‘flow_indr_block_cb_remove’; did you mean ‘flow_indr_block_cb_register’? [-Werror=implicit-function-declaration]
flow_indr_block_cb_remove(block_cb, tcb);
^~~~~~~~~~~~~~~~~~~~~~~~~
flow_indr_block_cb_register
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/farch.o
/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c: In function ‘efx_init_tc’:
/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c:4951:30: error: passing argument 1 of ‘flow_indr_dev_register’ from incompatible pointer type [-Werror=incompatible-pointer-types]
rc = flow_indr_dev_register(efx_tc_indr_setup_cb, efx);
^~~~~~~~~~~~~~~~~~~~
In file included from ./include/net/sch_generic.h:21,
from ./include/linux/filter.h:25,
from ./include/net/sock.h:64,
from ./include/net/inet_sock.h:26,
from ./include/net/ip.h:31,
from /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/kernel_compat.h:60,
from /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c:12:
./include/net/flow_offload.h:539:55: note: expected ‘int ()(struct net_device *, void *, enum tc_setup_type, void )’ but argument is of type ‘int ()(struct net_device *, void *, enum tc_setup_type, void *, void , void ()(struct flow_block_cb *))’
int flow_indr_dev_register(flow_indr_block_bind_cb_t *cb, void cb_priv);
~~~~~~~~~~~~~~~~~~~~~~~~~~~^~
/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c: In function ‘efx_fini_tc’:
/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c:4972:28: error: passing argument 1 of ‘flow_indr_dev_unregister’ from incompatible pointer type [-Werror=incompatible-pointer-types]
flow_indr_dev_unregister(efx_tc_indr_setup_cb, efx, efx_tc_block_unbind);
^~~~~~~~~~~~~~~~~~~~
In file included from ./include/net/sch_generic.h:21,
from ./include/linux/filter.h:25,
from ./include/net/sock.h:64,
from ./include/net/inet_sock.h:26,
from ./include/net/ip.h:31,
from /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/kernel_compat.h:60,
from /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c:12:
./include/net/flow_offload.h:540:58: note: expected ‘int ()(struct net_device *, void *, enum tc_setup_type, void )’ but argument is of type ‘int ()(struct net_device *, void *, enum tc_setup_type, void *, void , void ()(struct flow_block_cb *))’
void flow_indr_dev_unregister(flow_indr_block_bind_cb_t *cb, void cb_priv,
~~~~~~~~~~~~~~~~~~~~~~~~~~~^~
/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c:4972:55: error: passing argument 3 of ‘flow_indr_dev_unregister’ from incompatible pointer type [-Werror=incompatible-pointer-types]
flow_indr_dev_unregister(efx_tc_indr_setup_cb, efx, efx_tc_block_unbind);
^~~~~~~~~~~~~~~~~~~
In file included from ./include/net/sch_generic.h:21,
from ./include/linux/filter.h:25,
from ./include/net/sock.h:64,
from ./include/net/inet_sock.h:26,
from ./include/net/ip.h:31,
from /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/kernel_compat.h:60,
from /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c:12:
./include/net/flow_offload.h:541:27: note: expected ‘int ()(enum tc_setup_type, void *, void )’ but argument is of type ‘void ()(void *)’
flow_setup_cb_t setup_cb);
~~~~~~~~~~~~~~~~~^~~~~~~~
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/siena.o
/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c: In function ‘efx_tc_netdev_event’:
/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c:5017:10: error: passing argument 3 of ‘__flow_indr_block_cb_register’ from incompatible pointer type [-Werror=incompatible-pointer-types]
efx_tc_indr_setup_cb, efx);
^~~~~~~~~~~~~~~~~~~~
In file included from ./include/net/sch_generic.h:21,
from ./include/linux/filter.h:25,
from ./include/net/sock.h:64,
from ./include/net/inet_sock.h:26,
from ./include/net/ip.h:31,
from /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/kernel_compat.h:60,
from /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c:12:
./include/net/flow_offload.h:552:34: note: expected ‘int ()(struct net_device *, void *, enum tc_setup_type, void )’ but argument is of type ‘int ()(struct net_device *, void *, enum tc_setup_type, void *, void , void ()(struct flow_block_cb *))’
flow_indr_block_bind_cb_t cb,
~~~~~~~~~~~~~~~~~~~~~~~~~~~^~
/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c:5027:44: error: passing argument 2 of ‘__flow_indr_block_cb_unregister’ from incompatible pointer type [-Werror=incompatible-pointer-types]
__flow_indr_block_cb_unregister(net_dev, efx_tc_indr_setup_cb,
^~~~~~~~~~~~~~~~~~~~
In file included from ./include/net/sch_generic.h:21,
from ./include/linux/filter.h:25,
from ./include/net/sock.h:64,
from ./include/net/inet_sock.h:26,
from ./include/net/ip.h:31,
from /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/kernel_compat.h:60,
from /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.c:12:
./include/net/flow_offload.h:556:37: note: expected ‘int ()(struct net_device *, void *, enum tc_setup_type, void )’ but argument is of type ‘int ()(struct net_device *, void *, enum tc_setup_type, void *, void , void ()(struct flow_block_cb *))’
flow_indr_block_bind_cb_t *cb,
~~~~~~~~~~~~~~~~~~~~~~~~~~~^~
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/sriov.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/mcdi_mon.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/ptp.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/mtd.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/ef100_vdpa.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/ef100_vdpa_ops.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/mcdi_vdpa.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/ioctl.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/ioctl_common.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/kernel_compat.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/linux_mdio.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/sfctool.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/aoe.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/debugfs.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/dump.o
CC [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/driverlink.o
LD [M] /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/sfc_driverlink.o
cc1: some warnings being treated as errors
make[8]: *** [scripts/Makefile.build:315: /root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc/tc.o] Error 1
make[7]: *** [Makefile:1544: module/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc] Error 2
make[7]: Leaving directory '/usr/src/kernels/4.18.0-240.1.1.el8_3.x86_64'
make[6]: *** [/root/onload/onload/src/driver/linux_net/drivers/net/ethernet/sfc/Makefile:260: modules] Error 2
make[6]: Leaving directory '/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet/sfc'
make[5]: *** [/root/onload/onload/src/driver/linux_net/drivers/net/ethernet/mmake.mk:8: all] Error 2
make[5]: Leaving directory '/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net/ethernet'
make[4]: *** [/root/onload/onload/src/driver/linux_net/drivers/net/mmake.mk:8: all] Error 2
make[4]: Leaving directory '/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers/net'
make[3]: *** [/root/onload/onload/src/driver/linux_net/drivers/mmake.mk:8: all] Error 2
make[3]: Leaving directory '/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net/drivers'
make[2]: *** [/root/onload/onload/src/driver/linux_net/mmake.mk:8: all] Error 2
make[2]: Leaving directory '/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver/linux_net'
make[1]: *** [/root/onload/onload/src/driver/mmake.mk:19: all] Error 2
make[1]: Leaving directory '/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64/driver'
make: *** [/root/onload/onload/src/mmake.mk:25: all] Error 2
make: Leaving directory '/root/onload/onload/build/x86_64_linux-4.18.0-240.1.1.el8_3.x86_64'
onload_build: ERROR: Failed to build driver components.
onload_install: ERROR: Build failed. Not installing.

I used "efsend_pio" to send 32 udp packets. It took about 70us. Now I don’t know where the problem occurred. Can you give me a test data?

OS: centos7.6
CPU:i9-9980XE
NIC:X2522

1、Hi, onload team, have you test AF_XDP on Redhat Enterprise Linux 8.2 ? Only 8.3 supports AF_XDP ?
2、Is AF_XDP function stable now ?
3、Can onload AF_XDP run on Centos running on VMWARE ?

Any reply will be greatly appreciated. Thank you very much.

The AF_XDP support is currently under development and is not yet at final release quality.
The following operating system distributions are known to provide an adequate level of AF_XDP support for Onload:
Ubuntu LTS 20.04
Ubuntu 20.10
Debian 10 with Linux kernel 5.9
Redhat Enterprise Linux 8.3

Default shebang is Bourne shell in run_unit_tests.sh. Under Ubuntu 18.04.5 LTS:
root@aptd:~/onload# ./scripts/run_unit_tests.sh
./scripts/run_unit_tests.sh: 12: ./scripts/run_unit_tests.sh: Syntax error: "(" unexpected

Changing to "#! /bin/bash -e" allows script to execute:

root@aptd:~/onload# ./scripts/run_unit_tests.sh
Building tests
make: Entering directory '/home/paolini/onload/build/gnu_x86_64'
make[1]: Entering directory '/home/paolini/onload/build/gnu_x86_64/include'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/include'
make[1]: Entering directory '/home/paolini/onload/build/gnu_x86_64/lib'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/lib/citools'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/lib/citools'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/lib/ciapp'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/lib/ciapp'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/lib/onload_ext'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/lib/onload_ext'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/lib/ciul'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/lib/ciul'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/lib/kcompat'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/lib/kcompat'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/lib/cplane'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/lib/cplane'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/lib/transport'
make[3]: Entering directory '/home/paolini/onload/build/gnu_x86_64/lib/transport/ip'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/lib/transport/ip'
make[3]: Entering directory '/home/paolini/onload/build/gnu_x86_64/lib/transport/common'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/lib/transport/common'
make[3]: Entering directory '/home/paolini/onload/build/gnu_x86_64/lib/transport/unix'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/lib/transport/unix'
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/lib/transport'
make[1]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/lib'
make[1]: Entering directory '/home/paolini/onload/build/gnu_x86_64/driver'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/driver/linux_net'
make[3]: Entering directory '/home/paolini/onload/build/gnu_x86_64/driver/linux_net/drivers'
make[4]: Entering directory '/home/paolini/onload/build/gnu_x86_64/driver/linux_net/drivers/net'
make[5]: Entering directory '/home/paolini/onload/build/gnu_x86_64/driver/linux_net/drivers/net/ethernet'
make[6]: Entering directory '/home/paolini/onload/build/gnu_x86_64/driver/linux_net/drivers/net/ethernet/sfc'
make[6]: Nothing to be done for 'all'.
make[6]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/driver/linux_net/drivers/net/ethernet/sfc'
make[5]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/driver/linux_net/drivers/net/ethernet'
make[4]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/driver/linux_net/drivers/net'
make[3]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/driver/linux_net/drivers'
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/driver/linux_net'
make[1]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/driver'
make[1]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tools'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tools/cplane'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tools/cplane'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tools/onload_helper'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tools/onload_helper'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tools/ip'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tools/ip'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tools/solar_clusterd'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tools/solar_clusterd'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tools/onload_remote_monitor'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tools/onload_remote_monitor'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tools/onload_mibdump'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tools/onload_mibdump'
make[1]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tools'
make[1]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/ef_vi'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/ef_vi'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/onload'
make[3]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/wire_order'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/wire_order'
make[3]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/tproxy_preload'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/tproxy_preload'
make[3]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/hwtimestamping'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/hwtimestamping'
make[3]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/sync_preload'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/sync_preload'
make[3]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/oof'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/oof'
make[3]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/onload_remote_monitor'
make[4]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/onload_remote_monitor/internal_tests'
make[4]: Nothing to be done for 'all'.
make[4]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/onload_remote_monitor/internal_tests'
make[3]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/onload_remote_monitor'
make[3]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/cplane_unit'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/cplane_unit'
make[3]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/cplane_sysunit'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/cplane_sysunit'
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/onload'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/orm_test_client'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/orm_test_client'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/rtt'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/rtt'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/syscalls'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/syscalls'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/tap'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/tap'
make[2]: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/trade_sim'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/trade_sim'
make[1]: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests'
make: Leaving directory '/home/paolini/onload/build/gnu_x86_64'
Running tests
make: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/oof'
/usr/bin/timeout 240 prove --exec ' '
"./oof_test sanity" "./oof_test multicast_sanity" "./oof_test namespace_sanity" "./oof_test namespace_macvlan_move"
./oof_test sanity .................. ok
./oof_test multicast_sanity ........ ok
./oof_test namespace_sanity ........ ok
./oof_test namespace_macvlan_move .. ok
All tests successful.
Files=4, Tests=158, 0 wallclock secs ( 0.04 usr + 0.02 sys = 0.06 CPU)
Result: PASS
make: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/oof'
make: Entering directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/cplane_unit'
/usr/bin/timeout 600 prove -j2 --merge --exec ''
./test_route ./test_route_expire ./test_arp_expire ./test_route_stress ./test_teambond ./test_namespace ./test_service_dnat
./test_route ......... ok
./test_route_expire .. ok
./test_arp_expire .... ok
./test_route_stress .. ok
./test_namespace ..... ok
./test_service_dnat .. ok
./test_teambond ...... ok
All tests successful.
Files=7, Tests=167, 50 wallclock secs (16.74 usr 0.70 sys + 76.20 cusr 0.47 csys = 94.11 CPU)
Result: PASS
make: Leaving directory '/home/paolini/onload/build/gnu_x86_64/tests/onload/cplane_unit'
All tests PASSED

I get this error after the command:

make -C "$(mmaketool --toppath)/build/$(mmaketool --userbuild)"

nice cc -I. -I./../../include -I/build/onload/src/include -I/build/onload/src/lib/citools    -fPIC  -m64  -Werror -Wall -Wundef -Wpointer-arith -Wstrict-prototypes -Wnested-externs -Wno-stringop-truncation -Wno-format-truncation -Wimplicit-fallthrough=2 -Wno-array-bounds -Wno-stringop-overflow -Wno-deprecated-declarations -DTRANSPORT_CONFIG_OPT_HDR='<ci/internal/transport_config_opt_extra.h>' -mpclmul -msse4.1 -O2 -g $cflags $cppflags -c /build/onload/src/lib/citools/ippacket.c -o ci_tools_ippacket.o
nice cc -I. -I./../../include -I/build/onload/src/include -I/build/onload/src/lib/citools    -fPIC  -m64  -Werror -Wall -Wundef -Wpointer-arith -Wstrict-prototypes -Wnested-externs -Wno-stringop-truncation -Wno-format-truncation -Wimplicit-fallthrough=2 -Wno-array-bounds -Wno-stringop-overflow -Wno-deprecated-declarations -DTRANSPORT_CONFIG_OPT_HDR='<ci/internal/transport_config_opt_extra.h>' -mpclmul -msse4.1 -O2 -g $cflags $cppflags -c /build/onload/src/lib/citools/namespace.c -o ci_tools_namespace.o
nice cc -I. -I./../../include -I/build/onload/src/include -I/build/onload/src/lib/citools    -fPIC  -m64  -Werror -Wall -Wundef -Wpointer-arith -Wstrict-prototypes -Wnested-externs -Wno-stringop-truncation -Wno-format-truncation -Wimplicit-fallthrough=2 -Wno-array-bounds -Wno-stringop-overflow -Wno-deprecated-declarations -DTRANSPORT_CONFIG_OPT_HDR='<ci/internal/transport_config_opt_extra.h>' -mpclmul -msse4.1 -O2 -g $cflags $cppflags -c /build/onload/src/lib/citools/get_cpu_khz.c -o ci_tools_get_cpu_khz.o
nice cc -I. -I./../../include -I/build/onload/src/include -I/build/onload/src/lib/citools    -fPIC  -m64  -Werror -Wall -Wundef -Wpointer-arith -Wstrict-prototypes -Wnested-externs -Wno-stringop-truncation -Wno-format-truncation -Wimplicit-fallthrough=2 -Wno-array-bounds -Wno-stringop-overflow -Wno-deprecated-declarations -DTRANSPORT_CONFIG_OPT_HDR='<ci/internal/transport_config_opt_extra.h>' -mpclmul -msse4.1 -O2 -g $cflags $cppflags -c /build/onload/src/lib/citools/log_fn.c -o ci_tools_log_fn.o
/build/onload/src/lib/citools/log_fn.c: In function ‘ci_log_stderr’:
/build/onload/src/lib/citools/log_fn.c:30:3: error: ignoring return value of ‘writev’ declared with attribute ‘warn_unused_result’ [-Werror=unused-result]
   30 |   writev(STDERR_FILENO, v, 2);
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~
/build/onload/src/lib/citools/log_fn.c: In function ‘ci_log_stdout’:
/build/onload/src/lib/citools/log_fn.c:43:3: error: ignoring return value of ‘writev’ declared with attribute ‘warn_unused_result’ [-Werror=unused-result]
   43 |   writev(STDOUT_FILENO, v, 2);
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
make[2]: *** [/build/onload/mk/after.mk:151: ci_tools_log_fn.o] Error 1
make[2]: Leaving directory '/build/onload/build/gnu_x86_64/lib/citools'
make[1]: *** [/build/onload/src/lib/mmake.mk:19: all] Error 2
make[1]: Leaving directory '/build/onload/build/gnu_x86_64/lib'
make: *** [/build/onload/src/mmake.mk:25: all] Error 2
make: Leaving directory '/build/onload/build/gnu_x86_64

Hello everyone,
I use a DELL Latitude 5520 computer (Intel i5), with operating system Ubuntu 18.04.5 LTS, Linux kernel 5.10.0-0.bpo.3-rt-amd64 and architecture x86-64. I tried to install OpenOnLoad, following the instructions in the paragraph "Building directly from repository" (https://github.com/Xilinx-CNS/onload/blob/master/DEVELOPING.md).
In particular there seems to be some problems related to the drivers: as you can see from the figure, the kernel build is not found.

Instead, running the same instructions on a generic kernel, the whole installation process went smoothly.

Has anyone had the same problem as me? Can you help me?

Thank you,
Best regards,
Emma

The latency of above two funcitons sometimes reaches to 2000ns~10000ns, and the frequency is uncertain.
Is that normal or it should not appear?

Thanks for help.