Receiving more packets then sent + plus kring error, when pkt-gen use multiple threads about netmap HOT 20 CLOSED

Sorry, I forgat to say,
I apply the following ethtool commands also to the NIC:

ethtool -A eth3 autoneg off rx off tx off
ethtool -K eth3 rx off
ethtool -K eth3 gro off

thanks.

Hi berraazra,

I need some additional details on how you compile your ixgbe driver. Netmap 
support is only included if you pass CONFIG_NETMAP=m to make. Moreover, since 
you are building outside the netmap directory, you should also pass some -I... 
options via EXTRA_CFLAGS, otherwise you should get errors for missing files. If 
you have done nothing of the sort, you have likely compiled the ixgbe driver 
without native netmap support. You can check this easily: if you are able to 
load the ixgbe driver before netmap_lin.ko, then netmap support is not there, 
since otherwise you would get errors for missing symbols.

If netmap support is not compiled in your ixgbe driver, you are using the 
generic netmap adapter. The errors you get need to be addressed either way, but 
first we need to know the exact setup.

Hi,

I wish to use newest ixgbe driver. All steps of compiling the driver and netmap 
is avail in the attachment. So, first I apply the matching patch to the driver 
source code, make it. On the other hand, I make netmap with "NODRIVER=1" 
option. So I think, "#ifdef DEV_NETMAP" parts are not included in ixgbe.ko, 
isn't it ? How should I compile the newest driver ? Copying the source codes to 
the "../drivers/net/ethernet/intel/ixgbe", and compiling netmap with drivers 
didn't work.

Thanks.

Hi, 
When I try to compile netmap module with original driver sources in the kernel 
source, I couldn't achieve. 
Path for kernel configuration-headers is 
"/usr/src/kernels/3.8.13-16-2-1.el6uk.x86_64"
Path for kernel source is 
"/usr/src/debug/kernel-3.8.13/linux-3.8.13-16-2-1.el6uk"

Results are in the attachment.

Thanks.

Hi,
Can you please use the 'next' branch of netmap? It contains a configure script 
that should make things a little easier. 

In the LINUX subdirectory, run:

./configure --kernel-sources=/path/to/full/kernel/sources

The problem you mention in #4 is caused by the patches of igb and e1000e. If 
you are not interested in these drivers, you can add the following options to 
configure:

--no-driver=igb --no-driver=e1000e

The compilation of the out-of-tree version of ixgbe is not directly supported, 
but you can still do it, if you are sufficiently determined :)
Once ./configure has finished, you should obtain a drivers.mak file in the 
LINUX directory. Open it and look for the get-ixgbe target, then change the cp 
operation to something like

cp -Rp /path/to/your/preferred/sources ixgbe

(be sure to also replace the '.' with 'ixgbe', otherwise you might get the 
wrong directory name from the copy)

Now you can try to run make.

Very sorry, may be so dummy but

What does "Can you please use the 'next' branch of netmap?" mean ? 
I performed "git clone https://code.google.com/p/netmap/" but there is no new 
configure script. Where is the 'next' branch of netmap.

Thanks a lot.

That is a git branch, you already have it if you have cloned the repository, 
you just have to check it out by issuing

git checkout -b next origin/next

in your netmap directory.

Ok, I got it, I will try your solution.

To inform;
In #2, when you say "The errors you get need to be addressed either way...", I 
understand that duplicate packets are still problem for generic netmap adapter 
without driver changes, isn't it ? 
Because, I modify my packet sender program, in order to hold a packet order 
from 1 to 20M in the IP checksum and frag id fields of each packet; also 
changed the pkt-gen to count total number of capture for each unique packet id. 
Thus, I proved the multiple capture of a unique packet by multiple threads up 
to the number specified by "-p " option. 

Very thanks.

Yes, we need to investigate the problem you are seeing. It may be a bug in the 
generic adapter, or somewhere else. Therefore, it is interesting to know if you 
still can see the problem when you use the modified driver.

Hi,

I made a fresh start by downloading linux-3.8.13 from kernel.org and recompile.
Now, my server is booted with 3.8.13 kernel, and sources,headers etc. are in 
the same place.
I compiled netmap and ixgbe ( original one in the kernel source ) but can't 
load netmap module. "insmod netmap.ko" results in "-1 unknown symbol in 
module". There is a problem with 'split_page' whatever.
Details are in the attachment.

Thanks.

In addition, after compilation, I expect "netmap_lin.ko" but there is no 
"netmap_lin.ko" module, only "netmap.ko" was produced.

I have pushed some updates that should, hopefully, solve the split_page() 
issue. 

In the latest versions of netmap the module is called 'netmap.ko', as you have 
noticed already.

Hi Giuseppe,

Last update solved the compilation problem.

Without any error, I compiled ixgbe and also netmap.ko.

As you expressed before, ixgbe can't be loaded before netmap had been loaded.

There is no kring errors in messages when pkt-gen is run.

But new problem is DROPS ! :)

I'm sending 20M packets in 49 seconds, average packet length is ~780 byte; 
roughly at 400K packets/sec speed. Then ~%75 packets are dropped.

IP cheksum of the packets are valid.
I tried "-b 65536" for the pkt-gen, but there is no change.
I changed ring size to 2**17, but it doesn't have any affect.
I try single thread also.
There are two attachements, one for system log when I loaded netmap,ixgbe and 
run pkt-gen. The other one is the pkt-gen output. ifconfing reports severe 
drops.

I didn't care SFP warning, because without netmap, my simple packet handler 
kernel module can capture without drops.

Thanks for all.

Hi,

I repeated the test after setting 
"/sys/devices/system/cpu/*/cpufreq/scaling_governor" to the "performance".

I realized that my new ixgbe driver with netmap support drops packets directly 
qhile no pkt-gen capture is running. 
When "ethtool -S" reported "rx_missed_errors" for the drops.

SystemStatus file includes /var/log/messages, lsmod, top, and severel ethtool 
results before I begin the test.
Results include ifconfig and ethtool -S results.

~7M packets over 20M packets are dropped.

20M packets are sent in 145 seconds,resulting  ~138K packets/sec speed and ~825 
Mbit/sec bandwidth. Average packet size is ~780 byte.

Where is the problem ?

I couldn't hear any voice since October 20, I wonder if I closed my issue 
mistakenly ? If so, #14 and #15 are still waiting to be replied. Can you inform 
at least "we are looking for the problem..." or "we closed the issue due to..."

thanks.

My problem is solved with setting the coalesce variable rx-usesc to 1000 when 
multiple RSS queues are used. 

In addition, I have changed ixgbe driver code a little to disable multiple RSS 
queues to use single queue. I repeated the tests for multiple queue disabled. 
In this case, I realized that there is no need to set rx-usecs, netmap runs 
correctly without drop.

So, the problem ( multiple captured packets + kring_error) is valid for generic 
netmap adapter.

Hi berraazra,

can you please try the attached patch to see if it has the same effect as 
'rx-usecs 1000'?

Cheers,
Giuseppe

I tried the patch but it did no affect.

For single-queue, there is no packet loss when rx-usecs = 1

For multi-queue, there is no packet loss when rx-usecs ~ > 600.

I attached patch results and multi-queue output when 40M packets are sent and 
rx-usecs=300.

I have another question:

When I try to run two independant pkt-gen application to receive packets, 
neither one can capture the packets; both capture a few hundred packets. This 
is a expected result ?