Performance about nebula HOT 8 CLOSED

It would be nice if there is some kind of comparitive tests done the devs or people with good knowledge of this stuff against Wireguard and Tinc.

thanks

from nebula.

I get the same result in digitalocean when using the cheapest instance.

The main issue is related to cpu use for syscalls/encryption on smaller instances and DO not supporting jumbo frames.

In the iperf test, it is twice as fast if you use a host with at least 2 cpus, since the encryption tasks can be split among them.

This will also likely improve if/when we do dpdk or XDP, per #5

from nebula.

It would be nice if there is some kind of comparitive tests done the devs or people with good knowledge of this stuff against Wireguard and Tinc.

thanks

I was also looking for something like this. I'll be doing a bunch of testing in the coming weeks and will try to post a writeup of the results. I don't have blog or anything so I'll likely just make a new issue to hold all the information and then mark it closed.

from nebula.

That is very not expected. Can you share your config for each host and the test you are running?

The main setting to be concerned with is tun.mtu

from nebula.

One more note: a slightly more representative test is to do something like scp between nodes as opposed to iperf. I just SCP'd the raspbian iso between them and it maxed out at 500mbit for me, which is just a bit faster than nebula with an unoptimized MTU setting of 1300.

from nebula.

I agree with @rawdigits, if we are talking about a single core box. I launched two of the cheapest instances in DO and did some quick config tuning. Got to a bit above 400Mb/s with iperf3. Looked like we may be able to squeeze some more out of them as well.

The main config options I played with to get a raw bandwidth test looking better:

• tun.mtu can be higher, 1440 in this test
• tun.tx_queue was raised to 5000 although I didn't see many tx queue drops in that interface before
• listen.read_buffer and listen.write_buffer were raised to 10485760
• listen.batch was raised to 128 although that didn't make much of a difference for me

from nebula.

Sorry for leaving this and then not answering.

I tested using the default settings from your config.yaml example. The DO servers were the smallest ones.

I tried it again on two i7 Hetzner servers. The result was much better. 895 Mbits/sec with nebula vs 944 Mbit/sec without nebula

from nebula.

Nice! That's more in line with what we would expect to see. One thing to note regarding different CPU types:

AES-NI acceleratrion makes intel cpus very fast when using AES for your nebula network. Most arm/mobile cpus do not include AES instructions, so we offer chachapoly as an alternative. If you use a number of non x86 cpus and see low performance, check out the config option for chachapoly.

NOTE: the entire nebula network has to be either AES or chachapoly. You cannot mix the two, by design.

from nebula.