Poor performance sending to WebRTC about usrsctp HOT 16 CLOSED

I am no expert on the user stack. Not sure about debugging I am sure
Michael had some methods.

This does sound sort of like a nagle issue. Have you tried turning that off?

(SCTP_NODELAY socket option to on)

R

Sent from my iPhone

On Nov 23, 2015, at 12:41 PM, Nick Chadwick [email protected]
wrote:

Hi, sorry if this isn't the right place to ask questions like this, but I'm
trying to use usrsctp in a native app to connect to Chrome, and I'm getting
really poor performance.

I'm using sctp wrapped in dtls sent over an libnice connection, using this
small library: https://github.com/xhs/librtcdc

While I can get my library to connect and send messages, it seems to be
only sending a single message per second, and any further messages get
queued in the send queue (which doesn't seem to be getting serviced).

My main question is how I can go about debugging issues like this, and
where to start looking when I experience issues using the usrctp library.
I've tried hacking in some debug printfs, which has shown that my messages
are being queued, but I couldn't figure out how to enable detailed logging
from the library.

Any help much appreciated :)

—
Reply to this email directly or view it on GitHub
#36.

from usrsctp.

I've tried with and without nagle. I'm wondering if usrsctp spawns an internal thread to handle sending packets from its queue, or if this behaviour is triggered whenever a new packet is sent/received.

From the timings I'm seeing, it almost seems like the stack will only send a packet when it receives data from the network, and is only sending data once every few seconds. I'm wondering if there's a timeout that might be causing the data to be sent only periodically.

I'm also setting SCTP_EOR on every message I send; not sure if that's required (I thought it was the default behaviour).

I'll try to put a minimum example together tonight to try and illustrate the behaviour.

from usrsctp.

You can have a look at the examples in the programs/ directory. Sending of messages is triggered by

• calling send
• packets are received
• a timer runs off

Having an example to look at might help.

from usrsctp.

@tuexen Thanks for the advice. I think my main question here is, knowing that these are the events that trigger sends, what causes my messages to get queued?

The behavior I'm seeing is that I'm sending 10 packets of around 1.5KB on the connection, then seeing them all slowly "dribble out" over the next 20(!) seconds

from usrsctp.

No idea. You can enable debug output by using --enable-debug when running the configure script and then setting

#ifdef SCTP_DEBUG
    usrsctp_sysctl_set_sctp_debug_on(SCTP_DEBUG_ALL);
#endif

You must also provide a debug_printf() routine:

void
debug_printf(const char *format, ...)
{
    va_list ap;

    va_start(ap, format);
    vprintf(format, ap);
    va_end(ap);
}

and use it in

    usrsctp_init(0, conn_output, debug_printf);

See programs/ekr_loop.c for example.

We have not yet played with librtcdc, but this is on our table...

from usrsctp.

Thanks a lot, I'll let you know how I go tonight.

If you're looking at librtcdc, make sure to take a look at my branch - I've added a few changes that add support for connecting to chrome (firefox is still a work in progress). If there's anything I can do to help, please let me know.

from usrsctp.

Quick question - should usrsctp_conninput be fed message-by-message, or can it handle being fed arbitrary chunks of data (potentially containing multiple messages).

Looking at my logs, the first chunk of data received (an INIT) is handled fine, but the next two chunks are concatenated and sent to usrsctp_conninput. But the response from usrsctp_conninput is that there is a bad checksum.

from usrsctp.

usrsctp_conninput() should be fed message-by-message, but not chunk-by-chunk. Whenever you receive an SCTP packet (a common header followed by a sequence of chunks), you have to call usrsctp_conninput() with that packet.

from usrsctp.

Okay I think that makes sense. Here's where I think librtcdc is getting things confused:

https://github.com/xhs/librtcdc/blob/master/src/sctp.c#L225

The library is buffering data its received from the network (via ice->dtls) and is buffering using an OpenSSL BIO. But its reading as much data as it can from the BIO and sending it through using usrsctp_conninput

If I understand correctly, it should be buffering these messages but sending them through message-by-message. I'm also curious as to what the max size usrsctp_sendv expects to receive is - should I be feeding it mtu-by-mtu or can it handle large (~5KB) messages

My final question is whats recommended for wrapping and sending outputs. Should librtcdc be taking each output from SCTP and wrapping and sending it in a separate DTLS/ICE chunk?

Thanks for all your help, hopefully I can put together a simple demo at the end of this for future reference.

EDIT: Having wrapped the call to usrsctp_conninput with message-based queues, I can now reliably crash the browser. I think I'm being hit by: https://code.google.com/p/chromium/issues/detail?id=537189 as librtcdc is putting multiple DTLS messages into a single UDP packet. That's tomorrow's task.

from usrsctp.

Hmm, but isn't BIO_read() giving you a user message from a DTLS record. At least this is my understanding. The fact that you might have multiple DTLS records in a single UDP packet should not affect that. This is explicitly allowed by RFC 6347. A similar issues was fixed recently for Firefox...

from usrsctp.

I believe the behavior of BIO_read is to read as much data from the underlying socket as possible, or in the case of a memory BIO to read as much data as possible. In most use cases, BIO_read is called immediately after SSL_Write. In the librtcdc case however, there is a thread calling SSL_Write, and a thread calling BIO_read, and no guarantees about how many times SSL_write is called before BIO_read gets to read the data.

Either way, it sure seems like chrome has a bug.

from usrsctp.

That is a problem. DTLS is message based and you normally use BIOs which preserve the message boundaries. You can't use BIOs which don't preserve them, because you can't reconstruct the message boundaries. So I guess this is an issue with librtcdc?

from usrsctp.

Looks like an issue with librtcdc. Taking a look at Janus and friends, it seems like all other uses of OpenSSL DTLS immediately read all data from the BIO after each write. This doesn't preserve boundaries, it just assumes that every call to SSL_write will result in exactly one message. That also means that when using ICE, each time the ice handler receives a message it needs a separate call to SSL_write, followed by processing all the data resulting from the BIO_read.

For future reference - an OpenSSL memory BIO DOES NOT do any tracking of messages boundaries. I checked the code here: https://github.com/openssl/openssl/blob/master/crypto/bio/bss_mem.c

from usrsctp.

Regarding the bios: Wouldn't you use BIO_new_dgram() for DTLS?

from usrsctp.

Not if you're not in control of the socket, as is the case with libnice.

Hey look, one of the main Janus developers had exactly this issue: http://openssl.6102.n7.nabble.com/openssl-users-DTLS-fragmentation-and-mem-BIO-td58462.html

I'm closing this issue for now, as clearly my issues isn't with usrsctp. I may come back and add a few more comments as I figure things out, so I'd appreciate you guys not deleting this off-topic thread until I've had a chance to digest everything I've learned.

from usrsctp.

Janus's friends are saying, one can use the boringssl lib.

from usrsctp.