Bind errors while running Kitura and Kitura-NIO tests about kitura-nio HOT 15 CLOSED

thanks @pushkarnk you uncovered an amazing embarrassing NIO bug: We currently only store one socket option value per type. Ie. only one of ChannelOptions.socket. So setting SO_REUSEADDR will unset all other ChannelOptions.socket(*) that were set before... Working on a fix, have a failing test already.

from kitura-nio.

I'm working on apple/swift-nio#597 to fix it and will release in 1.9.3

from kitura-nio.

@pushkarnk @saiHemak setting SO_REUSEPORT first resolves the issue.

import NIO 

let bootstrap1 = ServerBootstrap(group: MultiThreadedEventLoopGroup(numberOfThreads: System.coreCount))
     .serverChannelOption(ChannelOptions.socket(SocketOptionLevel(SOL_SOCKET), SO_REUSEPORT), value: 0)
    .serverChannelOption(ChannelOptions.socket(SocketOptionLevel(SOL_SOCKET), SO_REUSEADDR), value: 1)  
  
    .childChannelOption(ChannelOptions.socket(IPPROTO_TCP, TCP_NODELAY), value: 1)
var serverChannel1 = try! bootstrap1.bind(host: "0.0.0.0", port: 8080).wait()

let channel = try! ClientBootstrap(group: MultiThreadedEventLoopGroup(numberOfThreads: 1)) 
    .connect(host: "localhost", port: 8080).wait()

print("channel", channel)

try! serverChannel1.close().wait()

_ = try! bootstrap1.bind(host: "0.0.0.0", port: 8080).wait()

from kitura-nio.

Thanks @weissi and @pushkarnk

PR #82

from kitura-nio.

We should try using ephemeral ports for all the tests.

from kitura-nio.

Few KiturNetTests ephemeral ports but not all the tests are using the same. Discussed with @djones6 and agreed to port the KituraTests to use ephemeral ports.

from kitura-nio.

@pushkarnk setting

.serverChannelOption(ChannelOptions.socket(SocketOptionLevel(SOL_SOCKET), SO_REUSEADDR), value: 1)

should fix that. I reckon KituraNet just sets that automatically whilst NIO wants the user to specify it explicity.

from kitura-nio.

@pushkarnk ^ did you see this?

from kitura-nio.

@ianpartridge Yes, we tried SO_REUSEADDR in the past and it doesn't solve this problem.

@weissi I remember discussing this issue with you over Slack back in May. SO_REUSEADDR wasn't helping us and you suggested we try SO_REUSEPORT.

from kitura-nio.

@weissi Now I recall it better, we needed SO_REUSEPORT for a particular test called testServersSharingPort. Looks like SO_REUSEADDR is enough for this issue.

from kitura-nio.

@pushkarnk yes, testServersSharingPort sounds like two servers bind on the same port which would be SO_REUSEPORT (and a new enough Linux version)

from kitura-nio.

in any case, I'd recommend SO_REUSEADDR for all tests and SO_REUSEPORT for the ones where you actually need to have two (or more) servers bind to the same port at the same time.

from kitura-nio.

@weissi This is the gist of the problem we're facing using both SO_REUSEADDR and SO_REUSEPORT

Consider this NIO program (note the commented line of code):

import NIO 

let bootstrap1 = ServerBootstrap(group: MultiThreadedEventLoopGroup(numberOfThreads: System.coreCount))
    .serverChannelOption(ChannelOptions.socket(SocketOptionLevel(SOL_SOCKET), SO_REUSEADDR), value: 1)  
  //.serverChannelOption(ChannelOptions.socket(SocketOptionLevel(SOL_SOCKET), SO_REUSEPORT), value: 0)
    .childChannelOption(ChannelOptions.socket(IPPROTO_TCP, TCP_NODELAY), value: 1)
var serverChannel1 = try! bootstrap1.bind(host: "0.0.0.0", port: 8080).wait()

let channel = try! ClientBootstrap(group: MultiThreadedEventLoopGroup(numberOfThreads: 1)) 
    .connect(host: "localhost", port: 8080).wait()

print("channel", channel)

try! serverChannel1.close().wait()

_ = try! bootstrap1.bind(host: "0.0.0.0", port: 8080).wait()

Run the above program. You should see no error.

Now, remove the comment and include that line of code:

import NIO 

let bootstrap1 = ServerBootstrap(group: MultiThreadedEventLoopGroup(numberOfThreads: System.coreCount))
    .serverChannelOption(ChannelOptions.socket(SocketOptionLevel(SOL_SOCKET), SO_REUSEADDR), value: 1)  
    .serverChannelOption(ChannelOptions.socket(SocketOptionLevel(SOL_SOCKET), SO_REUSEPORT), value: 0)
    .childChannelOption(ChannelOptions.socket(IPPROTO_TCP, TCP_NODELAY), value: 1)
var serverChannel1 = try! bootstrap1.bind(host: "0.0.0.0", port: 8080).wait()

let channel = try! ClientBootstrap(group: MultiThreadedEventLoopGroup(numberOfThreads: 1)) 
    .connect(host: "localhost", port: 8080).wait()

print("channel", channel)

try! serverChannel1.close().wait()

_ = try! bootstrap1.bind(host: "0.0.0.0", port: 8080).wait()

Wait until netstat -antup does NOT show an entry for 127.0.0.1:8080.

Compile and run the second program. The second bind() fails with an error!

This means, not setting SO_REUSEPORT has a different effect than setting SO_REUSEPORT = 0. I'd think they are the same!

Is this expected? Perhaps its something to do with the loopback connection?

cc @nethraravindran

from kitura-nio.

@saiHemak @nethraravindran
We need to set SO_REUSEADDR to 1 by default and if allowPortReuse = false we must try to simply not include this line of code.

from kitura-nio.

@saiHemak ^^^

from kitura-nio.