Reconnect has potential race conditions about nats.go HOT 5 CLOSED

I will take a closer look at the trace, but in general under spinUpSockerWatchers, the nc.wg.Wait() is there to make sure the readLoop and flusher from the old connection are torn down before the new ones get spun up.

from nats.go.

I understand what the waitgroup is for. You can easily run into situations where it may not complete because the readLoop and flushers have moved to the new connection/buffered writer. It doesn't always happen but it can.

The stack just shows the situation when something similar happens with the parser state.

There is a lot of code checking for the state of the connection, closed, reconnecting, etc... but for most of the routines they will see connected or closed because of the locking and how they are driven.

from nats.go.

I looked briefly, and I think doReconnect() can only proceed at replacing the nc.bw and the connection itself after it is known that readLoop and flusher are shutdown. Will look at moving that logic gate to before reconnect begins in earnest.

from nats.go.

I found a bug here from the last commit which may cause you issues. I am working on this problem (this thread), and found it as I was moving the shutdown logic to before we began processing a reconnect.

https://github.com/apcera/nats/blob/master/nats.go#L1438

from nats.go.

I have it fixed, just an FYI..

from nats.go.