Stomp connections get terminated by server but not cleaned up about rabbitmq-stomp HOT 13 CLOSED

Thank for the really detailed issue report!

from rabbitmq-stomp.

@martink2 please send me the dump you have, it would help uncover what may be tripping up the parser (that answers your 1st open question).

As for the 2nd one, this may be a bug, and now that we have an error message that correlates with connections being open, it should be much easier to reproduce.

@martink2 do you run a single node or multiple ones?

from rabbitmq-stomp.

@michaelklishin We can see the error on both single nodes and clusters.
Dump is send via email.

Thanks

Martin

from rabbitmq-stomp.

@martink2 can you post the output of netstat -tcp when that happens? How many connections do you have on average?

What does sudo sysctl -a | grep output on the server?

from rabbitmq-stomp.

More questions: do your clients use NAT?

from rabbitmq-stomp.

On average we have 11.000 connections.

for a failed connection from 10.97.168.128:54194

netstat -n -tcp | grep 10.97.168.128:54194 : No output

For what sysctl key would you like me to grep ?

As to the nat a few of our clients are behind a NAT but not the majority of them and it
does not show a higher chance of happening for NAT'ed vs direct connections.

But our clients use a lot of short lived connections so the ephemeral port range rolls
over very quick.

Thanks

Martin

from rabbitmq-stomp.

Yes, the ephemeral port range roll overs is what I'm investigating.

I'm after the net.ipv4.tcp_tw_recycle and net.ipv4.tcp_tw_reuse sysctl keys. In fact, anything net.ipv4 may be useful (assuming your clients use IPv4).

You can send me the output privately, [email protected].

from rabbitmq-stomp.

Some findings from today:

• clients rolling over ip_local_port_range and attempting to re-connect constantly (sequentially) do not leave connections behind

Will try with clients doing the same thing concurrently.

from rabbitmq-stomp.

OK, I believe I have a reliable way to reproduce this. The issue only has so much to do with high TCP connection churn and all that: the reader does not tell the processor to terminate in every case. Phew.

from rabbitmq-stomp.

@michaelklishin

I deployed the fixed code to our production node and it looks very good, thanks a lot
for the quick help. Any timeline for the 3.5.2 release of rabbitmq yet?

Martin

from rabbitmq-stomp.

We try to release a bugfix release once a month. 3.5.2 should be released around the first half of May.

from rabbitmq-stomp.

That said, we also have nightly builds.

from rabbitmq-stomp.

@martink2 thank you very much for giving it a try so quickly!

from rabbitmq-stomp.