Flood publishing causes socket to be nil about bunny HOT 10 CLOSED

RabbitMQ will block publishers that overrun consumers. In theory it should not cause any socket exceptions but presumably it does. Network error handling is very simplistic right now, it is true.

from bunny.

Can you count after about what # of messages published you get this behavior? And post what's in the RabbitMQ log? Thanks!

from bunny.

Also, what's approximate median message size? So far I cannot reproduce this with small messages (1K) and 100,000 messages published.

from bunny.

It happens at about 270K messages. Some other details about my environment:

• I'm writing to a topic exchange.
• There is a durable queue that gets the messages from the exchange.

The message size doesn't really seems to matter. It happens even with 4B messages.
I believe the queue getting messages from the exchange is relevant.

These are the last lines in my Rabbit logs:

=INFO REPORT==== 15-Feb-2013::10:30:55 ===
closing AMQP connection <0.24366.1> (127.0.0.1:64089 -> 127.0.0.1:5672)

=INFO REPORT==== 15-Feb-2013::10:30:56 ===
accepting AMQP connection <0.24407.1> (127.0.0.1:64092 -> 127.0.0.1:5672)

=ERROR REPORT==== 15-Feb-2013::10:30:56 ===
closing AMQP connection <0.24407.1> (127.0.0.1:64092 -> 127.0.0.1:5672):
{bad_header,<<1,0,2,0,0,0,25,65>>}

from bunny.

That means RabbitMQ somehow gets an empty frame header, considers it a fatal error and closes the connection.
Do you have an example script that reproduces the issue? What RabbitMQ version do you run?

from bunny.

Yes, the script I posted in the first message reproduces the issue when there is a durable queue subscribed to the exchange.
I'm running 3.0.1

from bunny.

Can you check that RabbitMQ log does not have anything related to alarms when this happens?

I cannot reproduce this. My theory is that when your machine goes low on memory or disk space (< 1 GB), RabbitMQ blocks the publisher, eventually OS socket buffer gets full and this results in exceptions
that clear @socket in the transport. And because the publisher is not stopped, you see this nil pointer
exception.

from bunny.

Setting RabbitMQ memory watermark to a really low value suggests it is not the issue:

sudo rabbitmqctl set_vm_memory_high_watermark 0.05

will cause the publisher quickly get blocked but it does not cause the script to fail with any exceptions.

from bunny.

I can't reproduce it across 2 OSes, 2 RabbitMQ versions, 2 Ruby implementations. We have a test for a very
similar scenario, unless an example that reproduces it can be found, I consider this issue resolved.

from bunny.

0.9.0.pre8 is out, please give it a try.

from bunny.