possible frame parsing bug about bunny HOT 11 CLOSED

It could. What we need is a script to reproduce it (or more than one script) and information about how your applications use threads and channels.

from bunny.

Hi Michael,

It may take a bit of time to create scripts that isolate the problem. In the mean time I can tell you that we are not threading, but we have 5 resque worker processes running code very similar to this ( some names changed to protect the innocent :). The workers communicate with a pool of java processes via rabbit, and the java processes are not encountering any errors.

amqp_config = {:spec => '09', :insist => true, :host => 'host', :pass => 'pass', :user => 'user'}
exchange = 'to_device'
bunny = Bunny.new(amqp_config).tap { |b| b.start }
bunny.exchange 'reply'
bunny.exchange exchange, :type => :fanout
reply = bunny.queue
reply.bind('reply', :key => reply.name)
amqp.exchange(exchange).publish(msg.to_kv, :reply_to => reply.name)
reply.subscribe(:message_max => 1) do |delivery|
  response = delivery[:payload].to_kv_hash
end

from bunny.

The code above runs fine in moderate volume situations. When message throughput gets high, it fails when attempting to create the reply queue ( Client#check_response ) as noted above. I will work to create scripts that simulate the workers and the corresponding java processes and reproduce the problem with the simplest implementation, let me know if anything occurs to you in the meantime, especially if there is some way we can change our usage of Bunny to prevent the problem.

Thanks!
Eric

from bunny.

Sorry to butt in, however, have you tried using the qos() method before you subscribe? For example, in the code you showed earlier you would include the line -

bunny.qos()

after you create your Bunny instance and before your reply.subscribe() call.

Also, you need to set the :ack flag in the subscribe call, so the call should read -

reply.subscribe(:message_max => 1, :ack => true)

Regards,

Chris

from bunny.

Hi Eric,

Have you managed to resolve your problem? Can this issue be closed?

from bunny.

Hi Eric,

I'm going to close this issue, however, if you are still having problems please feel free to re-open it.

from bunny.

@celldee I think it should be OK to use basic.qos before adding any consumers. I did it many times with amqp gem and hot bunnies and it is a channel setting, not a consumer one.

from bunny.

Hi there,

I agree that this should be closed, because I am not able to give you code to reproduce the problem. However, I believe there is a problem, and it's worth keeping an eye on this area.

The problem persists after trying this things:

• Verified that each forked child has its own connection to Rabbit (in both passenger and resque_pool)
• Added a qos call as celldee described

However, because we don't know the root cause, and because it only happens under load, we cannot reliably reproduce it. When it does happen, all further requests from bunny to rabbit on that specific connection log garbled response data from rabbit. Given this, we've implemented a workaround that just reconnects to rabbit whenever this error occurs, thus restarting the response parsing sequence and alleviating the problem.

In addition to using bunny in our web server and workers, we also use the amqp gem from another server to connect to our rabbit instance and have no problems there. We also connect to our rabbit instance from several java processes and no problems there. Given that we're pretty sure the problem is a very subtle one in bunny's rabbit response parsing that requires a 'perfect storm' set of circumstances under load.

from bunny.

FYI, our future plans are to stop using bunny and run a separate event machine daemon that receives requests, passes them on to rabbit using the amqp gem, and enqueues workers to handle the responses.

from bunny.

If it only happens under load I am growing convinced it is a concurrency issue. We have fixed a few of those between amqp gem 0.8.0 and 0.8.4 (the latest release) and all were around framing and only affected apps that do hundreds and thousands messages per second.

from bunny.

@edraut Thanks for your efforts in trying to solve this issue. @michaelklishin Do you have some sort of test setup that could be used to find the problem area in Bunny?

from bunny.