persistent queues remove jobs from the database before being completed about girl_friday HOT 11 CLOSED

Those semantics are the same for Redis and in-memory too.

Adding the semantics as you request would add a lot of complexity and push GF closer to being a full MQ system. That's not my vision for girl_friday. Do you have any thoughts on how to change this simply?

from girl_friday.

When the pop occurs it needs to pass along a job id also.

      def pop  
        val = collection.find_and_modify(:sort => [['$natural', 1]],
                                         :query  => {:status => 'push'},
                                         :update => {:status => 'pop'},
                                         )
        [ Marshal.load(val["work"]), val['_id'] ] if val 
      end 

Right now it seems like the job removal can be done in a simple way outside of the library code.

GirlFriday::WorkQueue.new(store: GirlFriday::Store::Mongo) do |opts, id|
   JobProcessor.process(opts)
   collection.remove({:_id => id, :status => 'pop'})
end

So it seems like there should be a way to push that back into the library in a simple way. If not in a more direct way, perhaps using a GirFirday::MongoQueue that wraps its block with the removal logic (and sets the store option).

from girl_friday.

This was brought up the other day when I was demoing GF at a lunch-and-learn and sounded like a bit of a block for that team.

from girl_friday.

So we aren't the first ones using Mongoid with find_and_modify instead of a queue:
http://blog.boxedice.com/2011/09/28/replacing-rabbitmq-with-mongodb/

They say to avoid the remove (causes write locks) and just use a capped collection (fixed size and will start dropping old records once it fills) . The weakness of using a capped collection is sharding, but it seems that we are already sharding per job queue, and you could always start creating multiple queues per job type when you need too.

That means for MongoDB no changes to GirlFriday are requied. So as far as I am concerned this issue is solved for MongoDB. I will send a pull request next week.

I will leave this issue open as a reminder to you to make the documentation clear that the in memory and Redis stores will drop the queue on a server crash.

from girl_friday.

@gregwebs thanks for the info, but just to be clear, we only support the in-memory & Redis backends. The mongo backend was just an example I threw together one morning, which is why it's in the examples folder. But I'll definitely check out a pull request if it improves the example.

from girl_friday.

I updated the wiki to state that the queue will be lost when Redis goes down.

I now discovered that MogoDB has tailable cursors! This is perfect for a queue, you get messages without polling, but the implementation may not match up well with the current GirlFriday backend API.

from girl_friday.

Why would the queue be lost if Redis goes down? Redis is persistent, that's why we support it - in-memory for best effort processing, redis for more reliable processing. Its list structure is a natural fit for a queue.

from girl_friday.

I have never used Redis. I was under the impression that it is always used
to enhance performance but never as an authoritative data store.

Here is their persistence documentation: http://redis.io/topics/persistence
So it does appear that you can achieve persistence by using an append only
file and fsyncing every operation. However, they seem to discourage that by
calling it "very slow", and i have the (possibly mistaken) impression that
very few users are using this setting.

But even if you use these settings you are still in big trouble when you
have a hard drive failure- personally I wouldn't consider a store persistent
unless it supported replication.

Feel free to update the wiki as you see fit.

On Fri, Sep 30, 2011 at 7:44 AM, Mike Perham <
[email protected]>wrote:

Why would the queue be lost if Redis goes down? Redis is persistent,
that's why we support it - in-memory for best effort processing, redis for
more reliable processing. Its list structure is a natural fit for a queue.

Reply to this email directly or view it on GitHub:
#35 (comment)

from girl_friday.

Redis supports replication also. http://redis.io/topics/replication

from girl_friday.

We're not guaranteeing the durability of your out of app database, just that if your app were to go down, your jobs would not necessarily disappear.

from girl_friday.

Cool, obviously I don't know much about Redis :). I updated the wiki to note
that Redis needs to be configured for the data to last a Redis crash. I
think it is much more important to be clear on the limitations a project has
then its actual features. That way people like me won't get confused :)

On Fri, Sep 30, 2011 at 9:06 AM, Jesse Cooke <
[email protected]>wrote:

We're not guaranteeing the durability of your out of app database, just
that if your app were to go down, your jobs would not necessarily disappear.

Reply to this email directly or view it on GitHub:
#35 (comment)

from girl_friday.