Giter Club home page Giter Club logo

edamame's Introduction

Beanstalk + Tokyo Tyrant = Edamame, a fast persistent distributed priority job queue

Edamame combines the Beanstalk priority queue with a Tokyo Tyrant database and God monitoring to produce a persistent distributed priority job queue system.

  • fast, scalable, lightweight and distributed
  • persistent and recoverable
  • scalable up to your memory limits
  • queryable and enumerable jobs
  • named jobs
  • reasonably-good availability.

Like beanstalk, it is a job queue, not just a message queue:

  • priority job scheduling, not just FIFO
  • Supports multiple queues (‘tubes’)
  • reliable scheduling: jobs that time out are re-assigned

It includes a few nifty toys:

  • Scripts for God to monitor and restart the daemons
  • Command-line management scripts to load. enumerate, empty, and show stats for the db+queue
  • The start of a lightweight web frontend in Sinatra.

Documentation

The bulk of the documentation is at http://mrflip.github.com/edamame Go there instead.

Help!

Send Edamame questions to the Infinite Monkeywrench mailing list

Requirements and Installation

Install

Get the code

We’re still actively developing edamame. The newest version is available via Git on github:

$ git clone git://github.com/mrflip/edamame

A gem is available from gemcutter:

$ sudo gem install edamame --source=http://gemcutter.org

(don’t use the gems.github.com version — it’s way out of date.)

You can instead download this project in either zip or tar formats.

Get the Dependencies

To finish setting up, see the detailed setup instructions and then read the usage notes

See the Detailed install instructions (it also has hints about installing Tokyo*, Beanstalkd and friends.

Endnotes

Caveats

Weaknesses? Mainly that it will make an Erlang’er cry for its lack of concurrency correctness. Its goal is to work pretty well and to recover gracefully, but its design limits .

  • We store jobs in two places: the central DB and the distributed queue.
  • As always, your jobs must either be idempotent, or harmless if re-run: a job could start and do some or all of its job — but lose contact with the queue, causing the job to be re-run. This is inherent in beanstalkd (and most comparable solutions), not just edamame.
  • Although God will watch the daemons, it won’t repopulate the queue or restart a worker that fails.

TODOs

  • Restarting is still manual: you have to run bin/edamame-sync to reload the queue from the database
  • The sinatra queue viewer doesn’t work at the moment.

Links

There’s a fuller set of docs at http://mrflip.github.com/edamame


More info

Credits

Edamame was written by Philip (flip) Kromer ([email protected] / @mrflip) for the infochimps project

Help!

Send wuclan questions to the Infinite Monkeywrench mailing list

edamame's People

Stargazers

 avatar Angus H. avatar  avatar Davin Potts avatar Alex Zinchenko avatar Kostas Nasis avatar  avatar Cuper avatar Pham Cong Dinh avatar Max Murphy avatar Ryuzo Yamamoto avatar Flinn Mueller avatar François Lamotte avatar Alice Davis avatar Ilya Grigorik avatar Joshua Hull avatar Dhruv Bansal avatar Jérémy Chatard avatar Philip (flip) Kromer avatar  avatar John Mettraux avatar Tatsuhiko Miyagawa avatar Orlin M Bozhinov avatar  avatar Adrian Madrid avatar Nadeem Bitar avatar Lourens Naudé avatar

Watchers

Philip (flip) Kromer avatar Dhruv Bansal avatar  avatar  avatar

Forkers

applio

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.