Giter Club home page Giter Club logo

bossbat's Introduction

bossbat

Stupid simple distributed job scheduling in node, backed by redis.

npm Version License Build Status Coverage Status

Bossbat combines schedulers and workers into a single concept, which aligns better with most node applications. All of the jobs run with a redis lock, preventing more than once instance from performing a given job at a time.

Usage

Bossbat is published on npm, and can be installed simply:

npm install bossbat --save

Or if you use Yarn in your project:

yarn add bossbat

You can then import the module and use it normally. For more details, see the API documentation.

import Bossbat from 'bossbat';

// Make sure you name your variables clever things:
const fred = new Bossbat({
  connection: {
    host: '127.0.0.1',
    port: 6379,
  },
  // Set the redis key prefix:
  prefix: 'bossbat:',
});

// Hire for a job.
fred.hire('engineers', {
  every: '10 minutes',
  work: () => {
    // Do something...
  },
});

// Hire something as soon as possible:
fred.demand('engineers');

// You can also "qa" work:
fred.qa((jobName, jobDefinition, next) => {
  return next();
});

// Fire a job, this will cancel any pending jobs:
fred.fire('engineers');

// Shut down our instance.
fred.quit();

API

new Bossbat(options: Object)

Creates a new bossbat instance. All arguments are optional.

  • options.connection: Used to configure the connection to your redis. This accepts the same arguments that ioredis does.
  • options.prefix: A string that all redis keys will be prefixed with. Defaults to bossbat.
  • options.ttl: The number of milliseconds before a job times out. Setting it will change the maximum duration that jobs can hold a lock. By default, job locks will timeout if a job does not complete in 2000ms.
  • options.tz: An optional timezone used with jobDefinition.cron expressions.
  • options.disableRedisConfig: Disable usage of the redis CONFIG command, as it might be disabled in certain redis configurations. NOTE: If this option is used, the redis configuration should contain notify-keyspace-events Ex

bossbat.hire(jobName: String, jobDefinition: Object)

Schedules recurring work to be done.

  • jobName: A unique name for the job.
  • jobDefinition: The job definition can contain three properties: work, every, and cron.
    • jobDefinition.work: A function that is invoked to perform the job. If the work is async, this function should return a promise to properly reflect the completion of the job.
    • jobDefinition.every: A string which describes the interval the job will be run. This can either be a human-friendly string (which will be parsed by the timestring module), or a number reflecting the milliseconds to wait between each job run.
    • jobDefinition.cron: A string used to schedule work in more complex intervals. These are parsed with cron-parser.

It's possible to omit the every or cron properties if you don't wish to schedule recurring work, and just want to register a job.

bossbat.demand(jobName: String)

Runs a job as soon as possible, outside of the scheduled jobs. Demanded jobs still run with the same locking mechanism that scheduled jobs run with, ensuring only one instance runs the job at a time. This does not prevent any scheduled jobs from running, unless the demand is running at the same time as a scheduled job and all instances fail to acquire a lock on the job.

bossbat.qa(qaFunction: Function)

QA is used to register functions that will be invoked any time a job is run. This function can be called multiple times to register multiple QA functions. The passed qaFunction function will be called with jobName, and jobDefinition from the hire function, as well as a next function, which should be called when the QA function is complete. The next function returns a promise that can be used to run code after a job is completed.

For example, here is what a time logging QA function might look like.

bossbat.qa((jobName, jobDefinition, next) => {
  const startTime = Date.now();
  return next().then(() => {
    const endTime = Date.now();
    logToServer(`${jobName} completed in ${endTime - startTime}ms`);
  })
});

bossbat.fire(jobName: String)

Cancels any scheduled jobs with name jobName. This does not stop any demanded jobs from running.

bossbat.quit()

Shuts down a bossbat instance, closing all redis connections. This does not cancel any scheduled work, and does not stop it from running in any other bossbat instances.

How it works

Constructing a new bossbat instance sets up an expired key listener on your redis database. When you hire for a new job, Bossbat sets a key in Redis that expire when the first run should occur. When the key expires, the expired key listener is called and Bossbat does the following:

  1. Attempt to get a lock to perform the work. Only one instance of bossbat will acquire the lock.
  2. If the lock is acquired, then perform the work and move on.
  3. If the lock is not acquired, then move on.
  4. Schedule the next run of the job by setting a key that expires when the next run should occur.

It's worth noting that every instance of Bossbat attempts to schedule the next run of the job. This is okay because Redis will only schedule the first one that it receives, thanks to the NX option in SET. Calling demand performs the same operation as hire, except with a special key for demands.

bossbat's People

Contributors

kesne avatar minttu avatar tjokimie avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

bossbat's Issues

If `work()` resolves too quickly, multiple machines execute at the same time.

Consider this code:

this.bossbat.hire('cronjob', {
  every: '1 seconds',
  work: async () => {
    console.log(new Date(), 'Cronjob fired');
    await new Promise(resolve => setTimeout(resolve, 1000)); // This is a workaround that fixes this
  },
});

If the 'await new Promise' wasn't there, multiple machines would execute the same command because the lock was already released before the other server tried to get it.

Cron worker runs multiple times during one second

Cron worker runs multiple times during one second with every minute expression.

The following piece of code demonstrates the issue:

const Bossbat = require('bossbat');
const bossbat = new Bossbat();
let i = 0;

bossbat.hire('work', {
  cron: '*/1 * * * *',
  work: () => {
    console.log('Working...', new Date().toString(), i);
    i++;
  }
});

Output:

Working... Sun Feb 26 2017 14:30:00 GMT+0200 (EET) 0
Working... Sun Feb 26 2017 14:30:00 GMT+0200 (EET) 1
Working... Sun Feb 26 2017 14:30:00 GMT+0200 (EET) 2
Working... Sun Feb 26 2017 14:30:00 GMT+0200 (EET) 3
Working... Sun Feb 26 2017 14:30:00 GMT+0200 (EET) 4
Working... Sun Feb 26 2017 14:30:00 GMT+0200 (EET) 5
Working... Sun Feb 26 2017 14:30:00 GMT+0200 (EET) 6
Working... Sun Feb 26 2017 14:30:00 GMT+0200 (EET) 7
Working... Sun Feb 26 2017 14:30:00 GMT+0200 (EET) 8
Working... Sun Feb 26 2017 14:30:00 GMT+0200 (EET) 9
Working... Sun Feb 26 2017 14:30:01 GMT+0200 (EET) 10
Working... Sun Feb 26 2017 14:31:00 GMT+0200 (EET) 11
Working... Sun Feb 26 2017 14:31:00 GMT+0200 (EET) 12
Working... Sun Feb 26 2017 14:31:00 GMT+0200 (EET) 13
Working... Sun Feb 26 2017 14:31:00 GMT+0200 (EET) 14
Working... Sun Feb 26 2017 14:31:00 GMT+0200 (EET) 15
Working... Sun Feb 26 2017 14:31:00 GMT+0200 (EET) 16
Working... Sun Feb 26 2017 14:31:00 GMT+0200 (EET) 17
Working... Sun Feb 26 2017 14:31:00 GMT+0200 (EET) 18
Working... Sun Feb 26 2017 14:31:00 GMT+0200 (EET) 19
Working... Sun Feb 26 2017 14:31:00 GMT+0200 (EET) 20
Working... Sun Feb 26 2017 14:31:01 GMT+0200 (EET) 21

Expected output:

Working... Sun Feb 26 2017 14:30:00 GMT+0200 (EET) 0
Working... Sun Feb 26 2017 14:31:00 GMT+0200 (EET) 1

The issue occurs only if the runner completes relatively fast (in a couple of milliseconds).

Allow ":" character in job names.

Right now we use some lame split logic that prevents us from using : in job names. Let's fix that so that we can use whatever name we like. We'll need to ensure the logic works for demands as well as scheduled jobs.

Support cron formats

In addition to the every key (which is a string or ms value), we should support a new cron value on the JobDefinition that can be any cron value. We'll parse it and schedule based on it.

This should help support a wider variety of scheduled jobbing outside of just scheduled intervals.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.