An opinionated background job runner. Job payloads, logs, and metadata are backed by Postgres. Job queueing and execution are backed by RabbitMQ.
Also, ES6/7.
Lots of job queues are backed by Redis, but in order to leverage Redis's queueing in a consistent way, you have to write a lot of Lua scripts, and you can still easily run into issues with jobs getting stuck in weird, inconsistent states.
Redis is also not a great option if you want to be able to search and filter jobs (by type, by payload, etc). It's up to you (or your queueing library) to implement your own indexing.
Probably not. It wouldn't be my first Terrible Idea™.
Not implemented yet:
- Automatic failure retries
- Tests
Feel free to add any other triggers/indexes you'd like.
CREATE TABLE heretic_jobs (
id SERIAL NOT NULL PRIMARY KEY,
queue_name text NOT NULL,
status text DEFAULT 'pending',
payload jsonb,
attempt_logs jsonb[] DEFAULT '{}',
max_attempts int NOT NULL DEFAULT 1,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
last_attempted_at timestamptz
);
CREATE INDEX ON heretic_jobs (queue_name);
CREATE INDEX ON heretic_jobs (status);
CREATE FUNCTION heretic_updated_at_timestamp() RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = now();
RETURN NEW;
END;
$$ LANGUAGE 'plpgsql';
CREATE TRIGGER update_heretic_jobs_updated_at
BEFORE UPDATE ON heretic_jobs
FOR EACH ROW EXECUTE PROCEDURE heretic_updated_at_timestamp();import Promise from 'bluebird';
import Heretic from 'heretic';
import knex from 'knex';
const heretic = new Heretic('amqp://127.0.0.1', knex({
client : 'pg',
connection : 'postgres://127.0.0.1',
}));
heretic.process('my.job', function(job, message, done) {
Promise.resolve()
.then(() => {
// do some processing or something. whatever you want!
})
.nodeify(done);
});
heretic.process('other.job', function(job, message, done) {
setTimeout(() => {
done(new Error('job failed'));
}, 1000);
});
// the 'jobError' event happens when a job message is published in RabbitMQ that
// can never be handled correctly (malformed JSON, job id doesn't exist in the
// database, etc.). The message will be dead-lettered for later inspection (by you)
heretic.on('jobError', (err) => {
console.error('Error with job!', err.stack);
});
// the 'jobFailed' event happens when a job fails, but in a recoverable way. it
// will be automatically retried up to the maximum number of retries.
heretic.on('jobFailed', (err) => {
console.error('Job execution failed!', err.stack);
});
// enqueue a job
heretic.enqueue('email.send', {
to : '[email protected]',
body : 'Hi',
});
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent