Comments (7)
For us, one of the advantages of delayed is that it does not need LISTEN/NOTIFY or advisory locks because we need to run pg_bouncer in transaction mode.
Due to how we deploy our application, there are many server and worker processes, each having an ActiveRecord pool. They can easily open up to 2000-3000 database connection but are mostly idle. With pg_bouncers transaction mode, the actual active server connection count can be reduced to ~20, but some connection-level features are not available.
For that reason, we cannot use good_job, but only delayed, that does efficient polling only.
We are still testing a few things, especially with the job timeout things because we have some jobs that can run for hours or days, and are looking into unique job keys to restrict concurrent scheduling. We would like to use this to schedule cron jobs with a distributed scheduler.
That is why we are testing with delayed now. We definitely prefer the co-transactionality over our current sidekiq cluster too! Hopefully, delayed will do a good job for us soon too ;)
from delayed.
We've been planning to put a feature matrix into the README to help clarify some of this but the in general our philosophy drives us to the most constrained solution that can reasonably work for general purpose background workloads (though obviously through Betterment's lens). We use it and rely upon it for our most important stuff at pretty significant scale.
On the topic of dashboarding, we've got some stuff that we've been considering tidying up for public consumption, so it's great to hear it's important to you for adoption.
from delayed.
I've used good_job
previously but as I'm experimenting moving Postgres -> SQLite for most cases I need a job backend that supports SQLite... doesn't seem there are many out there but this is one of them
from delayed.
This is a very good point, we recently switched from MariaDB to Postgres specifically because many gems (including good_job) require it. It is a big advantage that this gem supports other databases.
from delayed.
see also this thread on dashboarding - feel free to add or upvote specifics you're looking for in dashboarding.
from delayed.
My initial drive to good_job was the strong DB transaction guarantees for queueing jobs that you don't have out of the box with separate queuing mechanisms like Redis (unless you have some transactionally staged job queues)
As I don't need to process thousands of jobs per second I prefer having my job queue in the same DB with transactional guarantees AND not having to run a separate queueing backend, i.e. Redis
from delayed.
For us, one of the advantages of delayed is that it does not need LISTEN/NOTIFY or advisory locks because we need to run pg_bouncer in transaction mode.
Due to how we deploy our application, there are many server and worker processes, each having an ActiveRecord pool. They can easily open up to 2000-3000 database connection but are mostly idle. With pg_bouncers transaction mode, the actual active server connection count can be reduced to ~20, but some connection-level features are not available.
For that reason, we cannot use good_job, but only delayed, that does efficient polling only.
We are still testing a few things, especially with the job timeout things because we have some jobs that can run for hours or days, and are looking into unique job keys to restrict concurrent scheduling. We would like to use this to schedule cron jobs with a distributed scheduler.
That is why we are testing with delayed now. We definitely prefer the co-transactionality over our current sidekiq cluster too! Hopefully, delayed will do a good job for us soon too ;)
How did things fare with usage of delayed so far? We're exploring new solutions and the pg_bouncer constraint is something we have too!
from delayed.
Related Issues (20)
- Configuring Exception Notification on Failure HOT 4
- cron, separate gem or pull request? HOT 3
- Running against a secondary database HOT 10
- how many simultaneous workers can we run? HOT 2
- Periodic (Cron) jobs HOT 8
- Open to PRs? HOT 10
- Concurrent job not running HOT 4
- Error while reserving job(s): PG::SyntaxError: ERROR: syntax error at or near "SKIP" HOT 5
- how to monitor worker processes HOT 2
- Clarification on idempotency requirement HOT 8
- Difference in error behavior when a job is undefined HOT 4
- Changes in schema usage/assumptions from `delayed_job`? HOT 2
- Consider adding jitter to retry interval
- Handling duplicate jobs HOT 4
- Job failed to load: undefined class/module Delayed::JobWrapper. HOT 2
- Locks not being cleared on SIGKILL HOT 2
- Support raise_signal_exceptions HOT 5
- Possible unthrottled spinloop when jobs fail to deserailize HOT 1
- Dashboard UI HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from delayed.