Concurrent job not running about delayed HOT 4 CLOSED

Yes totally makes sense, as it works in most cases as well - not always easy but almost always works.

Unfortunately as I'm doing live recordings the job has to run as long as the recording. It's some kind of background job but doesn't fit this most common definition of it - it fits more a daemon or supervisor process... but it's scheduled as well 😅

from delayed.

Hi @AxelTheGerman!

So, yes, you've hit on something that I think is expected behavior, which is that when a worker picks up a set of jobs, it will wait until the last job in that batch completes before it attempts to pick up any additional jobs. As a result, if you have long-running jobs and need higher worker availability, you must run multiple workers at once.

A recommendation for improving queue health in general is to break long-running jobs down into shorter-running jobs (delayed is optimized for handling a huge volume of relatively short jobs, not a low volume of long-running jobs), but if that's not possible, then in addition to adding workers, you could consider having dedicated queues for particular scheduled tasks. It's also worth double checking your Delayed::Worker.max_run_time to make sure that things aren't timing out.

from delayed.

Hi @smudge thank you for the in depth answer... just wondering if that's worth documenting somewhere - but maybe it's edge case enough that this issue is enough documentation.

Most job frameworks seem to be optimized for a high volume of shorter jobs - I guess it's the easier problem to solve and (in most cases) write your code accordingly.

For now delayed still works good enough for me, though I'll keep my eyes open for a tool actually built for longer running jobs (making sure they didn't die, surviving re-deploys etc.)

Great tip on having multiple queues and workers per queue... would be a shame to have some important smaller jobs wait behind hour long ones :P

from delayed.

Hi Axel, a common perspective in modern infrastructure and service design is to keep units of work small so that they can be resumed and retried quickly in the event of infra failure or network partition. Delayed is designed for short jobs because of that quality rather than because it’s easier per se; though I admit it’s also easier. It sort of mirrors the invention and subsequent adoption of RAID arrays instead of expensive highly reliable disk drives.

We have techniques for breaking down big batches of work into small jobs that reliably complete in aggregate which I think we may look into open sourcing in the future.

from delayed.