Run Task One Another / Control Concurrency in Queues about coravel HOT 1 OPEN

Coravel could offer something like that.

The problem with this though is that this would apply to all messages in the queue and would significantly reduce throughput across the board.

So this would depend on having multiple independent queues, and then implement what you've suggested per queue. This would be a bit of work 😅.

One approach that could work, is you have your queue message handler put a "job" item in storage (a database table, document, etc.). Then, using a the scheduler, every X seconds (or whatever), you can purge that database table (etc.) using the solution you suggested.

This would work well because the scheduler has the option to PreventOverlap so that you can more reliably process these items.

For example, the queue doesn't support preventing any overlap which could occur in the example you gave, if implemented inside of the queue message handler (e.g. your first queue handler is doing web scrapping but then the queue triggers a purge/consumation 30 seconds later. The first one isn't done yet...). You could use a static semaphore here to "fix" that.

That would be my suggestion at the moment: queued message complete quickly and don't get blocked/slowed by writing a job object to storage. Have a scheduled job to process those jobs every X seconds and use PreventOverlap to prevent overloading your RAM, CPU, etc.

Have any thoughts on this approach? (It's a common pattern to use for this specific scenario and is exactly why the PreventOverlapping scheduler feature exists 👍).

from coravel.