Large retry queue causes reaper to run too slow about sidekiq-unique-jobs HOT 6 CLOSED

The original line of thinking was that we only need one item but if performance suffers because of that then I am more than happy to increase it. We could certainly do more.

I guess redis has to do more iterations, the testing perhaps becomes a little more complex. Would be cool to have a real world test for the performance of this since we have had some issues around the reaper performance.

If I recall correctly I have some performance related tests already.

from sidekiq-unique-jobs.

Hi @dedman,

I have tried to run some performance tests on the line of code you suggested.

With 1_000_000 (1 million) rows in the retry queue, I get this:

  1) SidekiqUniqueJobs::Orphans::RubyReaper#in_sorted_set? when retried with the job in retry is expected to perform under 2 ms
     Failure/Error: it { expect { service.send(:in_sorted_set?, "retry", digest) }.to perform_under(2).ms }
       expected block to perform under 2 ms, but performed above 4.26 ms (± 6.11 ms)

Setting the count to 1 or 1000 is hardly noticeable. Agreeably this isn't the real world so might not get real world results.

from sidekiq-unique-jobs.

You can check the progress at #777

from sidekiq-unique-jobs.

I think we're hitting this problem too at the moment

from sidekiq-unique-jobs.

Sorry I didn't follow up here earlier. I chatted to my management about this issue earlier in the year and we decided to wait until we were ready to use Sidekiq Enterprise which has a unique jobs feature.

For the last 6 months we kept an eye on the retry queue and tried to keep it under 200-300 jobs. We also increased the timeout on the reaper which lessened the problem a little. We had cron jobs not run probably once a month because of this issue, but we generally noticed early enough to stop it having any big impact.

We were already thinking about moving to Sidekiq Enterprise, so when this issue came up it was the final confirmation it would make sense for us to move over to Enterprise.

from sidekiq-unique-jobs.

Great stuff @dedman, if people paid me those monthly fees I could afford the same quality :)

from sidekiq-unique-jobs.