Comments (11)
+1
from sidekiq-unique-jobs.
+1
from sidekiq-unique-jobs.
+1
from sidekiq-unique-jobs.
+1
from sidekiq-unique-jobs.
Ok so be gentle with me while I try to explain. I'll be the first one to admit I suck at both documentation and explaining this. Work in progress sort of speak.
- Worker arguments are taken into account. It is also possible to select what arguments should be taken into consideration for uniqueness by specifying a lambda or a class method to handle this. Might have been added since after you asked your question by the by.
- The expiration doesn't need to be set, it can be thought of as a simple time out for uniqueness meaning if you set it to two hours no jobs with the same arguments or unique_arguments will be scheduled until that time has passed.
Unfortunately it seems like workers calling nested workers causes jobs to be duplicated still (like in #10 ). If anyone want to take a stab at reproducing the problem in a test we should be able to fix it.
from sidekiq-unique-jobs.
I don't understand what exactly the expiration parameter does, either. Does it only affect jobs scheduled with #perform_in
, but not with #perform
?
If it affects #perform
, I think a better default should be 0, instead of the current 30 * 60
(30 minutes).
from sidekiq-unique-jobs.
@nberger it only affects jobs scheduled with perform_in
or perform_at
.
from sidekiq-unique-jobs.
@mhenrixon I wouldn't mind also a brief explanation in the README of exactly how the uniqueness is established. This would be nice so I don't have to dig through the code to ensure locking is performed properly. I would much rather take your word for it!
That said, I noticed that setex
is used to increment a counter on a key that is the arguments to the worker. Is that correct? I noticed a pattern in the redis documentation using setnx (which now states to use plain old set
to implement a locking system instead.
A small explanation of the locking procedure and how it is thread safe would definitely help me, and I'm sure many others, gain even more confidence in using the gem. Any chance you could clarify it for me? Thanks!
from sidekiq-unique-jobs.
Hello everyone,
I had a few questions about this gem and this issue thread seems to be somewhat centered around my questions.
Essentially, if I have a worker ("DoStuff") and I queue a job for that worker by the following:
DoStuff.perform_async("with unique argument")
and then I run that same command again with the same arguments, I want it to not add the second instance of the job to the queue if the first instance has not completed yet.
The way I've read the documentation, I expect the job not to be duplicated no matter how long it has been if a job with the same worker, queue, and arguments is still waiting to be processed.
What I'm currently experiencing is that the job won't add to the queue if it's within the expiration time. However, if the first job has not been completed, but the unique job expiration time has passed (10 minutes, in my case) and I run it again, it does add the duplicate job even if the first one has not completed!
Is this the expected behavior of the gem? If not, is there a configuration option I am missing?
Here is an example of a worker and the options I'm using:
class DoStuff
include Sidekiq::Worker
sidekiq_options :queue => "queue_1"
sidekiq_options unique: true, unique_job_expiration: 60 * 10
def perform(arguments)
# Do some unique things.
end
end
Thanks,
Tye
from sidekiq-unique-jobs.
@tyetrask yeah that is sort of expected. I suggest you try something like sidekiq-throttler instead. That should better help you achieve what you want I am opening an issue for deciding on how to proceed with this.
from sidekiq-unique-jobs.
Hey @mhenrixon, thanks for the information! We needed to move forward with our project so we ended up writing the middleware that performed how we needed. I appreciate all of the work on this and will be keeping an eye on it in the future. Thanks again!
from sidekiq-unique-jobs.
Related Issues (20)
- Reaper manager registration is subject to race conditions HOT 1
- Should client middleware also be added to the Sidekiq server config? HOT 2
- After sidekiq shutdown (with running job) lock digest is not pushned back HOT 8
- Returning same job id HOT 7
- Sidekiq 7: Unsupported command argument type: TrueClass HOT 1
- register_reaper_process nx: true crash Sidekiq on startup HOT 2
- uniquejobs:digests sorted set seems to grow forever HOT 3
- should respond to `has_valid_sidekiq_options?` HOT 4
- Compatibility with Sidekiq7 HOT 4
- until_and_while_executing not entering perform method on initial run HOT 7
- No 'Changelog' link is being displayed on https://rubygems.org/gems/sidekiq-unique-jobs HOT 1
- Documentation of lock_ttl is inconsistent HOT 5
- Catched a deadlock on Sidekiq 7+ set up using middleware.
- Conflicts with sidekiq-lock gem HOT 4
- Read Timeouts and high CPU spike after upgrading from 8.0.5 to 8.0.10 HOT 9
- Issue with until_and_while_executing lock when job is not enqueued through sidekiq
- until_and_while_executing with separate server and client lock_args broken since 8.0.8 HOT 4
- Replace MD5 with SHA256+ HOT 2
- lock_prefix argument is not respected
- TypeError ยท Unsupported command argument type: Array HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sidekiq-unique-jobs.