Comments (17)
@girak could you point me to where that job is re-enqueued in sidekiq? I'm pretty sure we could mimic that behaviour.
from sidekiq-unique-jobs.
@mhenrixon https://github.com/mperham/sidekiq/blob/d40f7cc6af61ef9046c34429bef2932ca85f889f/lib/sidekiq/client.rb#L182
anywhere that refers get_sidekiq_options
from sidekiq-unique-jobs.
Ok so this happened to me as well now that I split up our backend processing in multiple different projects. I will see if I can get this solved somehow. First a failing test case.
from sidekiq-unique-jobs.
For us it looks like the error is with Sidekiq. Could you provide a stacktrace?
from sidekiq-unique-jobs.
I've got a quirky little RSpec spec here: https://gist.github.com/Felixyz/8781057
It test drives (synchronously) the Sidekiq retry mechanism with sidekiq-unique-jobs inserted in the middleware chain and attempts to mimic the setup we have, with two queues. The script does not reproduce the bug, from which I draw the conclusion that the bug (if it is a bug) is probably in Sidekiq's higher-level coordination classes, like Manager, Fetcher or Launcher.
Whatever it is, it's perhaps not so much a bug as an uncompatibility in sidekiq-unique-job's handling of the parameters? By which I mean: this is probably not an issue that we would report with Sidekiq itself, right?
from sidekiq-unique-jobs.
@girak Would you care to share your patch?
from sidekiq-unique-jobs.
One more observation that might be useful: we're getting fewer of these errors every day. My impression is that they might occur after a restart of the Sidekiq processes. We are running Sidekiq Pro. Are others who have been seeing this problem running Sidekiq or Sidekiq Pro?
from sidekiq-unique-jobs.
Yeah so @felixyz is right and that's why I was asking for a stacktrace. @girak I can't replicate that sidekiq unique jobs should be the problem in this situation. The crash actually happens in Sidekiq and the reason it happens in Sidekiq for us is because the wrong server was processing jobs it could not handle (we assigned it to the wrong queue).
from sidekiq-unique-jobs.
@mhenrixon I'm not sure you can get a great stack trace because of the async nature of Sidekiq, but (as you probably know) the line that causes the exception is:
/app/vendor/bundle/ruby/2.0.0/gems/sidekiq-unique-jobs-2.6.7/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:11:in `call'
In any case: yes, we also concluded that a worker was assigned a job from the wrong queue. However, I haven't been able to track down the reason why this happens. Any tips for that?
(The only thing I can think of is that our two kinds of workers inherit from a common base class, which is the one that mixes in Sidekiq::Worker.)
from sidekiq-unique-jobs.
@felixyz for us it was just me assigning the job to the wrong queue simple as that. Unfortunately we had some trouble updating Heroku so deploys would be listed as successful even though it failed in CI. Took me a while to figure out why the code changes didn't "take". We have seen a couple of incidents after but I honestly have no idea. The only thing I've been able to think of is that Sidekiq simply picks the wrong job.
from sidekiq-unique-jobs.
we have the same problem , I hope this helps...
NoMethodError - undefined method get_sidekiq_options' for "DataTAWorker":String: /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sidekiq-unique-jobs-2.3.2/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:11:in
call'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:124:in block in invoke' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:127:in
call'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:127:in invoke' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:159:in
process_single'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:47:in push' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:86:in
push'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:260:in block (2 levels) in retry' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:257:in
map'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:257:in block in retry' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/connection_pool-1.2.0/lib/connection_pool.rb:55:in
with'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq.rb:67:in redis' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:252:in
retry'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:301:in call' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:301:in
block (2 levels) in each'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:300:in each' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:300:in
block in each'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:292:in loop' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:292:in
each'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:401:in retry_all' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/web.rb:162:in
block in class:Web'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:1593:in call' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:1593:in
block in compile!'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:957:in []' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:957:in
block (3 levels) in route!'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:976:in `route_eval'
from sidekiq-unique-jobs.
@felixyz Hey guys, here's the monkey patch i did on our repos.
module SidekiqUniqueJobs
module Middleware
module Client
class UniqueJobs
def unique_enabled?
if worker_class.respond_to? :get_sidekiq_options
worker_class.get_sidekiq_options['unique'] || item['unique']
else
false
end
end
def unique_job_expiration
if worker_class.respond_to? :get_sidekiq_options
worker_class.get_sidekiq_options['unique_job_expiration']
else
SidekiqUniqueJobs::Config.default_expiration
end
end
end
end
end
end
from sidekiq-unique-jobs.
thx a lot.... When do u think u can merge it to master?
from sidekiq-unique-jobs.
we had problems using monkey patch in the new sidekiq 3.0, for now, we are doing this in sidetiq
def perform
uniq = Hash.new
Sidekiq::DeadSet.new.each {|job|
md5_arguments = {class: job['class'], queue: job['queue'], args: job['args']}
digest = Digest::MD5.hexdigest(Sidekiq.dump_json(md5_arguments))
if uniq[digest]
job.delete
else
uniq[digest] = job['jid']
end
}
uniq = Hash.new
Sidekiq::RetrySet.new.each {|job|
md5_arguments = {class: job['class'], queue: job['queue'], args: job['args']}
digest = Digest::MD5.hexdigest(Sidekiq.dump_json(md5_arguments))
if uniq[digest]
job.delete
else
uniq[digest] = job['jid']
end
}
end
from sidekiq-unique-jobs.
Any updates on this issue?
2014-08-05T18:54:21.401Z 14837 TID-ox8y02exc ERROR: undefined method `get_sidekiq_options' for "Reporter":String
2014-08-05T18:54:21.401Z 14837 TID-ox8y02exc ERROR: /Users/nunya/.rvm/gems/ruby-2.1.0@reporting-bot/gems/sidekiq-unique-jobs-3.0.2/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:91:in `unique_enabled?'
from sidekiq-unique-jobs.
I've run into this issue too (I'm using sidekiq-unique-jobs 3.0.9). It comes when trying to use Sidekiq::RetrySet.new.retry_all
The problem appears to be that retry_all is inserting the failed jobs back into the execution queue without inflating the worker class (which is reasonable, this is Sidekiq queue maintenance, not Worker work). The problem comes about because sidekiq-unique-jobs intercepts the insertion and is expecting it to be something that has been called with Worker.perform_async, but it's actually just the class name (as a string)
This is why @girak 's monkey patch works. String does not respond to get_sidekiq_options, so we ignore it.
Thinking out loud
I'm not sure about the best way to handle this atm. Ideally we want the sidekiq queue insertion to be consistent, but we need sidekiq-unique-jobs to be able to handle the case where sidekiq-unique-jobs has been inserted into the middleware chain, but the worker class has not been "required". Maybe we need to add more metadata into the hash so sidekiq-unique-jobs can tell whether this job has already been "uniquified"
from sidekiq-unique-jobs.
This should be fixed since some time ago.
from sidekiq-unique-jobs.
Related Issues (20)
- Migrate away from deprecated Redis commands HOT 2
- `unlocked` reflection is never called
- Logging::Middleware.context can include empty string as key when lock is not set in job hash HOT 1
- PRIMED key grows indefinitely when job re-queues itself in `after_unlock` callback HOT 1
- until_and_while_executing and lock_ttl: jobs silently dropped HOT 3
- incompatibility with sidekiq-failures HOT 4
- Deprecation Warning: Use of rpoplpush command in Redis HOT 1
- Redis has deprecated the `rpoplpush`command
- Release new gem version with "fix replace deprecated rpoplpush w/lmove" HOT 3
- `while_executing` with `on_conflict: :reschedule` Reschedule job after job execution HOT 7
- Reaper manager registration is subject to race conditions HOT 1
- Should client middleware also be added to the Sidekiq server config? HOT 2
- After sidekiq shutdown (with running job) lock digest is not pushned back HOT 8
- Returning same job id HOT 7
- Sidekiq 7: Unsupported command argument type: TrueClass HOT 1
- register_reaper_process nx: true crash Sidekiq on startup HOT 2
- uniquejobs:digests sorted set seems to grow forever HOT 3
- should respond to `has_valid_sidekiq_options?` HOT 4
- Compatibility with Sidekiq7 HOT 4
- until_and_while_executing not entering perform method on initial run HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sidekiq-unique-jobs.