undefined method `get_sidekiq_options' for "MyScheduledWorker":String about sidekiq-unique-jobs HOT 17 CLOSED

@girak could you point me to where that job is re-enqueued in sidekiq? I'm pretty sure we could mimic that behaviour.

from sidekiq-unique-jobs.

@mhenrixon https://github.com/mperham/sidekiq/blob/d40f7cc6af61ef9046c34429bef2932ca85f889f/lib/sidekiq/client.rb#L182

anywhere that refers get_sidekiq_options

from sidekiq-unique-jobs.

Ok so this happened to me as well now that I split up our backend processing in multiple different projects. I will see if I can get this solved somehow. First a failing test case.

from sidekiq-unique-jobs.

For us it looks like the error is with Sidekiq. Could you provide a stacktrace?

from sidekiq-unique-jobs.

I've got a quirky little RSpec spec here: https://gist.github.com/Felixyz/8781057
It test drives (synchronously) the Sidekiq retry mechanism with sidekiq-unique-jobs inserted in the middleware chain and attempts to mimic the setup we have, with two queues. The script does not reproduce the bug, from which I draw the conclusion that the bug (if it is a bug) is probably in Sidekiq's higher-level coordination classes, like Manager, Fetcher or Launcher.

Whatever it is, it's perhaps not so much a bug as an uncompatibility in sidekiq-unique-job's handling of the parameters? By which I mean: this is probably not an issue that we would report with Sidekiq itself, right?

from sidekiq-unique-jobs.

@girak Would you care to share your patch?

from sidekiq-unique-jobs.

One more observation that might be useful: we're getting fewer of these errors every day. My impression is that they might occur after a restart of the Sidekiq processes. We are running Sidekiq Pro. Are others who have been seeing this problem running Sidekiq or Sidekiq Pro?

from sidekiq-unique-jobs.

Yeah so @felixyz is right and that's why I was asking for a stacktrace. @girak I can't replicate that sidekiq unique jobs should be the problem in this situation. The crash actually happens in Sidekiq and the reason it happens in Sidekiq for us is because the wrong server was processing jobs it could not handle (we assigned it to the wrong queue).

from sidekiq-unique-jobs.

@mhenrixon I'm not sure you can get a great stack trace because of the async nature of Sidekiq, but (as you probably know) the line that causes the exception is:
/app/vendor/bundle/ruby/2.0.0/gems/sidekiq-unique-jobs-2.6.7/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:11:in `call'

In any case: yes, we also concluded that a worker was assigned a job from the wrong queue. However, I haven't been able to track down the reason why this happens. Any tips for that?
(The only thing I can think of is that our two kinds of workers inherit from a common base class, which is the one that mixes in Sidekiq::Worker.)

from sidekiq-unique-jobs.

@felixyz for us it was just me assigning the job to the wrong queue simple as that. Unfortunately we had some trouble updating Heroku so deploys would be listed as successful even though it failed in CI. Took me a while to figure out why the code changes didn't "take". We have seen a couple of incidents after but I honestly have no idea. The only thing I've been able to think of is that Sidekiq simply picks the wrong job.

from sidekiq-unique-jobs.

we have the same problem , I hope this helps...

NoMethodError - undefined method get_sidekiq_options' for "DataTAWorker":String: /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sidekiq-unique-jobs-2.3.2/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:11:incall'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:124:in block in invoke' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:127:incall'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:127:in invoke' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:159:inprocess_single'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:47:in push' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:86:inpush'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:260:in block (2 levels) in retry' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:257:inmap'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:257:in block in retry' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/connection_pool-1.2.0/lib/connection_pool.rb:55:inwith'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq.rb:67:in redis' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:252:inretry'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:301:in call' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:301:inblock (2 levels) in each'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:300:in each' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:300:inblock in each'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:292:in loop' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:292:ineach'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:401:in retry_all' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/web.rb:162:inblock in class:Web'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:1593:in call' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:1593:inblock in compile!'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:957:in []' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:957:inblock (3 levels) in route!'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:976:in `route_eval'

from sidekiq-unique-jobs.

@felixyz Hey guys, here's the monkey patch i did on our repos.

module SidekiqUniqueJobs
  module Middleware
    module Client
      class UniqueJobs
        def unique_enabled?
          if worker_class.respond_to? :get_sidekiq_options
            worker_class.get_sidekiq_options['unique'] || item['unique']
          else
            false
          end
        end

        def unique_job_expiration
          if worker_class.respond_to? :get_sidekiq_options
            worker_class.get_sidekiq_options['unique_job_expiration']
          else
            SidekiqUniqueJobs::Config.default_expiration
          end
        end
      end
    end
  end
end

from sidekiq-unique-jobs.

thx a lot.... When do u think u can merge it to master?

from sidekiq-unique-jobs.

we had problems using monkey patch in the new sidekiq 3.0, for now, we are doing this in sidetiq

def perform

uniq = Hash.new
Sidekiq::DeadSet.new.each {|job|
  md5_arguments = {class: job['class'], queue: job['queue'], args: job['args']}
  digest = Digest::MD5.hexdigest(Sidekiq.dump_json(md5_arguments))

  if uniq[digest]
    job.delete
  else
    uniq[digest] = job['jid']
  end

}

uniq = Hash.new
Sidekiq::RetrySet.new.each {|job|
  md5_arguments = {class: job['class'], queue: job['queue'], args: job['args']}
  digest = Digest::MD5.hexdigest(Sidekiq.dump_json(md5_arguments))

  if uniq[digest]
    job.delete
  else
    uniq[digest] = job['jid']
  end

}

end

from sidekiq-unique-jobs.

Any updates on this issue?

2014-08-05T18:54:21.401Z 14837 TID-ox8y02exc ERROR: undefined method `get_sidekiq_options' for "Reporter":String
2014-08-05T18:54:21.401Z 14837 TID-ox8y02exc ERROR: /Users/nunya/.rvm/gems/ruby-2.1.0@reporting-bot/gems/sidekiq-unique-jobs-3.0.2/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:91:in `unique_enabled?'

from sidekiq-unique-jobs.

I've run into this issue too (I'm using sidekiq-unique-jobs 3.0.9). It comes when trying to use Sidekiq::RetrySet.new.retry_all
The problem appears to be that retry_all is inserting the failed jobs back into the execution queue without inflating the worker class (which is reasonable, this is Sidekiq queue maintenance, not Worker work). The problem comes about because sidekiq-unique-jobs intercepts the insertion and is expecting it to be something that has been called with Worker.perform_async, but it's actually just the class name (as a string)

This is why @girak 's monkey patch works. String does not respond to get_sidekiq_options, so we ignore it.

Thinking out loud
I'm not sure about the best way to handle this atm. Ideally we want the sidekiq queue insertion to be consistent, but we need sidekiq-unique-jobs to be able to handle the case where sidekiq-unique-jobs has been inserted into the middleware chain, but the worker class has not been "required". Maybe we need to add more metadata into the hash so sidekiq-unique-jobs can tell whether this job has already been "uniquified"

from sidekiq-unique-jobs.

This should be fixed since some time ago.

from sidekiq-unique-jobs.