On a project that was using IndexQueue we would occasionally see certain indexing jobs completely lock up. Timeouts would keep the workers from getting completely locked up.

Hey all,

We're seeing quite a few jobs in our queue for objects that were destroyed — can we add a check for destroyed? (well, if it's an ActiveRecord instance) before enqueue? Or am I doing something wrong?

Happy to provide a PR for this.

Thanks!

A couple forks have added delay support. I don't fully understand the reasoning or what is causing the need for it. My best assumption is that the worker system is pulling the index job off the queue before the record can be saved to the database.

The provided IndexJob and RemovalJob should be good for most situations but the meat is SessionProxy. Let people use indexing / removal jobs that fit their use case better.

Sidekiq uses threads to handle multiple workers, and the without_proxy helper is not threadsafe because it modifies Sunspot.session. The Sunspot::Queue::Sidekiq job workers can be modified to work safely like this:

    def perform(klass, id)
      object = constantize(klass).find(id)
      base_session = Sunspot.session.session
      base_session.index(object)
    end

Hi

I am doing a request rspec test, in my before all block I created all the testing data. In my "it" block I did "MyClass.reindex", and from the redis monitor I can see nothing happened. Works fine in development.

Any idea?

Thanks

Thanks for building this. Much cleaner than the random gist out there.

Only thing I noticed is it would be cool to be able to configure the queue name to one I already have (critical, high, low). But really no big deal (and if it was that big of a problem I would have submitted a pull request :)

Seriously though, thanks for the gem!

bundle exec sidekiq -c 3 -q critical,3 -q default,2 -q sunspot
uninitialized constant Sunspot::Queue::Helpers
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/sunspot-queue-0.10.0/lib/sunspot/queue/sidekiq/index_job.rb:5:in `<class:IndexJob>'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/sunspot-queue-0.10.0/lib/sunspot/queue/sidekiq/index_job.rb:4:in `<module:Sidekiq>'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/sunspot-queue-0.10.0/lib/sunspot/queue/sidekiq/index_job.rb:3:in `<top (required)>'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/backports-2.6.7/lib/backports/tools.rb:314:in `require'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/backports-2.6.7/lib/backports/tools.rb:314:in `require_with_backports'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/sunspot-queue-0.10.0/lib/sunspot/queue/sidekiq/backend.rb:1:in `<top (required)>'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/backports-2.6.7/lib/backports/tools.rb:314:in `require'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/backports-2.6.7/lib/backports/tools.rb:314:in `require_with_backports'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/sunspot-queue-0.10.0/lib/sunspot/queue/sidekiq.rb:5:in `<module:Sidekiq>'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/sunspot-queue-0.10.0/lib/sunspot/queue/sidekiq.rb:4:in `<module:Queue>'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/sunspot-queue-0.10.0/lib/sunspot/queue/sidekiq.rb:3:in `<top (required)>'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/backports-2.6.7/lib/backports/tools.rb:314:in `require'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/backports-2.6.7/lib/backports/tools.rb:314:in `require_with_backports'
/Users/dickeyxxx/src/production_beast/config/initializers/sunspot.rb:1:in `<top (required)>'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/bundler/gems/rails-dev-boost-3e25b06c5c53/lib/rails_development_boost/loadable_patch.rb:14:in `load'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/railties-3.2.13/lib/rails/engine.rb:588:in `block (2 levels) in <class:Engine>'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/railties-3.2.13/lib/rails/engine.rb:587:in `each'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/railties-3.2.13/lib/rails/engine.rb:587:in `block in <class:Engine>'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/railties-3.2.13/lib/rails/initializable.rb:30:in `instance_exec'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/railties-3.2.13/lib/rails/initializable.rb:30:in `run'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/railties-3.2.13/lib/rails/initializable.rb:55:in `block in run_initializers'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/railties-3.2.13/lib/rails/initializable.rb:54:in `each'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/railties-3.2.13/lib/rails/initializable.rb:54:in `run_initializers'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/railties-3.2.13/lib/rails/application.rb:136:in `initialize!'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/railties-3.2.13/lib/rails/railtie/configurable.rb:30:in `method_missing'
/Users/dickeyxxx/src/production_beast/config/environment.rb:5:in `<top (required)>'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/sidekiq-2.12.0/lib/sidekiq/cli.rb:197:in `require'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/sidekiq-2.12.0/lib/sidekiq/cli.rb:197:in `boot_system'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/sidekiq-2.12.0/lib/sidekiq/cli.rb:39:in `parse'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/sidekiq-2.12.0/bin/sidekiq:7:in `<top (required)>'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/bin/sidekiq:23:in `load'
/Users/dickeyxxx/.rbenv/versions/2.0.0-p195/bin/sidekiq:23:in `<main>'

adding a require to my initializer seems to fix the problem though:

require "sunspot/queue/helpers"
require "sunspot/queue/sidekiq"
backend = Sunspot::Queue::Sidekiq::Backend.new
Sunspot.session = Sunspot::Queue::SessionProxy.new(Sunspot.session, backend)

Temporary work around:

Sunspot::Queue::Resque::IndexJob.class_eval do
  def self.queue
    "low"
  end
end

Sunspot::Queue::Resque::RemovalJob.class_eval do
  def self.queue
    "low"
  end
end

Would love to have support for ActiveJob. I'd be willing to submit a PR or create a separate gem for it.

When I try to run a task (in this case via Account.index) every single task fails with this, I have no idea why it is loading the controllers in my app to do this. The methods sort_column, fetch_people, and people that it is trying to run are all involved in fetching json for something else, is it trying to read json from the index action or something?

/Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/activemodel-3.2.12/lib/active_model/attribute_methods.rb:407:in method_missing' /Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/activerecord-3.2.12/lib/active_record/attribute_methods.rb:149:inmethod_missing'
/Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/util.rb:241:in __proxy_method__' /Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/util.rb:236:inmethod_missing'
/Users/wazoo/projects/serenity/app/controllers/people_controller.rb:241:in sort_column' /Users/wazoo/projects/serenity/app/controllers/people_controller.rb:222:infetch_people'
/Users/wazoo/projects/serenity/app/controllers/people_controller.rb:207:in people' /Users/wazoo/projects/serenity/app/models/account.rb:66:inblock (2 levels) in class:Account'
/Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/util.rb:208:in instance_eval' /Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/util.rb:208:ininstance_eval_with_context'
/Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/util.rb:86:in instance_eval_or_call' /Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/data_extractor.rb:33:invalue_for'
/Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/field_factory.rb:58:in populate_document' /Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/indexer.rb:95:inblock in prepare'
/Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/indexer.rb:94:in each' /Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/indexer.rb:94:inprepare'
/Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/indexer.rb:24:in block in add' /Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/indexer.rb:24:inmap'
/Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/indexer.rb:24:in add' /Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/session.rb:91:inindex'
/Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot/session_proxy/abstract_session_proxy.rb:11:in index' /Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-1.3.3/lib/sunspot.rb:184:inindex'
/Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot_rails-1.3.3/lib/sunspot/rails/searchable.rb:372:in solr_index' /Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-queue-0.10.0/lib/sunspot/queue/resque/index_job.rb:13:inblock in perform'
/Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-queue-0.10.0/lib/sunspot/queue/helpers.rb:15:in without_proxy' /Users/wazoo/.rvm/gems/ruby-1.9.3-p362@serenity/gems/sunspot-queue-0.10.0/lib/sunspot/queue/resque/index_job.rb:12:inperform'

I have looked for everything that I could possibly think of to figure this error out but I cant find anything.

Hey,

Loving sunspot-queue, not loving how recent changes (in particular ReneB's compatibility restore aren't yet available on rubygems. Any chance you could publish a new version? Life get so much easier if gems are available in the default source.

Ta!

Any other possible manner to use this gem for sunspot_rails 1.1.0 ?

Solr processes updates most efficiently when it's receiving documents in batches at a relatively low concurrency. Highly parallel single document updates are probably one of the easiest ways to starve the CPU and send Solr into a GC death spiral.

Some support for limiting the concurrency would be welcome. Buffering updates into batches would be even better. The implementations here may vary in difficulty (or possibility) based on the background job system in question.

For example, in Sidekiq, a simple Mutex could do the trick. That's the easiest use case here to limit concurrency to one Solr update job per Sidekiq process, though it's a bit harsh. Its easy baked in support for connection pooling can help loosen that up a bit. Better still, individual jobs could append to an in-memory Queue (or re-enqueued back into another Redis list), which can then be processed in batches by a single Sunspot worker thread forked from the class.

We see this particular use case all the time over at websolr.com, so I'd be happy to help support some efforts to get nicer batched queuing out into the real world for Sunspot users. This gem looks like the best bet for that! I don't have time right this second to contribute code, but I can certainly answer questions and help sketch out the requirements and design.