pat / ts-delayed-delta Goto Github PK

View Code? Open in Web Editor NEW

73.0 7.0 33.0 100 KB

Manage delta indexes via Delayed Job for Thinking Sphinx

Home Page: http://ts.freelancing-gods.com

License: MIT License

Ruby 100.00%

ts-delayed-delta's Introduction

Delayed Deltas for Thinking Sphinx

Installation

This version works with Thinking Sphinx v1.5.0, v2.1.0 and v3.×.

In your Gemfile, it’ll likely look something like this:

gem 'ts-delayed-delta', '~> 2.1'

Or, if you’re referring directly to the git repository:

gem 'ts-delayed-delta',
  :git    => 'git://github.com/pat/ts-delayed-delta.git',
  :branch => 'master',
  :ref    => 'use-latest-commit-here'

And given this gem’s built for Delayed Job, we’re presuming you’ve got that set up (with the database table), and you know how to run it. This gem no longer provides a custom task for that purpose – there’s no need.

Usage

In your index definitions, make sure you set the :delta setting to ThinkingSphinx::Deltas::DelayedDelta:

ThinkingSphinx::Index.define :book, :with => :active_record, :delta => ThinkingSphinx::Deltas::DelayedDelta do
  # ...
end

For those still on pre-v3 versions of Thinking Sphinx, the delta option goes within the define_index block:

define_index do
  # ...

  set_property :delta => ThinkingSphinx::Deltas::DelayedDelta
end

If you’ve never used delta indices before, you’ll want to add the boolean column named delta to each model that is using the approach. It probably doesn’t hurt to add an index to that column as well.

add_column :books, :delta, :boolean, :default => true, :null => false
add_index  :books, :delta

And that’s pretty much it. Make sure Delayed Job is running, and it’ll take care of the jobs to keep your Sphinx indices up to date.

Please keep in mind that you need to run the Delayed Job workers on the same machine as Sphinx, as direct access to Sphinx files (and the Sphinx command-line tools) is essential.

Contributing

Contributions are very much welcome – but keep in mind the following:

Keep patches in a separate branch
Don’t mess with the version number. I’ll take care of that when the patch is merged in.
Please write tests – currently, there’s only acceptance tests in place, because the underlying code is simple enough. If you get stuck into anything complex, unit tests are a smart move.

Contributors

Ryan Schlesinger (Allowing installs as a plugin)
Maximilian Schulz (Ensuring compatibility with Bundler)
Edgars Beigarts (Adding intelligent description for tasks)
Alexander Simonov (Explicit table definition)
David Goodlad (Delayed Job/ActiveRecord load order fix)
Ben Hutton (Delayed Job compatibility update)
Reinier de Lange (Fix for table name reference)
Enrico Brunetta (Adding Railtie for Rails 3)
Jonathan Viney and James Healy (Rails 3.2 deprecation fixes)
Georg Ledermann (DJ 2.0.x backwards compatibility)
Jonathan Lim (Cleaning up version references)
Manjunath Manohar (Allow setting of job queue, smarter cleaning up of jobs)
Mikko Koski (Ensure new jobs are created if all existing jobs are failures)
Jason Barnabe (Fix Rails 5.x compatibility)

Copyright

Copyright © 2009-2018, Thinking Sphinx Delayed Deltas (ts-delayed-delta) is developed and maintained by Pat Allan, and is released under the open MIT Licence. Many thanks to all who have contributed patches.

ts-delayed-delta's People

Contributors

Stargazers

Watchers

ts-delayed-delta's Issues

Necessary to add 'reconnect: true' to thinking_sphinx.yml

Something to do with delayed_job & rails & daemon & thinking_sphinx results in random but very frequent (900k total count over two weeks) "MySQL server has gone away" errors in FlagAsDeletedJob since I upgraded to TS3.

Adding the following to thinking_sphinx.yml worked around this issue:

production:
  # ...
  connection_options:
    reconnect: true

Job succeeds but delta remains true

This is from my delayed_job log

I keep getting this line in my log file even when I don't modify anything.

2012-06-21T16:45:18+0300: [Worker(delayed_job host:abdo-ubuntu pid:14167)] 5 jobs processed at 13.9375 j/s, 0 failed ...

I'm using rails 3.2.6, ruby 1.9.3 p194 , delayed_job 3.0.3, delayed_job_active_record 0.3.2, and ts-delayed-delta 1.1.3

The delta column remains true after the delayed job succeeds.

Any idea?

Not indexing updates after upgrading gem

I upgraded to 2.0.2 and since then, it looks like updates stop getting indexed after a while (maybe after a deploy?).

[Mon Mar 26 00:20:23.625 2018] [19628] caught SIGHUP (seamless=1, in queue=1)
[Mon Mar 26 00:20:23.639 2018] [19628] rotating index 'script_delta': started
[Mon Mar 26 00:20:23.646 2018] [19628] rotating index 'script_delta': success
[Mon Mar 26 00:20:23.646 2018] [19628] rotating index 'script_core': started
[Mon Mar 26 00:20:23.656 2018] [19628] rotating index 'script_core': success
[Mon Mar 26 00:20:23.658 2018] [19628] rotating index: all indexes done

And nothing in production.searchd.log since then.

I was previously on 0aef219, so there shouldn't be any changes when going to 2.0.2, but maybe the update of riddle from 2.2.0 to 2.3.0 is causing the issue?

Gems for edge branch

Hey!

I had an issue earlier where stuff broke because my Gemfile has thinking-sphinx locked to a Gem version, but as ts-delayed-delta tracking Github.

I've now locked ts-delayed-delta to a ref in my Gemfile to get round this, but I'm wondering if/when you're thinking of releasing the edge branch as actual gems, and whether there is any help you need to get there?

Thanks!

Error when run rake ts:dd

Hi folks,

I got this error when i run rake ts:dd in development environment.

Invoke ts:dd (first_time)
Invoke thinking_sphinx:delayed_delta (first_time)
Invoke thinking_sphinx:app_env (first_time)
Execute thinking_sphinx:app_env
Invoke environment (first_time)
Execute environment
Execute thinking_sphinx:delayed_delta
Starting job worker host:MacBook-de-Ricardo-Domingos.local pid:11614
rake aborted!
You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.join
/Library/Ruby/Gems/1.8/gems/ts-delayed-delta-1.1.0/lib/thinking_sphinx/deltas/delayed_delta/delta_job.rb:17:in display_name' /Library/Ruby/Gems/1.8/gems/delayed_job-1.8.4/lib/delayed/job.rb:54:inname'
/Library/Ruby/Gems/1.8/gems/delayed_job-1.8.4/lib/delayed/job.rb:84:in run_with_lock' /Library/Ruby/Gems/1.8/gems/delayed_job-1.8.4/lib/delayed/job.rb:153:inreserve_and_run_one_job'
/Library/Ruby/Gems/1.8/gems/delayed_job-1.8.4/lib/delayed/job.rb:152:in each' /Library/Ruby/Gems/1.8/gems/delayed_job-1.8.4/lib/delayed/job.rb:152:inreserve_and_run_one_job'
/Library/Ruby/Gems/1.8/gems/delayed_job-1.8.4/lib/delayed/job.rb:199:in work_off' /Library/Ruby/Gems/1.8/gems/delayed_job-1.8.4/lib/delayed/job.rb:198:intimes'
/Library/Ruby/Gems/1.8/gems/delayed_job-1.8.4/lib/delayed/job.rb:198:in work_off' /Library/Ruby/Gems/1.8/gems/delayed_job-1.8.4/lib/delayed/worker.rb:28:instart'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/benchmark.rb:308:in realtime' /Library/Ruby/Gems/1.8/gems/delayed_job-1.8.4/lib/delayed/worker.rb:27:instart'
/Library/Ruby/Gems/1.8/gems/delayed_job-1.8.4/lib/delayed/worker.rb:24:in loop' /Library/Ruby/Gems/1.8/gems/delayed_job-1.8.4/lib/delayed/worker.rb:24:instart'
/Library/Ruby/Gems/1.8/gems/ts-delayed-delta-1.1.0/lib/thinking_sphinx/deltas/delayed_delta/tasks.rb:11
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in call' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:inexecute'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in each' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:inexecute'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:597:in invoke_with_call_chain' /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/monitor.rb:242:insynchronize'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:in invoke_with_call_chain' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:607:ininvoke_prerequisites'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:604:in each' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:604:ininvoke_prerequisites'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:596:in invoke_with_call_chain' /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/monitor.rb:242:insynchronize'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:in invoke_with_call_chain' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:583:ininvoke'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2051:in invoke_task' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:intop_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in each' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:intop_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in standard_exception_handling' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2023:intop_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2001:in run' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:instandard_exception_handling'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in run' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/bin/rake:31 /usr/bin/rake:19:inload'
/usr/bin/rake:19

I using Sphinx 0.9.8-rc2 (r1234) and I setup in my environment.rb:

config.gem 'delayed_job', :version => '1.8.4'
config.gem 'thinking-sphinx', :version => '1.3.18', :lib => 'thinking_sphinx'
config.gem 'ts-delayed-delta', :version => '1.1.0', :lib => 'thinking_sphinx/deltas/delayed_delta', :source => 'http://gemcutter.org'

Somebody knows what's happend? It's problem with gems version?

thanks

delta flag has not set back to false after re-index

Hi Pat,
In our Project, there are 30,000 records in a table.

Suppose we are updating some columns in this table for 9000 existing records and running delta index.
so all these 9000 records have been set delta as true for indexing.

As you are saying in this case #37, delta will not set to false after re-indexing.(Actullay this re-indexing is taking 30 min), so after re-indexing, if user creates a new record in this table, sphinx will rotate all 9001 records instead of 1 record.(Again re-index will take 30 min instead of milliseconds)
can you help me on this issue.

ActiveRecord::ConnectionNotEstablished with TS

Hello,

I'm working on a new project using last version of Thinking Sphinx, which works pretty well.
But I've planned to use delayed delta as well with the ts-delayed-delta gem and I got the following exception when I try to lauch a rake task or the rails server.

ActiveRecord::ConnectionNotEstablished (ActiveRecord::ConnectionNotEstablished)

Here is the full gist : https://gist.github.com/956698

I'm using:

gem 'thinking-sphinx',
  :git     => 'git://github.com/freelancing-god/thinking-sphinx.git',
  :branch  => 'rails3'
gem 'ts-delayed-delta', '1.1.2',
  :require => 'thinking_sphinx/deltas/delayed_delta'
gem 'delayed_job', "~> 2.1.4"

Am I wrong with my gem versions or is it a bug ?

Many thanks

Release 1.1.3 breaks compatibility with Rails 2.3

Today's release 1.1.3 includes a commit (ae47c81) which breaks compatibility with DelayedJob 2.0.7 (which is needed for using it with Rails 2.3).

It's a small thing about setting priority as integer vs. hash.

private method `warn' called for nil:NilClass

My config is

# Gemfile
gem 'delayed_job'
gem 'thinking-sphinx'
gem 'ts-delayed-delta', :require => 'thinking_sphinx/deltas/delayed_delta'

# Rakefile
require 'thinking_sphinx/deltas/delayed_delta/tasks'

# config/environment.rb
ThinkingSphinx.suppress_delta_output = true

# define index in model
define_index do
    indexes title

    # Set Delta Indexes
    set_property :delta => true
    set_property :delta => :delayed
end

and error is

$ rake ts:index --trace
rake aborted!
private method `warn' called for nil:NilClass
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.1.3/lib/active_record/named_scope.rb:194:in `valid_scope_name?'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.1.3/lib/active_record/named_scope.rb:175:in `scope'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/kaminari-0.12.4/lib/kaminari/models/active_record_extension.rb:15:in `block in inherited'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/kaminari-0.12.4/lib/kaminari/models/active_record_extension.rb:10:in `class_eval'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/kaminari-0.12.4/lib/kaminari/models/active_record_extension.rb:10:in `inherited'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/ts-delayed-delta-1.1.2/lib/thinking_sphinx/deltas/delayed_delta/job.rb:16:in `<top (required)>'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/activesupport-3.1.3/lib/active_support/dependencies.rb:240:in `require'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/activesupport-3.1.3/lib/active_support/dependencies.rb:240:in `block in require'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/activesupport-3.1.3/lib/active_support/dependencies.rb:223:in `block in load_dependency'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/activesupport-3.1.3/lib/active_support/dependencies.rb:640:in `new_constants_in'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/activesupport-3.1.3/lib/active_support/dependencies.rb:223:in `load_dependency'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/activesupport-3.1.3/lib/active_support/dependencies.rb:240:in `require'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/ts-delayed-delta-1.1.2/lib/thinking_sphinx/deltas/delayed_delta.rb:6:in `<top (required)>'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/bundler-1.0.21/lib/bundler/runtime.rb:68:in `require'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/bundler-1.0.21/lib/bundler/runtime.rb:68:in `block (2 levels) in require'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/bundler-1.0.21/lib/bundler/runtime.rb:66:in `each'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/bundler-1.0.21/lib/bundler/runtime.rb:66:in `block in require'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/bundler-1.0.21/lib/bundler/runtime.rb:55:in `each'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/bundler-1.0.21/lib/bundler/runtime.rb:55:in `require'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/bundler-1.0.21/lib/bundler.rb:122:in `require'
/Users/phuwanart/Works/user_mw3/config/application.rb:7:in `<top (required)>'
/Users/phuwanart/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
/Users/phuwanart/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
/Users/phuwanart/Works/user_mw3/Rakefile:5:in `<top (required)>'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/rake-0.9.2.2/lib/rake/rake_module.rb:25:in `load'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/rake-0.9.2.2/lib/rake/rake_module.rb:25:in `load_rakefile'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/rake-0.9.2.2/lib/rake/application.rb:501:in `raw_load_rakefile'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/rake-0.9.2.2/lib/rake/application.rb:82:in `block in load_rakefile'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/rake-0.9.2.2/lib/rake/application.rb:81:in `load_rakefile'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/rake-0.9.2.2/lib/rake/application.rb:65:in `block in run'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/rake-0.9.2.2/lib/rake/application.rb:63:in `run'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/gems/rake-0.9.2.2/bin/rake:33:in `<top (required)>'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/bin/rake:19:in `load'
/Users/phuwanart/.rvm/gems/ruby-1.9.2-p290/bin/rake:19:in `<main>'

please suggest me to fix this problem.

Seems to be broken with DJ 2.0.1

^Dlanghorst-3586:PleaseDB langhorst$ script/generate delayed_job
/Library/Ruby/Gems/1.8/gems/rails-2.3.5/lib/rails/gem_dependency.rb:119:Warning: Gem::Dependency#version_requirements is deprecated and will be removed on or after August 2010. Use #requirement
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.5/lib/active_support/dependencies.rb:440:in load_missing_constant': uninitialized constant Delayed::Job (NameError) from /Library/Ruby/Gems/1.8/gems/activesupport-2.3.5/lib/active_support/dependencies.rb:80:inconst_missing'
from /Library/Ruby/Gems/1.8/gems/ts-delayed-delta-1.0.4/lib/thinking_sphinx/deltas/delayed_delta/job.rb:7
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in gem_original_require' from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:inrequire'
from /Library/Ruby/Gems/1.8/gems/activesupport-2.3.5/lib/active_support/dependencies.rb:156:in require' from /Library/Ruby/Gems/1.8/gems/activesupport-2.3.5/lib/active_support/dependencies.rb:521:innew_constants_in'
from /Library/Ruby/Gems/1.8/gems/activesupport-2.3.5/lib/active_support/dependencies.rb:156:in require' from /Library/Ruby/Gems/1.8/gems/ts-delayed-delta-1.0.4/lib/thinking_sphinx/deltas/delayed_delta.rb:5 from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:ingem_original_require'
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in require' from /Library/Ruby/Gems/1.8/gems/activesupport-2.3.5/lib/active_support/dependencies.rb:156:inrequire'
from /Library/Ruby/Gems/1.8/gems/activesupport-2.3.5/lib/active_support/dependencies.rb:521:in new_constants_in' from /Library/Ruby/Gems/1.8/gems/activesupport-2.3.5/lib/active_support/dependencies.rb:156:inrequire'
from /Library/Ruby/Gems/1.8/gems/rails-2.3.5/lib/rails/gem_dependency.rb:208:in load' from /Library/Ruby/Gems/1.8/gems/rails-2.3.5/lib/initializer.rb:307:inload_gems'
from /Library/Ruby/Gems/1.8/gems/rails-2.3.5/lib/initializer.rb:307:in each' from /Library/Ruby/Gems/1.8/gems/rails-2.3.5/lib/initializer.rb:307:inload_gems'
from /Library/Ruby/Gems/1.8/gems/rails-2.3.5/lib/initializer.rb:164:in process' from /Library/Ruby/Gems/1.8/gems/rails-2.3.5/lib/initializer.rb:113:insend'
from /Library/Ruby/Gems/1.8/gems/rails-2.3.5/lib/initializer.rb:113:in run' from /Users/langhorst/Documents/PolBase/source/config/environment.rb:13 from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:ingem_original_require'
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in require' from /Library/Ruby/Gems/1.8/gems/rails-2.3.5/lib/commands/generate.rb:1 from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:ingem_original_require'
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `require'
from script/generate:3

Comma missing in README.textile

Should be

config.gem 'ts-delayed-delta',
:lib => 'thinking_sphinx/deltas/delayed_delta',
:version => '>= 1.0.0',
:source => 'http://gemcutter.org'

Instead of

config.gem 'ts-delayed-delta',
:lib => 'thinking_sphinx/deltas/delayed_delta'
:version => '>= 1.0.0',
:source => 'http://gemcutter.org'

Delta flag not resetting

I'm having a strange issue. I'm using ts-delayed-delta 2.0.2 and thinking-sphinx 3.1.1 with activerecord and delayed_job, and as far as I can tell the delta indices are being processed and returning correct results, but for some reason the delta boolean flag on my indexed model is staying set to true rather than resetting to false after the delta is processed.

Here's my index:

ThinkingSphinx::Index.define :page, :with => :active_record, :delta => ThinkingSphinx::Deltas::DelayedDelta do

indexes :title, as: :page_title
indexes :menu_cat, as: :page_menu_cat
indexes :menu_title, as: :page_menu_title
indexes :site_cat, as: :page_site_cat
indexes :sub_cat, as: :page_sub_cat
indexes :page_url, as: :page_page_url
indexes :content, as: :page_content
indexes :updated_by, as: :page_updated_by

has :id, as: :page_id
has :updated_at, :created_at

set_property :enable_star => 1
set_property :min_infix_len => 3

end

Here's the output of delayed_job.log after making a small edit to a page:

2014-08-21T11:20:34-0400: [Worker(delayed_job host:gm157-bec.broadinstitute.org pid:91282)] Job Thinking Sphinx: Process page_delta (id=17) RUNNING
2014-08-21T11:20:34-0400: [Worker(delayed_job host:gm157-bec.broadinstitute.org pid:91282)] Job Thinking Sphinx: Process page_delta (id=17) COMPLETED after 0.0635
2014-08-21T11:20:34-0400: [Worker(delayed_job host:gm157-bec.broadinstitute.org pid:91282)] Job Thinking Sphinx: Mark 26 in page_core as deleted (id=18) RUNNING
2014-08-21T11:20:34-0400: [Worker(delayed_job host:gm157-bec.broadinstitute.org pid:91282)] Job Thinking Sphinx: Mark 26 in page_core as deleted (id=18) COMPLETED after 0.0366
2014-08-21T11:20:34-0400: [Worker(delayed_job host:gm157-bec.broadinstitute.org pid:91282)] 2 jobs processed at 7.8692 j/s, 0 failed

And development.searchd.log:

[Thu Aug 21 11:20:34.359 2014] [91272] caught SIGHUP (seamless=1, in queue=1)
[Thu Aug 21 11:20:34.396 2014] [91272] rotating index 'page_delta': started
[Thu Aug 21 11:20:34.405 2014] [91272] rotating index 'page_delta': success
[Thu Aug 21 11:20:34.405 2014] [91272] rotating index: all indexes done

This behavior is being replicated across all applications I'm running with TS and ts-delayed-delta right now. It worked fine on TS2 but since the upgrade to TS3 I've had this issue. Any thoughts?

Rails 3: DEPRECATION WARNING: Base.named_scope has been deprecated, please use Base.scope instead

When ts-delayed-delta is added to your Gemfile you'll get "DEPRECATION WARNING: Base.named_scope has been deprecated, please use Base.scope instead. (called from /rails_app/config/application.rb:7)" warnings whenever you run a rake command. Line 7 is "Bundler.require(:default, Rails.env) if defined?(Bundler)", thus boilerplate rails code.

This is happening on a brand new Rails app. Here's my Gemfile for reference: http://gist.github.com/557786

I tried forking this repo, update the gem dependencies (if that might have been the cause) and then installing my forked gem locally - but to no avail. Still this deprecation warning.

I've also deleted all previous local ts and ts-delta gems from my computer and just installed using a fresh bundle install - still deprec issues.

Why is this happening?

Delta column stays false

After automaticaly rotating an index with delayed_job, the delta column stays false. I don't think that's normal. In my config file, i have :

source purchase_order_delta_0 : purchase_order_core_0
{
type = pgsql
sql_host = localhost
sql_user = xxx
sql_pass = xxx
sql_db = xxx_development
sql_query_pre =
sql_query = SELECT "purchase_orders"."id" * 13 + 7 AS "id" , "purchase_orders"."reference"$
sql_query_range = SELECT COALESCE(MIN("id"), 1::bigint), COALESCE(MAX("id"), 1::bigint) FR$
sql_attr_uint = sphinx_internal_id
sql_attr_uint = class_crc
sql_attr_uint = sphinx_deleted
sql_attr_multi = uint subclass_crcs from field
sql_query_info = SELECT * FROM "purchase_orders" WHERE "id" = (($id - 7) / 13)
}

The "sql_query_pre" is empty.
I think it should be

UPDATE "purchase_orders" SET "delta" = FALSE WHERE "delta" = TRUE

like in the purchase_order_core.

Delta index runs even if object was not persisted

I just exposed some APIs to my app, and discovered this problem. When I invoke my API to create a new record, i.e.

business.customers.create(:first_name => 'foo', ...)

and there is a validation error (i.e. email is already taken). I get an exception on my log:

Started POST "/api/customers" for 127.0.0.1 at 2012-01-29 14:10:49 -0600
  User Load (0.7ms)  SELECT `users`.* FROM `users` WHERE `users`.`authentication_token` = 'vZSDZumq7doM4tgxoTdSHHXR3m1Pqs8C' AND `users`.`current_state` IS NULL LIMIT 1
  Business Load (0.6ms)  SELECT `businesses`.* FROM `businesses` WHERE `businesses`.`id` = 1 LIMIT 1
   (0.1ms)  BEGIN
   (0.3ms)  SELECT 1 FROM `customers` WHERE (LOWER(`customers`.`email`) = LOWER('[email protected]') AND `customers`.`business_id` = 1) LIMIT 1
   (0.1ms)  COMMIT
     (0.4ms)  SELECT COUNT(*) FROM `delayed_jobs` WHERE `delayed_jobs`.`handler` = '--- !ruby/object:ThinkingSphinx::Deltas::DeltaJob \nindices: \n- customer_delta\n' AND `delayed_jobs`.`locked_at` IS NULL
You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.*

I believe this is due to the fact that delayed delta checks whether the delta column has been toggled(which in my case was true), but not whether the object has been persisted.

I wanted to provide a patch, but I'm not sure where to add the check: I found 2 places that look like they need it:

delta.rb: line 45

delayed_delta.rb: line 63-65

am I wrong? any feedback?

Delta Indexes doesn't work

I've done everything as said in readme.
I'm using Rails 3.2.13, ruby 2.0.0p0, thinking-sphinx (3.0.3), ts-delayed-delta (2.0.0)

in DJ log:

2013-06-23T23:22:27+0300: [Worker(delayed_job host:rvm pid:1610)] Thinking Sphinx: Process page_delta completed after 0.1642
2013-06-23T23:22:27+0300: [Worker(delayed_job host:rvm pid:1610)] Thinking Sphinx: Mark 202 in page_core as deleted completed after 0.0242
2013-06-23T23:22:27+0300: [Worker(delayed_job host:rvm pid:1610)] 2 jobs processed at 8.7708 j/s, 0 failed ...

Search log for new string in content:

Sphinx  Found 0 results

My thinking_sphinx.yml:

development:
  html_strip: true
  enable_star: true
  min_prefix_len: 3
  mem_limit: 128M

My page_index.rb:

ThinkingSphinx::Index.define :page, :with => :active_record, :delta => ThinkingSphinx::Deltas::DelayedDelta do
  indexes content
  indexes title

  has created_at, updated_at
  has sites.id, :as => :site_id
end

Page structure:

    create_table :pages do |t|
      t.string :name
      t.string :title
      t.text :content
      t.string :keywords
      t.string :description

      t.timestamps
    end

class Page < ActiveRecord::Base
 has_and_belongs_to_many :sites

After rake ts:index new string isn't found, too.
After rake ts:rebuild new string is found.

No jobs in queue

Hi, I've just added delayed delta 2.0.1 to thinking sphinx 3.0.1. When I modify an indexed record, the delta field is changed from 0 to 1, but it doesn't look like an indexing job is added to the queue. If I stop the delayed_delta service, I should see new jobs in the delayed_jobs table, but it remains empty.

My index is defined as:

ThinkingSphinx::Index.define :member, with: :active_record, delta: ThinkingSphinx::Deltas::DelayedDelta do

I added a delta column to the members table. Delayed_delta was previously installed, and I've confirmed that it's working correctly when I add jobs directly.

Can set priority?

Can set delayed delta priority.

2.0

Which repository did the ts-delayed-delta 2.0.0 gem come from? Rubygems has it listed as coming from here, but the last update here was a year ago.

Also 2.0.0 requires thinking-sphinx 2.1 and will not work with 2.0.14 or older. Did not test with 1.5 or 3, the other current ts gems.

Minor documentation update

Hi,

The documentation for getting ts-delayed-delta running is still Rails 2 specific. I recently tried to get things running by just adding ts-delayed-delta to my Gemfile and having never worked with Rails 2 completely overlooked that ":lib => 'thinking_sphinx/deltas/delayed_delta'" equated to a "require" statement in Bundler parlance. So I basically just had:

gem 'ts-delayed-delta'

in my Gemfile. This caused the following odd error when running any sphinx rake task:

Generating Configuration to /config/development.sphinx.conf
rake aborted!
uninitialized constant ThinkingSphinx::Deltas::DelayedDelta

Once I added the "require" statement, everything started humming perfectly:

gem 'ts-delayed-delta', :require => 'thinking_sphinx/deltas/delayed_delta'

This should be added to the documentation if possible. Thanks for your hard work!

undefined method `join' for nil:NilClass

I'm getting the same problem as these guys described here. I'm running:

ruby 1.9.3p124
delayed_job 2.1.4
thinking-sphinx 2.0.11
ts-delayed-delta 1.1.3

DJ crashes and here's the stack trace:

[Worker(host:admin.artfox.com pid:10963)] Starting job worker
rake aborted!
undefined method `join' for nil:NilClass
/var/www/production/shared/bundle/ruby/1.9.1/gems/ts-delayed-delta-1.1.3/lib/thinking_sphinx/deltas/delayed_delta/delta_job.rb:17:in `display_name'
/var/www/production/shared/bundle/ruby/1.9.1/gems/delayed_job-2.1.4/lib/delayed/backend/base.rb:67:in `name'
/var/www/production/shared/bundle/ruby/1.9.1/gems/delayed_job-2.1.4/lib/delayed/worker.rb:169:in `handle_failed_job'
/var/www/production/shared/bundle/ruby/1.9.1/gems/delayed_job-2.1.4/lib/delayed/worker.rb:129:in `rescue in run'
/var/www/production/shared/bundle/ruby/1.9.1/gems/delayed_job-2.1.4/lib/delayed/worker.rb:119:in `run'
/var/www/production/shared/bundle/ruby/1.9.1/gems/delayed_job-2.1.4/lib/delayed/worker.rb:177:in `reserve_and_run_one_job'
/var/www/production/shared/bundle/ruby/1.9.1/gems/delayed_job-2.1.4/lib/delayed/worker.rb:104:in `block in work_off'
/var/www/production/shared/bundle/ruby/1.9.1/gems/delayed_job-2.1.4/lib/delayed/worker.rb:103:in `times'
/var/www/production/shared/bundle/ruby/1.9.1/gems/delayed_job-2.1.4/lib/delayed/worker.rb:103:in `work_off'
/var/www/production/shared/bundle/ruby/1.9.1/gems/delayed_job-2.1.4/lib/delayed/worker.rb:78:in `block (2 levels) in start'
/home/capistrano/.rbenv/versions/1.9.3-p125/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
/var/www/production/shared/bundle/ruby/1.9.1/gems/delayed_job-2.1.4/lib/delayed/worker.rb:77:in `block in start'
/var/www/production/shared/bundle/ruby/1.9.1/gems/delayed_job-2.1.4/lib/delayed/worker.rb:74:in `loop'
/var/www/production/shared/bundle/ruby/1.9.1/gems/delayed_job-2.1.4/lib/delayed/worker.rb:74:in `start'
/var/www/production/shared/bundle/ruby/1.9.1/gems/delayed_job-2.1.4/lib/delayed/tasks.rb:9:in `block (2 levels) in <top (required)>'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
/home/capistrano/.rbenv/versions/1.9.3-p125/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:116:in `invoke_task'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `block (2 levels) in top_level'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `each'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `block in top_level'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:88:in `top_level'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:66:in `block in run'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:63:in `run'
/var/www/production/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/bin/rake:33:in `<top (required)>'
/var/www/production/shared/bundle/ruby/1.9.1/bin/rake:19:in `load'
/var/www/production/shared/bundle/ruby/1.9.1/bin/rake:19:in `<main>'
Tasks: TOP => jobs:work

Rebuilding index when any attribute of model changes

Currently running into an issue where we have set_property :delta => :delayed set properly in the model, however we only have a subset of model attributes specified in the define_index block. We are seeing that even if one of the fields that isn't specified in define_index gets modified, the index will get rebuilt. Is there a way to make it so that the delayed delta index is only rebuilt when specific attributes specified in the define_index block are changed, as opposed to any field of the model?

Rails 3 ActiveRecord::ConnectionNotEstablished

Config

gem 'delayed_job', :git => "git://github.com/collectiveidea/delayed_job.git"
gem 'thinking-sphinx', :require => 'thinking_sphinx', :git => "git://github.com/freelancing-god/thinking-sphinx.git", :branch => 'rails3'
gem 'ts-delayed-delta', :require => 'thinking_sphinx/deltas/delayed_delta', :git => "git://github.com/freelancing-god/ts-delayed-delta.git"

Ruby 1.9.2
Rails 3.0

require 'thinking_sphinx/deltas/delayed_delta/tasks' at the bottom of Rakefile

With this config I've the following error when trying to run a task (whatever rake task)

$ rake --tasks
(in /Users/cyrille/Projets/v2/keldelice)
rake aborted!
ActiveRecord::ConnectionNotEstablished
/Users/cyrille/Projets/v2/keldelice/Rakefile:1:in `<top (required)>'
(See full trace by running task with --trace)

Any ideas ?

Delayed Delta in tests

Yo! To speed up tests, I don't want to use delayed_delta in tests. To achieve this I currently have the gem specified as only being in the dev/production style groups in my Gemfile, and I have this in my spec_helper.rb

module ThinkingSphinx
  module Deltas
    DelayedDelta = nil
  end
end

I'm wondering two things

Is there a risk to this approach in terms of my test env being different to my dev/production environment?
If this seems like a sensible thing to do, could we do something in ts-delayed-delta so that deltas are ignored in the test environment (or at least configurable)?

Not a priority but thanks in advance when you get 5 to think about it! :)

Feature Request: class method to force delta index a set of items

Not sure if this is already implemented - couldn't find it. Also not sure if it should go here or in the main TS project.

Anyways. I have a piece of code that looks like this now:

  def self.reindex relation
    relation.update_all(:delta => true)
    # We sometimes don't have an active sphinx index - eg tests.
    sphinx_indexes.first.try(:delta_object).try(:index, self)
  end

There's two situations that I'll call this: I've done a mass-column update on a 'displayed' flag on that model, or an object associated to the model has changed, and that association has many children.

Calling 'save' on each individual instance will take too long. Also, this happens frequently enough that forcing a full reindex each time isn't feasible.

I'd be happy to put in a PR for this if you can give me a bit of guidance on how you'd like it implemented.

ts-delayed-delta queues jobs even if TS is not running

Hi,

I noticed that our tests were both running for longer and were including lots of indexer output in our TS3 branch.

Further investigation showed that in TS3, with ts-delayed-delta 2, the delta SphinxQL mode does not skip running creating the indexing job if updates_enabled? or deltas_enabled? are false.

I then also saw that TS3 doesn't implement these features.

How do you recommend I disable delayed job delta indexing when running in tests, for those tests that Sphinx is not otherwise running?

enqueue deprecation

I have mountains of these in my logs. It would be nice to clean them up.

[DEPRECATION] Passing multiple arguments to #enqueue is deprecated. Pass a hash with :priority and :run_at.

Thanks & LOVE Thinking Sphinx!

delayed_delta indexing correctly, but leaving delta flag set to false

Pat, I'm using ts-delayed-delta on a project and thing seem to work fine when I search, except that the delta column is never set back to false by the delta reindexing task.

What I'm finding out is that as I add new records to the DB, the delta task is processing an increasing number of them. And if I look at the DB, the delta flag is never set to false for those new records (it is only set to false when I do a ts:rebuild)

indexing index 'transaction_delta'...
collected 1 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 1 docs, 99 bytes
total 0.010 sec, 9467 bytes/sec, 95.62 docs/sec
total 3 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 9 writes, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg
rotating indices: successfully sent SIGHUP to searchd (pid=36846).
[Worker(host:pi.local pid:36850)] Job Thinking Sphinx: Process transaction_delta (id=39) COMPLETED after 0.1610
[Worker(host:pi.local pid:36850)] Job Thinking Sphinx: Mark 121 in transaction_core as deleted (id=40) RUNNING
[Worker(host:pi.local pid:36850)] Job Thinking Sphinx: Mark 121 in transaction_core as deleted (id=40) COMPLETED after 0.0053
[Worker(host:pi.local pid:36850)] 2 jobs processed at 9.8449 j/s, 0 failed

and then after adding another transaction:

indexing index 'transaction_delta'...
collected 2 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 2 docs, 198 bytes
total 0.011 sec, 17128 bytes/sec, 173.01 docs/sec
total 3 reads, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg
total 9 writes, 0.000 sec, 0.2 kb/call avg, 0.0 msec/call avg
rotating indices: successfully sent SIGHUP to searchd (pid=36846).
[Worker(host:pi.local pid:36850)] Job Thinking Sphinx: Process transaction_delta (id=41) COMPLETED after 0.1240
[Worker(host:pi.local pid:36850)] Job Thinking Sphinx: Mark 124 in transaction_core as deleted (id=42) RUNNING
[Worker(host:pi.local pid:36850)] Job Thinking Sphinx: Mark 124 in transaction_core as deleted (id=42) COMPLETED after 0.0032
[Worker(host:pi.local pid:36850)] 2 jobs processed at 14.1306 j/s, 0 failed

I'm sure I'm missing something obvious...

here's my index:

ThinkingSphinx::Index.define :transaction,
                             :with => :active_record,
                             :delta => ThinkingSphinx::Deltas::DelayedDelta do
  has :account_id, :user_id, :type, :created_at
  indexes 'cast(transactions.amount/100 as decimal(10,2))', :as => :amount, :sortable => true
  indexes :card_name, :sortable => true
  indexes :card_type
...
  indexes :card_receipt_number
  indexes :card_address
  indexes :card_city
  indexes :card_postal_code
  indexes :card_phone
  indexes account(:name), :as => :account_name
  indexes [user(:first_name),user(:last_name)], :as => :username
end

And this is my Gemfile:

gem 'rails', '3.2.14'
...
gem 'delayed_job', '4.0.0'
gem 'delayed_job_active_record', '4.0.0'
gem 'thinking-sphinx', '3.0.4'
gem 'ts-delayed-delta', '2.0.0'

undefined method `client' for ThinkingSphinx::Configuration

Hi Pat,

I have thinking-sphinx v2.1.0, ts-delayed-delta v1.1.3, Ruby 1.9.1 and Rails 3.2.14. NewRelic has reported this error in our application, we have never faced such issue before. The full trace provided by new relic is as follows

NoMethodError: undefined method `client' for #ThinkingSphinx::Configuration:0x00000006668950

…ing_sphinx/deltas/delayed_delta/flag_as_deleted_job.rb:  30:in `block in perform'
…ing_sphinx/deltas/delayed_delta/flag_as_deleted_job.rb:  29:in `each'
…ing_sphinx/deltas/delayed_delta/flag_as_deleted_job.rb:  29:in `perform'
…9.1/gems/delayed_job-4.0.3/lib/delayed/backend/base.rb: 102:in `block in invoke_job'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  61:in `call'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  61:in `block in initialize'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  66:in `call'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  66:in `execute'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  40:in `run_callbacks'
…9.1/gems/delayed_job-4.0.3/lib/delayed/backend/base.rb:  99:in `invoke_job'
…ems/1.9.1/gems/delayed_job-4.0.3/lib/delayed/worker.rb: 203:in `block (2 levels) in run'
              /opt/ruby-1.9.3/lib/ruby/1.9.1/timeout.rb:  68:in `timeout'
…ems/1.9.1/gems/delayed_job-4.0.3/lib/delayed/worker.rb: 203:in `block in run'
…ems/1.9.1/gems/delayed_job-4.0.3/lib/delayed/worker.rb: 202:in `run'
…ems/1.9.1/gems/delayed_job-4.0.3/lib/delayed/worker.rb: 275:in `block in reserve_and_run_one_job'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  61:in `call'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  61:in `block in initialize'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  66:in `call'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  66:in `execute'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  40:in `run_callbacks'
…ems/1.9.1/gems/delayed_job-4.0.3/lib/delayed/worker.rb: 275:in `reserve_and_run_one_job'
…ems/1.9.1/gems/delayed_job-4.0.3/lib/delayed/worker.rb: 186:in `block in work_off'
…ems/1.9.1/gems/delayed_job-4.0.3/lib/delayed/worker.rb: 185:in `times'
…ems/1.9.1/gems/delayed_job-4.0.3/lib/delayed/worker.rb: 185:in `work_off'
…ems/1.9.1/gems/delayed_job-4.0.3/lib/delayed/worker.rb: 150:in `block (4 levels) in start'
…ems/1.9.1/gems/delayed_job-4.0.3/lib/delayed/worker.rb: 149:in `block (3 levels) in start'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  61:in `call'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  61:in `block in initialize'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  66:in `call'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  66:in `execute'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  40:in `run_callbacks'
…ems/1.9.1/gems/delayed_job-4.0.3/lib/delayed/worker.rb: 148:in `block (2 levels) in start'
…ems/1.9.1/gems/delayed_job-4.0.3/lib/delayed/worker.rb: 147:in `loop'
…ems/1.9.1/gems/delayed_job-4.0.3/lib/delayed/worker.rb: 147:in `block in start'
…s/delayed_job-4.0.3/lib/delayed/plugins/clear_locks.rb:   7:in `call'
…s/delayed_job-4.0.3/lib/delayed/plugins/clear_locks.rb:   7:in `block (2 levels) in <class:ClearLocks>'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  79:in `call'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  79:in `block (2 levels) in add'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  61:in `call'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  61:in `block in initialize'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  79:in `call'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  79:in `block in add'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  66:in `call'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  66:in `execute'
…/1.9.1/gems/delayed_job-4.0.3/lib/delayed/lifecycle.rb:  40:in `run_callbacks'
…ems/1.9.1/gems/delayed_job-4.0.3/lib/delayed/worker.rb: 146:in `start'
…ms/1.9.1/gems/delayed_job-4.0.3/lib/delayed/command.rb: 124:in `run'
…ms/1.9.1/gems/delayed_job-4.0.3/lib/delayed/command.rb: 112:in `block in run_process'
…ms/1.9.1/gems/daemons-1.1.8/lib/daemons/application.rb: 254:in `call'
…ms/1.9.1/gems/daemons-1.1.8/lib/daemons/application.rb: 254:in `block in start_proc'
…gems/1.9.1/gems/daemons-1.1.8/lib/daemons/daemonize.rb:  79:in `call'
…gems/1.9.1/gems/daemons-1.1.8/lib/daemons/daemonize.rb:  79:in `call_as_daemon'
…ms/1.9.1/gems/daemons-1.1.8/lib/daemons/application.rb: 258:in `start_proc'
…ms/1.9.1/gems/daemons-1.1.8/lib/daemons/application.rb: 295:in `start'
…ems/1.9.1/gems/daemons-1.1.8/lib/daemons/controller.rb:  70:in `run'
…/lib/ruby/gems/1.9.1/gems/daemons-1.1.8/lib/daemons.rb: 197:in `block in run_proc'
…y/gems/1.9.1/gems/daemons-1.1.8/lib/daemons/cmdline.rb: 109:in `call'
…y/gems/1.9.1/gems/daemons-1.1.8/lib/daemons/cmdline.rb: 109:in `catch_exceptions'
…/lib/ruby/gems/1.9.1/gems/daemons-1.1.8/lib/daemons.rb: 196:in `run_proc'
…ms/1.9.1/gems/delayed_job-4.0.3/lib/delayed/command.rb: 110:in `run_process'
…ms/1.9.1/gems/delayed_job-4.0.3/lib/delayed/command.rb:  86:in `daemonize'

I tried to investigate the issue too, and the error seems right to me as there is no method name 'client' is defined in thinking-sphinx-2.1.0/lib/thinking_sphinx/configuration.rb then why is the code trying to call the method , am I missing something ?

Question about callbacks

Hi!

In this article about upgrading Thinking Sphinx:
https://freelancing-gods.com/thinking-sphinx/v5/upgrading.html

...it says I need to add an indexing callback like this to my indexed models:

  ThinkingSphinx::Callbacks.append(
    self, :behaviours => [:sql, :deltas]
  )

Is this still needed if I'm using ts-delayed-delta? My indexes don't seem to be updating when I add new records.

I'm busy upgrading a project from Thinking Sphinx 4.0 to 5.4 and I'm not too familiar with Sphinx.

Inhibiting delayed deltas?

I'm curious if you have any suggestions for us here. We're using delayed deltas and they work great, except when we're importing data. When we import 5000 products, we end up creating 5000 delayed delta jobs. We'd obviously far prefer to inhibit delayed deltas during the import, and just create one after the fact. This wouldn't seem to be that unusual a request, do you know if anyone's solved it already?

Issue in handling delta jobs which are being created at the time of full Indexing.

Scenario:
When full indexer runs it first clears the delta jobs from the DJ Queue.
After that full indexer starts re-indexing the sphinx data. And during this time if any updates happen then there will two job added in the DJ Queue for each record update.
1.ThinkingSphinx::Deltas::DelayedDelta::DeltaJob
2.ThinkingSphinx::Deltas::DelayedDelta::FlagAsDeletedJob

Once the full indexer completes it toggle the delta flag of the updated records to false. But there will be Delta Jobs in the DJ Queue.

Now for a particular updated record this is how delta jobs being handled.

When ThinkingSphinx::Deltas::DelayedDelta::DeltaJob executes which generally add the document in the delta core if delta is true. But for this case delta will be false since full indexer has run just before. So it won't add the document in delta core.
When ThinkingSphinx::Deltas::DelayedDelta::FlagAsDeletedJob runs it will mark the document in the main core to be deleted irrespective of delta flag.

Now if we try to search those updated records it won't be available in search.

So, is it possible to have delta flag check in the ThinkingSphinx::Deltas::DelayedDelta::FlagAsDeletedJob too to fix the issue.?
or you have any other suggestion to fix the problem.

Delta Index not working sporadically

Pat,

I have a setup of Sphinx - 2.0.8-id64-release, TS 3.0.4, ts-delayed-delta 2.0.0.

Until a week back, I was using ts-delayed-delta 1.1.3. But after moving to the latest version of ts-delayed-delta. I have been facing issues related to delta indexing. Everything works fine in my development box. In production, delta indexes is good for some time, but after about 6-8 hours. the delta index does not work, from the DJ logs, I see that, the delta index is happening.

I have setup a hourly ts:in in my cron. So, after an hour it is fine, but still future delta indexes does not happen. But after I do a deploy, delta indexes work magically. I have been facing this issue, only after moving to the latest ts-delayed-delta.

jruby problem

Hi:

I use it in jruby environment, but when I modify  data, an error occurred:

Sphinx 2.0.4-id64-release (r3135)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file 'C:/test_jruby/config/development.sphinx.conf'...
indexing index 'Comment_delta'...
FATAL: failed to lock C:/test_jruby/db/sphinx/development/Comment_delta.spl: Invalid argum
ent, will not index. Try --rotate option.

Can someone help me? thanks!

Gem is not respecting the primary_key setting on indices

Don't have time right now to create a proper pull request, but this monkey patch fixes it in my project:

class ThinkingSphinx::Deltas::DelayedDelta
  module SphinxQL
    def delete(index, instance)
      Delayed::Job.enqueue(
        ThinkingSphinx::Deltas::DelayedDelta::FlagAsDeletedJob.new(
          index.name,
          index.document_id_for_key(instance.send(index.options[:primary_key] || :id))
        ), self.class.job_options
      )
    end
  end
end

Delayed delta only on update for subset of attributes

Hi we have a similar requirement as discussed in :
#13

I want delayed delta to index ONLY when a subset of model attributes (specified in the define_index block) is updated. Currently, its indexing even if other attributes (not part of index) are modified.Can this behavior be achieved ?

Thanks

Delayed Job Queue?

I can not find any documentation regarding how to specify a delayed job queue but it looks to be possible.

depends on activerecord < 3.0.0

Gem seems to depends on activerecord (>= 1.15.6, < 3.0.0).
In my gemfile I have ts-delayed-delta and paperclip and I have this error:

Bundler could not find compatible versions for gem "activerecord":
In Gemfile:
ts-delayed-delta (>= 1.1.0) depends on activerecord (>= 1.15.6, < 3.0.0)
paperclip depends on activerecord (3.0.3)

And of course I want to use rails 3.0.0.

Thanks!

pat / ts-delayed-delta Goto Github PK

ts-delayed-delta's Introduction

Delayed Deltas for Thinking Sphinx

Installation

Usage

Contributing

Contributors

Copyright

ts-delayed-delta's People

Contributors

Stargazers

Watchers

Forkers

ts-delayed-delta's Issues

Recommend Projects

Recommend Topics

Recommend Org