We have a nightly task that sometimes takes a bit to run. How difficult would it be to support a "singleton" CRON task? I know the matthuhiggins/sidekiq-cron project supports this, but we use sidecloq for a few different types of jobs.

I'd be happy to jump in and do the work if you have some advice on how to approach the implementation (or if you think its a bad idea).

I'm trying to configure sidecloq in a project with ConnectionPool for client and server configured like:

Sidekiq.configure_client do |config|
config.redis = ConnectionPool.new(size: 5, &redis_conn)
Sidekiq::Status.configure_client_middleware config, expiration: 1.month
end

Looking for logs, sidecloq can´t get the leader lock. It stucks there after creating the new Thread.

Any thoughts on this?

Regards!

I often see "Leader lock held" in my logs, but no indicator from where it came. It would be great if sidecloq used a log tag of like: logger.tagged("Sidecloq") { logger.info("Leader lock held") }.

Is there any plans on supporting newer versions of sidekiq?

@mattyr I am migrating from sidetiq, as its now deprecated, to sidecloq. The swap has been straightforward, however I am wondering how I can migrate our sidetiq specs to verify the same thing in sidecloq.

  describe 'when scheduling a job' do
    it 'performs the job daily' do
      valid = Time.current.utc.beginning_of_day + 1.day
      invalid = valid + 1.day

      Timecop.freeze(Time.current.utc.beginning_of_day)

      next_occurrence = MyJob.schedule.next_occurrence

      expect(next_occurrence).to eq(valid)
      expect(next_occurrence).not_to eq(invalid)
    end
  end

We are trying to configure a production only job per the README example:

production:
  checkout_scanner:
    class: Jobs::Scanner
    cron: "5,15,30,45 * * * *"

But are getting an exception...

#<Thread:0x00005619bb5df518@/data/gems/sidecloq/lib/sidecloq/runner.rb:15 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
	8: from /data/gems/sidecloq/lib/sidecloq/runner.rb:17:in `block in run'
	7: from /data/gems/sidecloq/lib/sidecloq/locker.rb:29:in `with_lock'
	6: from /data/gems/sidecloq/lib/sidecloq/runner.rb:20:in `block (2 levels) in run'
	5: from /data/gems/sidecloq/lib/sidecloq/scheduler.rb:19:in `run'
	4: from /data/gems/sidecloq/lib/sidecloq/scheduler.rb:49:in `load_schedule_into_rufus'
	3: from /data/gems/sidecloq/lib/sidecloq/scheduler.rb:49:in `each'
	2: from /data/gems/sidecloq/lib/sidecloq/scheduler.rb:50:in `block in load_schedule_into_rufus'
	1: from /data/gems/sidecloq/lib/sidecloq/scheduler.rb:59:in `load_into_rufus'
/data/gems/sidecloq/lib/sidecloq/utils.rb:49:in `will_never_run': undefined method `split' for nil:NilClass (NoMethodError)

Putting some debug in to print @schedule.job_specs it appears that the environment option is not being handled in the YML I might be missing something obvious, appreciate any tips or suggestions.

{
    "production" => {
        "checkout_scanner" => {
            "class" => "Jobs::Scanner",
             "cron" => "5,15,30,45 * * * *"
        },
                   "class" => "production",
                    "args" => []
    }
}

Config nesting by env is supported if Rails.env is set. So nesting will not work in non-Rails applications.
However there are many applications which use RACK_ENV and could benefit from nested configs.
Even Rails itself uses RACK_ENV as a fallback if RAILS_ENV is not set

My sidekiq configuration:

Sidekiq.configure_server do |config|
  config.redis = Iron::REDIS_SIDEKIQ_SERVER_CONNECTION_POOL
  config.logger = Rails.logger
  config.client_middleware do |chain|
    chain.add(SidekiqClientPaperTrailMiddleware)
    chain.add(SidekiqClientLogTaggingMiddleware)
    chain.add(SidekiqClientKillswitchMiddleware)
  end
  config.server_middleware do |chain|
    chain.add(SidekiqServerLogTaggingMiddleware)
    chain.add(SidekiqServerExceptionHandlerMiddleware)
    chain.add(SidekiqServerPaperTrailMiddleware)
  end
end

Sidekiq.configure_client do |config|
  config.redis = Iron::REDIS_SIDEKIQ_CLIENT_CONNECTION_POOL
  config.logger = Rails.logger
  config.client_middleware do |chain|
    chain.add(SidekiqClientPaperTrailMiddleware)
    chain.add(SidekiqClientKillswitchMiddleware)
    chain.add(SidekiqClientLogTaggingMiddleware)
  end
end

The output:

["Runner starting"]
["Starting locker check task"]
["Leader lock not held"]
["Leader lock not held"]
["Leader lock not held"]
["Leader lock not held"]
["Leader lock held"]
["Obtained leader lock"]
["Loading schedules into redis"]
["Starting scheduler"]
["Scheduling jobs"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]
["Leader lock held"]

In the Sidecloq config if you don't specify the queue for a job it will properly use the queue as defined by the class. For example, the following job and worker seems to work properly in that the BatchJobMonitor job is placed on the :my_queue queue minutely.

BatchJobMonitor:
  class: BatchJobMonitor
  cron: "* * * * *" # minutely

class BatchJobMonitor
  include Sidekiq::Worker
  sidekiq_options :queue => :my_queue
  ...
end

However, there is a bug in the "Recurring Jobs" UI in that it shows default, as seen below. Instead it should be showing my_queue.

JobEnquer uses String method constantize from gem "activesupport", however this dependency is missing in Gemfile.

Non-rails projects fail with

["error enqueuing example_job - NoMethodError: undefined method `constantize' for \"My::ExampleWorker\":String"]

I propose to add "activesupport" to Gemfile and use require "active_support/core_ext/string" in JobEnquer.

When running inside a rails app, does the scheduler use server time (as cron would), or rails app time (which could be different)?

Human readable schedule would be nice in the web UI for those of us who can't quickly comprehend a cron expression.

Another idea would be to add "next run at" to help admins see the next scheduled run event.

Would you consider a dependency with cron2english or other gem to add this feature to the web UI? I would offer a PR if the maintainers are interested in such a feature.

Thanks!

Rails (5.2) environment

config/sidecloq.yml

development: &default

test:
<<: *default

staging:
<<: *default

production:
update_system:
class: UpdateSystemJob
cron: "* 18 * * *"

Since development environment i don't want to run scheduler. When i startup sidekiq process in dev, it throws this error.
sidecloq-0.4.8/lib/sidecloq/schedule.rb:15:in `from_yaml'
from_hash(YAML.load_file(filename))

Is there a way i can only have some environments run a scheduled jobs and the others are blank?

... or set up the new gitlab CI?

https://docs.gitlab.com/ee/user/project/import/gemnasium.html

Are there plans to bump sidecloq to support Sidekiq 6? Are there blocking changes in the upgrade around what sidecloq uses?

One of the threaded tests must be locking up, and only occasionally. Been tough to replicate locally, even when matching ruby version and test seed. argh.

Hi @mattyr !

Our team is looking to move away from a single app server to a distributed infrastructure. Our plan is to have multiple app servers (Rails w/ Sidekiq on each), and a DB/Redis server. The current issue is where the CRON (currently using whenever, executing Sidekiq tasks) should go.

That's why we started looking for gems doing something similar. Right now we're hesitating between using sidecloq all the way, or use CRON + sidekiq-unique-jobs on each server.

Also found sidekiq-cron, but they have a lot of open issues, and a big one about missing half jobs.

Do you have an opinion on the matter?

Thanks for your work!

👋

Would you consider updating this gem to support Sidekiq 7?

Thanks!

gemspec locks us into Sidekiq 4, but there shouldn't be any significant problems with loosening this to include the 3.x versions of Sidekiq. Just check public API usage, and add to travis build matrix (maybe sinatra's approach of env variables to alter gemfile contents?)

We're currently using rufus-scheduler under the hood to parse and run the schedule. We can be a little more lightweight and increase testability if we replace it.

rufus-scheduler uses a periodically awoken thread to enqueue jobs:

https://github.com/jmettraux/rufus-scheduler/blob/master/lib/rufus/scheduler.rb#L565

Since we know our full schedules at boot, we can use precise thread sleeps to awaken only when a job actually needs to be enqueued. Use something like https://github.com/siebertm/parse-cron combined with concurrent-ruby's ScheduledTask. Ideally there is just a single "timer" thread that handles all schedules, so the Scheduler object will need to be able to merge any overlapping tasks (ie, don't have a ScheduledTask for each of the schedule's job specs).

There may be issues around clock drift if there are long waits between job executions. It may make sense to have a maximum sleep time so that the Scheduler can wake up and re-sync with the system clock. It's probably safe to have this be much longer than rufus-scheduler's default frequency (0.3 secs), as long as detection of skipped times is included (schedule any tasks missed because of clock drift immediately?).

If your sidecroq.yml file looks like this (empty):

development:
  # job:

You'll get this error.

3:16:42 worker.1 | undefined method `each_with_object' for nil:NilClass
13:16:42 worker.1 | /Users/ben/.rvm/gems/ruby-3.2.2/bundler/gems/sidecloq-c802bac8cbb9/lib/sidecloq/schedule.rb:37:in `from_hash'
13:16:42 worker.1 | /Users/ben/.rvm/gems/ruby-3.2.2/bundler/gems/sidecloq-c802bac8cbb9/lib/sidecloq/schedule.rb:15:in `from_yaml'
13:16:42 worker.1 | /Users/ben/.rvm/gems/ruby-3.2.2/bundler/gems/sidecloq-c802bac8cbb9/lib/sidecloq.rb:68:in `extract_schedule'
13:16:42 worker.1 | /Users/ben/.rvm/gems/ruby-3.2.2/bundler/gems/sidecloq-c802bac8cbb9/lib/sidecloq.rb:43:in `startup'
13:16:42 worker.1 | /Users/ben/.rvm/gems/ruby-3.2.2/bundler/gems/sidecloq-c802bac8cbb9/lib/sidecloq.rb:21:in `block (2 levels) in install'