betterment / delayed Goto Github PK

Hello!

We're receiving the following error after upgrading to Rails 7.1.1

Job failed to load: undefined class/module Delayed::JobWrapper. 
Handler: \"--- !ruby/object:Delayed::JobWrapper
job_data:
  job_class: TestJob
  job_id: af7d22a1-35c9-42d8-9f57-8af524bffcc5
  provider_job_id:
  queue_name: default
  priority:
  arguments: []
  executions: 0
  exception_executions: {}
  locale: en
  timezone: UTC
  enqueued_at: '2023-10-19T16:03:27.456958896Z'
  scheduled_at:
\"
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/backend/base.rb:84:in `rescue in payload_object'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/backend/base.rb:81:in `payload_object'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/backend/base.rb:133:in `max_run_time'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:194:in `max_run_time'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:137:in `block in run'
/usr/local/lib/ruby/3.2.0/benchmark.rb:311:in `realtime'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:136:in `run'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:210:in `block in run_job'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:61:in `block in initialize'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:66:in `execute'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:210:in `run_job'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:102:in `block (4 levels) in work_off'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/plugins/connection.rb:7:in `block (3 levels) in <class:Connection>'
/usr/local/bundle/ruby/3.2.0/gems/activerecord-7.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:229:in `with_connection'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/plugins/connection.rb:6:in `block (2 levels) in <class:Connection>'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:61:in `block in initialize'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:79:in `block in add'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:66:in `execute'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:121:in `run_thread_callbacks'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:101:in `block (3 levels) in work_off'
/usr/local/bundle/ruby/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
/usr/local/bundle/ruby/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
/usr/local/bundle/ruby/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
/usr/local/bundle/ruby/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
/usr/local/bundle/ruby/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
/usr/local/bundle/ruby/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'

Steps to reproduce

1. Create a new Rails Project
2. Add Delayed
3. Create a TestJob

    # frozen_string_literal: true
   
    class TestJob < ApplicationJob
      queue_as :default
   
      def perform
      end
    end
   

4. Queue the job to perform later
5. Run the worker process and observe the job failure in the delayed_jobs table

I'm thinking of porting https://github.com/codez/delayed_cron_job to delayed, should I make a separate gem or is there any change a PR would be merged?

https://github.com/Betterment/delayed#migrating-from-delayedjob states "that some configurations, like queue_attributes, exit_on_complete, backend, and raise_signal_exceptions have been removed entirely." I think the lack of raise_signal_exceptions (and the reliance on the behaviour described in https://github.com/Betterment/delayed#running-a-worker-process) could prevent me from suggesting switching over from delayed_job to delayed. Would it be difficult to support raise_signal_exceptions and are there any concerns with the idea of supporting it?

Hi! This project looks very exciting for teams looking to migrate away from delayed_job. However, I find myself having some lingering questions about the idempotency requirement of this gem:

• Rails enqueues a lot of built-in jobs to ActiveJob. Stuff like ActiveStorage and ActionMailer. Are all these built-in jobs already idempotent? Is there something that needs to be done to make them idempotent? Or can delayed not be used with these jobs?
• Under what circumstances do successful jobs run twice? And what's the time horizon on this? Can a job only be picked up a second time before it's completed? Or is there a window of time after some worker has fully completed the job that it can be picked up again? (This question is motivated by the idea of using Redis as an external locking mechanism to aid with idempotency with e.g., jobs that make external service calls. If a job releases the lock in Redis in after_perform would there still be a race condition?)

Massive thanks to the team at Betterment for this project!

Hello! Thanks for taking up the torch and improving DelayedJob with all the goodies. I'm particularly grateful that this supports Ruby3 kwargs, whereas DelayedJob said they won't for some reason.

We are in the midst of migrating from DJ to Delayed and have found a sticking point with respect to our handling of "Duplicate Jobs". Duplicate jobs are separate jobs created by different parts of the system that do the same thing. For instance, let's say I have an expensive calculation that updates any time a certain frequent activity occurs in our app. So, different actions can result in the same job being created. While the job itself is idempotent, I don't want to fill my job queue with "duplicates" of the same job.

A long time ago, I had created a gist to handle this and it was recently turned into a gem by some other devs, which validates the need for Duplicate checking.

The way this gem works is that it expands the ActiveRecord model to have a "signature" column which can be indexed. It will try to infer a signature for a given job or it can be explicitly defined with custom logic for each individual job. The signature is then utilized by a "standard" DJ plugin.

Now that we are migrating to Delayed gem, I'd like to build support for this functionality into Delayed. The issue I found is that the Delayed gem seems to only include the Delayed::Job class (the ActiveRecord) when Rails::Engine is not defined:

This prevents the gem from hooking into the ActiveRecord model without some fancy require logic. Is there a reason for this? Or is there another way to hook into the ActiveRecord class to extend functionality?

I see the above code was created via this commit and says that Delayed::Job should autoload. However, it's not present during Duplicate checking gem code. I can see that the load paths doesn't have /app/models at the time of the next gem, but does at the time my Rails console loads. So, I'm wondering if/how its possible to get the load path updated earlier or before this next gem loads

Thanks!

I saw that Delayed::Command has been removed from this gem in favor of running through rake.

Could you provide an example of how to achieve the same (or the replacement) behavior?

I saw that the QUEUE env is used to specify the queue used by the workers. Is MAX_CLAIMS your way of specifying concurrency? delayed_job would use OS level processes -- delayed uses threads, is that correct?

Hi there, I've been running delayed as my job backend for a while now and just started noticing that some concurrent jobs don't seem to run.

I only have 1 worker, so this might be expected but in https://github.com/Betterment/delayed#running-a-worker-process it says

By default, a worker process will pick up 2 jobs at a time (ordered by priority) and run each in a separate thread.

(That seems to be 5 by default now)

My jobs are very long running and I have:

• Job A 6am-8am (running for 2h)
• Job B 7am-8am (running for 1h)

Job B will never actually run - I think technically it does, but it just no-ops as it is past it's finished time.

So for my understanding, each worker will run multiple jobs in parallel but it does not check for additional jobs while 1 job is running. Even for less long running/scheduled tasks, having 100 jobs of 1 min duration, sprinkled in with some 5 min duration jobs we would sit idle every once in a while until the 5min job is completed?

I have been testing delayed for a few days, seems rock solid!
I like the philosophy on sticking with ActiveJob and ActiveRecord database being used.
Got a quick question on the number of workers we can run safely at the same time?
I want to utilize all the cores on a big server. I have kicked 4 workers with 5 max_claims and
have not experienced any issues so far. Would there be any adverse effects starting up to 10 workers?
Thank you.

In the case of many jobs getting enqueued at about the same time, they might overwhelm the system, fail and be retried at some later point.

However if there's so many and they all get retried at the same (exponential backoff) interval, they might still end up causing trouble.

Adding a(n option to add a) random jitter to the retry interval might distribute the load a bit better and resolve the retry stampede.

I'm trying to configure basic exception notification, but can't seem to get the event to trigger. It triggers if I subscribe to "delayed.job.error", just not "delayed.job.failure". I've confirmed that the job actually fails according to the logs, does anyone see anything I may be doing wrong here?

ActiveSupport::Notifications.subscribe("delayed.job.failure") do |*args|
  event = ActiveSupport::Notifications::Event.new(*args)
  job = event.job
  ExceptionNotifier.notify_exception(job.last_error, data: { handler: job.handler })
end

I am looking for a Sidekiq alternative and from what I can tell this is quite similar to good_job? - Could you please comment where you see the differences/advantages?

The lack of a dashboard in delayed is currently the reason that is making me hesitant, is that something on the roadmap or is there an immediate solution for that?

Our jobs are remaining locked after the worker process receives the SIGKILL (in our case from docker stop after the grace period elapses).

I'm not sure if this is by design or not, the README states (may being the keyword):

the process and may result in long-running jobs remaining locked until Delayed::Worker.max_run_time has elapsed.

In my tests they always remains locked when a job is running and Delayed also doesn't terminate with a SIGTERM.

class TestWorker
  def start
    trap('TERM') { quit! }
    trap('INT') { quit! }

    500.times do |i|
      puts "Run #{i}"
      sleep 1.second
    end
  ensure
    on_exit!
  end

  def quit!
    puts 'quit!'
  end

  def on_exit!
    puts 'on_exit!'
  end
end

namespace :test do
  desc "Test signal interupts"
  task work: :environment do
    TestWorker.new.start
  end
end

If you run that as as a rake task, it won't terminate on a SIGTERM, only SIGKILL which won't execute the ensure block which in Delayed is what unlocks the jobs.

class TestWorker
  def start
    trap('TERM') { quit! }
    trap('INT') { quit! }

    500.times do |i|
      puts "Run #{i}"
      sleep 1.second
    end
  ensure
    on_exit!
  end

  def quit!
    puts 'quit!'
    exit
  end

  def on_exit!
    puts 'on_exit!'
  end
end

namespace :test do
  desc "Test signal interupts"
  task work: :environment do
    TestWorker.new.start
  end
end

What's the recommended way to gracefully shut down a running job when the process is terminated? Should a job implement its own traps or is there a hook that Delayed offers?

• Rails 7.0.7
• Ruby 3.2.2
• Delayed 0.5.0

Any chance the private dashboard fork will be open-sourced?
If not, any good gems we can rely on for a decent UI similar to sidekiq?

Hi! We have migrated DelayedJob to Delayed and are running it.
While observing the worker, we came across an error and found a difference in behavior.

Here is the stack trace in the last_error column.

This error occurs when a job is enqueued even though the class definition of the job does not exist in the worker.
(For example, this can occur when adding a new job definition and trying to synchronize the webapp, worker Deployment with ArgoCD.)

I noticed that some kind of error occurs when executing a job, whether it is a DelayedJob or Delayed, but the difference is that Delayed does not retry and the failed_at column is updated.

The situation of enqueuing a non-existent job is not good, but we expect the worker to retry and complete the job execution once the code is up-to-date.

I am assuming that this difference in behavior exists because of the difference in serialization and deserialization of jobs.

In the case of Delayed, the job key is added to the serialization string of the job, and an error seems to occur when deserializing it.
DeserializationError is handled so that it is not retried.
(In DelayedJob, there is no job key and no error occurs during deserialization.)

--- !ruby/object:Delayed::JobWrapper
job_data:
 job_class: Sample
 job_id: c5d50b73-c8c9-4df0-8ea3-d2f6ca45b85d
 provider_job_id: 
 queue_name: v2
 priority: 
 arguments: &2 []
 executions: 0
 exception_executions: &1 {}
 locale: en
 timezone: UTC
 enqueued_at: '2023-01-20T04:17:09Z'
job: !ruby/object:Sample
 arguments: []
 job_id: c5d50b73-c8c9-4df0-8ea3-d2f6ca45b85d
 queue_name: v2
 priority: 
 executions: 0
 exception_executions: *1
 timezone: UTC
 provider_job_id: 
 serialized_arguments: *2
 locale: en
 enqueued_at: '2023-01-20T04:17:09Z'

I assume that the job can be executed even if this job key does not exist.

Is there any intention behind this difference in specifications?

We plan to use 'delayed' as an asynchronous processing worker on Kubernetes pods.
So we need a way to do healthcheck of the worker process and we have the following method to do the healthcheck now.

livenessProbe:
    exec:
      command:
        - bash
        - -lc
        - '[[ $(ps aux | grep delayed | grep -v grep | wc -l) > 0 ]]'

However, if there is a better method, we would like to adopt it.
So, do you have any other methods?
I'd like to know it for reference because I saw that you use kubertenes in other issues.

Hi there, I've raised several PRs to DelayedJob which aren't getting merged :(

I'm wondering if the owners of this repo are open to me porting some of the work here:

• Support for worker pools (in addition to queues)
• Multi-process support (without Daemons gem -- see collectiveidea/delayed_job#1138)
• Copy-on-write forking, to improve memory with multi-process

Let me know and I'll start raising PRs soon.

Ruby 3 changed the way it handles keyword arguments, and is a roadblock with delayed_job. I checked the relevant bits in the source of this gem (at least the bits that used to be relevant in delayed_job) and didn't see any changes or fixes for Ruby 3 keyword argument handling.

While the gemspec suggests Ruby 3 should work, does it really?

Hi and thank you for open sourcing this job framework!

I'm just starting to try it out but it looks quite promising. My mine motivator is the SQLite support.

One feature I'm missing (or haven't found yet) are periodic aka cron jobs. I have many use cases for some daily/weekly etc sync or update jobs.

Not sure if you think that should be part of delayed or are you maybe combining delayed with some kind of scheduler that runs alongside and simply queues jobs when their time has come? If so, any recommendations?

Hello again!

Thanks again for this awesome library. I've ended up with one more question as I dig through our code to evaluate switching.

In what ways have you changed either the schema of the delayed_jobs table, or the assumptions surrounding how it's used?

We have a number of places in our code which--whether or not this is something that should have been done--directly query the table.

Generally these do one of a few things:

• Query for last_error IS (NOT) NULL for monitoring errors
• Query for locked_at IS (NOT) NULL for monitoring current queue usage
• (This is the one I'm really worried about) Query for handler LIKE '%SomeJobClass%' to check for the number of instances of a particular job in the queue. This most often seems to be used to prevent enqueueing too many instances of expensive/side-effectful jobs at once.
• I've also found one file that... I think is trying to parse the handler column to get the name of each job in the queue, again for a reporting page.

Would you expect any of those queries, or queries against the table in general, to break? Does Delayed provide new/better ways to do any of these things? Are there any DB migrations that need to be run to switch libraries?

I am opening this issue primarily to get the thoughts and feedbacks from others on whether or not there are glaring problems with this idea that I'm not seeing. So, to the idea:

What if the job queue were run against a dedicated database, separate from the primary app database? Rails now supports multiple databases, so the infrastructure is in place. I began an experimental branch in a project of mine (leading to PR #12) where I am using a SQLite database to manage my job queue independently of my primary PostgreSQL database. To my initial thinking, this gives the benefits of a Redis-backed queue of not blocking or throttling application database work, while also giving the benefits of having a SQL-backed queue.

However, one of the the primary benefits of a SQL-backed jobs queue is co-transactionality. In the README I note:

Important: the above assumes that the connection used by the transaction is the one provided by ActiveRecord::Base. (Support for enqueuing jobs via other database connections is possible, but is not yet exposed as a configuration.)

So, this gets me thinking, what all do you know that I have yet to learn? And, how can I help, if possible, to bring co-transactional, but separate DB jobs into the world?

When i run 'rake delayed:work' to start the worker process, i am getting this error.

D, [2022-10-07T17:38:17.245081 #20659] DEBUG -- : 2022-10-07T17:38:17+0530: [Worker(host:Vs-MacBook-Pro.local pid:20659)] Starting Delayed::Worker

D, [2022-10-07T17:38:17.372640 #20659] DEBUG -- : Delayed::Job Load (0.5ms) UPDATE "delayed_jobs" SET locked_at = '2022-10-07 12:08:17.367212', locked_by = 'host:Vs-MacBook-Pro.local pid:20659' WHERE ctid = ANY (ARRAY (SELECT ctid FROM "delayed_jobs" WHERE (((run_at <= '2022-10-07 12:08:17.340440' AND (locked_at IS NULL OR locked_at < '2022-10-07 12:02:47.340461')) OR locked_by = 'host:Vs-MacBook-Pro.local pid:20659') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5 FOR UPDATE SKIP LOCKED)) RETURNING *

D, [2022-10-07T17:38:17.373088 #20659] DEBUG -- : 2022-10-07T17:38:17+0530: [Worker(host:Vs-MacBook-Pro.local pid:20659)] Error while reserving job(s): PG::SyntaxError: ERROR: syntax error at or near "SKIP"

LINE 1: ...ER BY priority ASC, run_at ASC LIMIT 5 FOR UPDATE SKIP LOCKE...

My postgresql version is :

$ postgres -V

postgres (PostgreSQL) 14.3

Anyone have any idea why i am getting this error ?

                                                         ^