Giter Club home page Giter Club logo

opsworks_delayed_job's Introduction

opsworks_delayed_job

These recipes set up an AWS OpsWorks instance to run delayed_job workers for a Rails application.

Configuration Examples

By default, 4 delayed_job workers will be configured. To customize this, you may override the delayed_job.pool_size attribute in your stack's custom JSON:

{
  "delayed_job": { "pool_size": 6 }
}

Specify an alternate delayed_job script location by overriding the delayed_job.path_to_script attribute in your stack's custom JSON. The default, bin was added in Rails 4. For pre-Rails 4 projects, set it to script:

{
  "delayed_job": { "pool_size": 3, "path_to_script": "script" }
}

By default, workers will read from all queues. You can also specify custom queues parameters by adding attributes corresponding to each instance name and worker process. You can set a specific value for --sleep-delay seconds with a sleep_delay parameter. A special default pool can be specified to supply configuration for instances that don't otherwise appear in the JSON:

{
  "delayed_job": {
    "YOUR_APP_NAME": {
      "pools": {
        "worker1": {
          "0": { "queues": "highpriority,images" },
          "1": { "queues": "emails" },
          "2": { "queues": "bidding", "sleep_delay": 1 }
        },
        "default": {
          "0": { "queues": "emails" },
          "1": { "queues": "emails" },
          "2": { "queues": "emails" }
        }
      }
    }
  }
}

OpsWorks Set-Up

Create a custom layer for the delayed_job instances. Add the AWS-OpsWorks-Rails-App-Server security group to the layer, so that the instances have the usual cache and database access. Then, assign this cookbook's custom chef recipes to OpsWorks events as follows:

  • Setup: opsworks_delayed_job::setup
  • Configure: opsworks_delayed_job::configure
  • Deploy: opsworks_delayed_job::deploy
  • Undeploy: opsworks_delayed_job::undeploy
  • Shutdown: opsworks_delayed_job::stop

To Do

  • Attributes should be better structured (allowing customization without requiring all worker processes to be listed).

© 2013-2015 Joey Aghion, Artsy. MIT License.

opsworks_delayed_job's People

Contributors

alexchumak avatar douglascamata avatar joeyaghion avatar robamaton avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

opsworks_delayed_job's Issues

opsworks_delayed_job::deploy - : Command timed out after 600s

Hi,

Thank you for sharing the code. I would like to know how to increase the deploy time to 1000s?

Currently I get a deploy timeout error at 600s. Thanks in advance. Regards,

[2014-05-06T02:23:33+00:00] ERROR: Running exception handlers
[2014-05-06T02:23:33+00:00] ERROR: Exception handlers complete
[2014-05-06T02:23:33+00:00] FATAL: Stacktrace dumped to /var/lib/aws/opsworks/cache/chef-stacktrace.out
[2014-05-06T02:23:33+00:00] ERROR: deploy[/srv/www/lakers_test] (opsworks_delayed_job::deploy line 65) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 600s:

Prevent Delayed Jobs From Running in Non Delayed Job Layer

Hi,
Thanks for sharing the code. Currently I have an opsworks stack with a layer for dealyed jobs.
Sometimes I have noticed that delayed jobs are locked by my regular layer.
Is there a way to prevent this? Could this be related with a lack of memory in the instance? Currently it is almost at 100%
Also could you please confirm if the following json configuration is fine? With the configuration below, does it mean that I can run 14 delayed concurrent jobs? Thanks in advance

{
  "delayed_job": {
    "pool_size": 14,
    "app_1": {
      "pool_size": 3
    },
    "app_2": {
      "pool_size": 3
    },
    "app_3": {
      "pool_size": 3
    },
    "app_4": {
      "pool_size": 3
    },
    "app_5": {
      "pool_size": 1
    },
    "app_6": {
      "pool_size": 1
    }
  }
}
  • Note: The available memory in the delayed job instance is ~ 29 GB.
    Thanks.

Workers not reloading on Deploy

After deploying an application, the workers are still working off the previous code, so I have to stop the workers myself after the deploy. It seems this would make sense to be built into the deploy recipe in this project. Is this as issue on my end? Or would a PR with this added functionality be appreciated?

mounting /tmp to /mnt/tmp interferes with mysql setup

The recipe mounts /tmp to /mnt/tmp (which in my case isn't even an ebs volume), and thus mysql server setup fails as correct permissions for /mnt/tmp are only set after mysql server install. Mysql server needs to be able to access /tmp to initialise the database.

Why not move this to the documentation, as this is quite server setup specific? Mounting /tmp to /mnt/tmp doesn't do anything unless other stuff is also done...

database.yml not generated on deploy

Hi, when I use this cookbook, the database.yml isn't generated during deployment. I have setup my custom layer exactly as described in the readme. What am I missing to where the database.yml isn't generated as it is in the built-in rails app layer?

Each of my layers, rails and custom, use their own ec2 instance. Should I be using a single ec2 instance that is part of both layers?

I can of course attach config information if you need it.

Berksfile on OpsWorks

How do you get around this dependency issue when using a Berksfile on Opsworks and including these recipes?

STDOUT: Resolving cookbook dependencies...
Fetching 'kit-chef' from source at .
Fetching 'opsworks_delayed_job' from git://github.com/joeyAghion/opsworks_delayed_job.git (at master)
Fetching 'papertrail' from git://github.com/firstbanco/chef-papertrail.git (at master)
Fetching cookbook index from https://supermarket.getchef.com...
STDERR: Unable to satisfy constraints on package deploy, which does not exist, due to solution constraint (opsworks_delayed_job = 0.5.0). Solution constraints that may result in a constraint on deploy: [(opsworks_delayed_job = 0.5.0) -> (deploy >= 0.0.0)]
Missing artifacts: deploy
Demand that cannot be met: (opsworks_delayed_job = 0.5.0)
Unable to find a solution for demands: cron (>= 0.0.0), kit-chef (0.1.0), opsworks_delayed_job (0.5.0), papertrail (0.0.2), postgresql (~> 3.4.2)
---- End output of /opt/aws/opsworks/local/bin/berks vendor /opt/aws/opsworks/current/berkshelf-cookbooks ----
Ran /opt/aws/opsworks/local/bin/berks vendor /opt/aws/opsworks/current/berkshelf-cookbooks returned 106 

Still getting "missing artifacts: deploy" with Berksfile

I would like to point my Berksfile to this repository, not one of its numerous forks, but per the issues encountered in #8, it doesn't work on Opsworks unless you remove depends 'deploy' from metadata.rb. And everything seems to work fine without it.

Could you explain under what conditions the depends line is necessary, and if it's possible to detect when those conditions are met programmatically, only conditionally invoke it?

Specifying more than 10 pools causes processes to fail

Perhaps this is a problem with the way the pool keys are parsed?

Problematic settings: https://gist.github.com/daniel-nelson/0dfa8a26bbdca45ffc5850f710410020

Resulted in:
$ sudo monit validate
'delayed_job.platejoy-1' process is not running
'delayed_job.platejoy-1' trying to restart
'delayed_job.platejoy-1' start: /bin/su
ERROR: there is already one or more instance(s) of the program running

Fixed by these settings: https://gist.github.com/daniel-nelson/753ea0f22911c002726d5a6a78ea7558

Strange Issue With Database Adapter

I'm getting weird behavior with my database.yml file. I'm using a standard setup.

In my chef log, I have the following:

[2015-10-12T03:55:30+00:00] INFO: No database adapter specified for app, guessing
[2015-10-12T03:55:30+00:00] INFO: Looks like app uses mysql2 in its Gemfile

Then down the line I see

[2015-10-12T03:55:30+00:00] INFO: Processing template[/srv/www/app/shared/config/database.yml] action create (opsworks_delayed_job::configure line 11)
[2015-10-12T03:55:30+00:00] INFO: template[/srv/www/app/shared/config/database.yml] backed up to /root/.chef/local-mode-cache/backup/srv/www/app/shared/config/database.yml.chef-20151012035530.839912
[2015-10-12T03:55:30+00:00] INFO: template[/srv/www/app/shared/config/database.yml] updated file contents /srv/www/app/shared/config/database.yml

My new database.yml file doesn't have a database adapter

development:
  adapter: ""
  ...

production:
  adapter: ""
  ...
staging:
  adapter: ""
  ...

However, that cached version does.

I commented this out, but I'm wondering if anyone else is seeing this

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.