Giter Club home page Giter Club logo

serverless-ruby-package's Introduction

serverless-ruby-package

Teaches the serverless framework how to package ruby services and their gem dependencies.

By default, serverless will package all files in your ruby service project. That includes test files, node_modules, readmes, etc. You don't need any of those files to execute your function - they just increase the size of the package and slow down updates. In most cases, it also won't include any rubygems that your function depends on, so the function will not work once it is deployed.

This plugin solves those problems in a couple ways. First, it excludes all files by default, forcing you to whitelist the files needed by your function (using the package/includes key in serverless.yml). Second, it automatically packages the gems from your default bundler group -- skipping gems from any custom groups, like development or test. Sometimes gems themselves will install their own test files - those will also be excluded from the package.

It also handles cross-compiling any gems with native extensions. For example, if you are developing your service on macOS, any gems with native extensions will be compiled locally for macOS. But to deploy to a provider like AWS Lambda, the gems need to be compiled for linux. The plugin will automatically detect this situation and build the extensions for linux using a local docker container. You only need to have docker installed with the daemon running.

Usage

The plugin will make it easier to work with rubygems in your serverless project, but it requires your project to follow some conventions:

  1. Add the plugin
npm install --save-dev serverless-ruby-package

Add the plugin to your serverless.yml file:

plugins:
  - serverless-ruby-package
  1. You must use bundler in standalone mode, with the path vendor/bundle:
bundle install --standalone --path vendor/bundle
  1. Add the following lines to the top of your service handler file:
load "vendor/bundle/bundler/setup.rb"
  1. The plugin will force the serverless packaging step to exclude all files by default. You need to explicitly add your service handler file to serverless.yml:
package:
  include:
    - handler.rb

(optional) If your service is implemented across multiple ruby files, it is recommended that you keep your handler file in the root, and the rest of the files in a lib/ directory. You can make them available to your handler by adding the following line to the top of your handler:

# handler.rb
load "vendor/bundle/bundler/setup.rb"
$LOAD_PATH.unshift(File.expand_path("lib", __dir__))

You will also need to edit your serverless.yml package settings to include those files:

package:
  include:
    - handler.rb
    - lib/**

Verify

Build your package to confirm it is being built as expected:

serverless package

Note - if you have gems with native extensions, and are not developing on linux, make sure have docker installed and running. The first time you package may take a really long time as the docker image is downloaded. Future packaging will be much faster.

That will create the package that is uploaded during a serverless deploy, but keep it around so you can examine it. You can use the following command to verify it only includes the files you expect:

unzip -l .serverless/<servicename>.zip

Configuration

By default, if any gems have native extensions, they will be compiled for the AWS Lambda Linux using Docker.

You can override the default behavior by adding to your serverless.yml file:

custom:
  rubyPackage:
    alwaysCrossCompileExtensions: false

You can also override this behavior using environment variable. If you set the environment variable, it will have precedence over the serverless.yml file:

CROSS_COMPILE_EXTENSIONS=false serverless package

Development

To work on this plugin, you should first run the following in your local directory:

./dev_setup.sh          # configures the demo service project needed by the tests

yarn test               # run the automated test suite

./integration-test.sh   # run serverless package on the demo project and observe the results

Troubleshooting the demo_service

  • Make sure the version of bundler specified in the demo_service Gemfile.lock is compatible with the version installed in the amazon/aws-lambda-ruby:3.2 image.

  • Make sure you can invoke the function n in the docker image:

./invoke-service.sh

Releasing a new version

  • Commit a description of the changes in CHANGELOG.md

  • Run yarn publish. It will prompt for the new version number, and npm credentials. It will create a new commit and tag with the version number changes. Make sure to push them.

serverless-ruby-package's People

Contributors

andersond11 avatar brunoadacosta avatar dependabot[bot] avatar joshuaflanagan avatar kmfukuda avatar lwoodson avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

serverless-ruby-package's Issues

Getting pg gem to install with serverless-ruby-package

Thanks

Firstly - thanks so much for serverless-ruby-package.

It's saved the day for me and the orgs I work for time-and-time-again and has opened the door for us to use
Ruby for our lambdas (which has...in turn...saved many days).

The Problem

I imagine using serverless-ruby-package to deploy Lambdas that use the pg gem is a well-trod path.
But I'm running into problems at build time.
I have a Lambda that uses pg (1.2.3) I get the following error on running serverless invoke local with the --docker
flag.

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /var/task/vendor/bundle/ruby/2.5.0/gems/pg-1.2.3/ext
/var/lang/bin/ruby -I /var/lang/lib/ruby/site_ruby/2.5.0 -r
./siteconf20201011-1-2uaspy.rb extconf.rb
checking for pg_config... no
No pg_config... trying anyway. If building fails, please try again with
 --with-pg-config=/path/to/pg_config
checking for libpq-fe.h... no
Can't find the 'libpq-fe.h header
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.
...SNIP

I understand the heart of this problem is with lambci/lambda:build-ruby2.5 either not including the postgresql-devel and postgresql-libs packages or putting them in an unexpected (for bundler) location.

I've thought of a few different way's it could be solved but each is a slippery slope.

For example...

I could create my own docker image image FROM: lambci/lambda:build-ruby2.5 which yum installs the appropriate libs, but then serverless-ruby-package is hardwired to use lambci/lambda:build-ruby2.X, then I'd have to fork serverless-ruby-package. That's enough work to make me imagine I'm working too hard. Hah.

Do you know a way to install the pg gem and stay on the serverless ==> serverless-ruby-package path?

Example App

You can re-produce this problemby running:

serverless invoke local --docker --function hello --path ./resources/fake-lambda-message.json

In the with-native-extensions branch of nodanaonlyzuul/kinesis-lambda-example

Recommended package configuration of Serverless will be incompatible

Hello there,

according to a deprecation message of the current Serverless-CLI (2.33.1), the support of package.include and package.exclude will be replaced:

Deprecation warning: Support for "package.include" and "package.exclude" will be removed with next major release. Please use "package.patterns" instead

Changing the configuration like documented in https://www.serverless.com/framework/docs/providers/aws/guide/packaging/ this package throws an error:

TypeError: Cannot read property 'push' of undefined at PackageRubyBundlePlugin.beforePackage ([...]node_modules\serverless-ruby-package\index.js:85:45) at PluginManager.invoke ([...]node_modules\serverless\lib\classes\PluginManager.js:551:20)

I already took a look at the index.js. It seems that the usage of exclude and include needs to be redone. I have no idea how to do this without breaking backward compatibility.

Bundle without test development

Agenda

Drop test and development gems during build:
bundle install --standalone --path vendor/bundle --without test development;

custom template using serverless-ruby-package...not really an issue

Sorry to create an issue, but I wanted to let you know about a custom template I created that uses the serverless-ruby-package.

After quite a few serverless ruby projects, I’ve packaged up the basic things I always install into a custom template to cut down on the setup time needed to get a new serverless ruby project started.

TBH, parts of this might make sense to merge into this project.

Hope someone else finds it useful.

https://github.com/apsoto/serverless-aws-ruby-template

Deprecation of include/exclude in v4 in favor of patterns

I ran into this deprecation after upgrading to serverless v.3.34.0:

1 deprecation triggered in the last command:

Support for "package.include" and "package.exclude" will be removed in the next major release. Please use "package.patterns" instead
More info: https://serverless.com/framework/docs/deprecations/#PACKAGE_PATTERNS

I put together a branch here that bumps the integration test version to 3.34.0 and displays the deprecation warnings in the integration test output.

Native extensions built on linux are not working properly

When developing on Linux, the gem native extensions are not compiled using the docker container that emulates the lambda environment - they are compiled using the local Linux environment.

I've observed that the compilation output is different. For one, in my local ubuntu environment, the extensions are compiled to vendor/bundle/ruby/2.5.0/extensions/x86_64-linux/2.5.0/, whereas lambda expects them at vendor/bundle/ruby/2.5.0/extensions/x86_64-linux/2.5.0-static/. I solved this by creating a symlink so that the package will include them at the 2.5.0-static location.

That was successful in including the correct files, however, I suspect the output files were not created as needed. When I deploy the package created on Ubuntu, the demo_service endpoint fails with this error:

$> curl https://deployedendpoint.execute-api.us-east-1.amazonaws.com/dev/
{"alpha":"first","beta":"second","redis_version":"4.1.0",
"error_class":"LoadError",
"error_message":"libruby-2.5.so.2.5: cannot open shared object file: No such file or directory - /var/task/vendor/bundle/ruby/2.5.0/gems/http_parser.rb-0.6.0/lib/ruby_http_parser.so"
}

It is failing at the part that uses the http gem, which relies on a native extension.

I suspect that in my local Ubuntu system, the extensions are being built with shared/dynamic linking, while the docker build container compiles them with static linking. Need to figure out how to force static compiling locally, or else always build extensions using the docker container, even when developing on Linux.

Getting TypeError: Cannot read property 'push' of undefined

When i add

`plugins:

  • serverless-ruby-package
    `

i get TypeError: Cannot read property 'push' of undefined

Type Error ----------------------------------------------

TypeError: Cannot read property 'push' of undefined
at PackageRubyBundlePlugin.beforePackage (/Users/xxxx/node_modules/serverless-ruby-package/index.js:85:45)
at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:576:20)
at async PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:634:7)
at async Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:327:5)
at async /usr/local/lib/node_modules/serverless/scripts/serverless.js:704:9

 For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com

Your Environment Information ---------------------------
Operating System: darwin
Node Version: 16.2.0
Framework Version: 2.43.0
Plugin Version: 5.1.3
SDK Version: 4.2.2
Components Version: 3.10.0

Postgres with ActiveRecord not working

I am trying to create a lambda that works with a rails project. This involve using the pg gem and the active_record gem.

Gemfile

source 'https://rubygems.org'
git_source(:github) { |repo| "https://github.com/#{repo}.git" }

ruby '2.7.2'

gem 'activerecord', '~> 6.0.3.4'
gem 'pg', '~> 1.2.3'
gem 'aws-sdk-ssm', '~> 1'

In order to use the pg gem, I had to extend the base lambci/lambda:build-ruby2.7 image in order for extensions to be installed correctly. Some of the packages installed are probably unnecessary, but shouldn't make a difference.

Dockerfile

FROM lambci/lambda:build-ruby2.7

RUN yum update -y && yum install -y \
    git-core \
    curl \
    zlib1g-dev \
    build-essential \
    libssl-dev \
    libreadline-dev \
    libyaml-dev \
    libsqlite3-dev \
    sqlite3 \
    libxml2-dev \
    libxslt1-dev \
    libcurl4-openssl-dev \
    python-software-properties \
    libffi-dev \
    postgresql-client-common \
    postgresql-client \
    libpq-dev \
    postgresql-devel

The extension files were not being picked up by my lambda, so I manually included them in lib borrowing from this script https://github.com/proton/lambda-ruby-pg/blob/main/deploy.bash .

I then include those files in my serverless.yml

package:
  include:
    - handlers.rb
    - lib/**

Finally, I create the ActiveRecord connection with

handlers.rb

load "vendor/bundle/bundler/setup.rb"
require 'json'
require 'pg'
require 'active_record'

def do_something(event:, context:)
  ActiveRecord::Base.establish_connection('HARDCODED_CONFIGURATION')

  { success: true }
end

Everything works expect the establish_connection call. I am getting

"Error loading the 'postgresql' Active Record adapter. Missing a gem it depends on? Could not find 'pg' (>= 0.18, < 2.0) among 301 total gem(s)\nChecked in 'GEM_PATH=/var/task/vendor/bundle/ruby/2.7.0:/opt/ruby/gems/2.7.0:/var/runtime', execute `gem env` for more information"

This error is being caused by this line https://github.com/rails/rails/blob/3de9669188b543192571932e5cb6f0d5a6ec1043/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb#L4

I confirmed this by manually commenting out that line in my /vendor/bundle before deploying and everything worked. I also have tried commenting out the establish_connection call and just calling PG.library_version, which works. I also tried just putting gem "pg", ">= 0.18", "< 2.0" directly in my code and got the same error result.

It seems like the GEM_PATH being used by this command does not use the same LOAD_PATHS variable that is populated by bundler in standalone mode.

I don't think this is necessarily a bug with this plugin, but any guidance would be greatly appreciated.

Package problems with mysql gem

I'm having trouble using the mysql2 gem in a project, the image does not contain the mysql headers and the compilation gives an error.

Is there any way I can use a custom docker image? or do we have another way of doing it?

Cannot Find Gems in 3.2 Runtime when bundled in lambci/lambda:build-ruby2.7

Hi @joshuaflanagan

Firstly - THANK YOU for this plugin. It's really simplified our build and deploy strategies.
It's great. I appreciate the work you and other contributors have put into it.

Cannot Find Gems in Deployed 3.2 Runtime

We recently updated a serverless project's runtime to ruby3.2 - and once it was deployed, it couldn't find gems.
A bit of digging uncovered that while the gems bundled (in the ephemeral lambci/lambda:build-ruby2.7 image) into vendor/bundle/ruby/2.7.0/...the setup.rb reads as:

require 'rbconfig'
ruby_engine = RUBY_ENGINE
ruby_version = RbConfig::CONFIG["ruby_version"]
path = File.expand_path('..', __FILE__)
$:.unshift "#{path}/../#{ruby_engine}/#{ruby_version}/gems/aws-eventstream-1.1.0/lib"
$:.unshift "#{path}/../#{ruby_engine}/#{ruby_version}/gems/aws-partitions-1.380.0/lib"
$:.unshift "#{path}/../#{ruby_engine}/#{ruby_version}/gems/aws-sigv4-1.2.2/lib"

So in the runtime RbConfig::CONFIG["ruby_version"] will be evaluated as "3.2.0" - and the $LOAD_PATH array won't will build paths like:

.../2.7.0/gems/aws-eventstream-1.1.0/lib instead of the correct
.../3.2.0/gems/aws-eventstream-1.1.0/lib.

Does that make sense / jive with your experience?

Using a Maintained Image for Building

lambci/lambda is deprecated. They won't be building a 3.2 image.
Has there been any investigation into switching this repo to use the aws/aws-lambda-base-images images per lambci/lambda's deprecation suggestion?

Deploying with no gems outside of dev group results in all gems packaged

Gemfile contains some development gems:

# frozen_string_literal: true

source "https://rubygems.org"

git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }

group :development do
  gem "rake"
  gem "rspec"
  gem "pry-byebug"
  gem "rubocop"
end

Relevant serverless.yml:

plugins:
  - serverless-localstack

provider:
  name: aws
  runtime: ruby2.5
  stage: ${opt:stage}
  region: ${opt:region}
  profile: ${opt:profile}

package:
  excludeDevDependencies: false # only applies to node
  include:
    - handlers.rb
    - lib/**

Running sls package and checking the size of the .serverless/<service>.zip file:

-rw-r--r--  1 lwoodson  staff   5.3M Mar 27 14:44 shipit-qbridge.zip

When I add the redis gem to the Gemfile:

# frozen_string_literal: true

source "https://rubygems.org"

git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }

gem "redis", "~> 3.3.3"

group :development do
  gem "rake"
  gem "rspec"
  gem "pry-byebug"
  gem "rubocop"
end

Packaging is optimized as expected. Here is the size of the .serverless/<service>.zip file after running sls package again:

-rw-r--r--  1 lwoodson  staff    67K Mar 27 14:46 shipit-qbridge.zip

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.