Comments (9)
All done! v1.0.0 released to RubyGems. I've run this thru half a million invokes and everything is working great. Please let me know if you try it and have any feedback.
https://github.com/customink/lambda_punch
from lamby.
New early return too. https://lumigo.io/blog/lambda-extensions-just-got-even-better/
from lamby.
https://twitter.com/julian_wood/status/1396935816325255168
from lamby.
https://twitter.com/dhruvsood/status/1396987415340847106
from lamby.
For ActiveJob adapter to use :async
(config.active_job.queue_adapter = :async
), gives 2 main scenarios:
Non-local (prod, stage)
I wouldn't run this in prod/stage given Lambdakiq
. Similar question of sucker_punch vs sidekiq. Given the option, I would always run Sidekiq for the visibility, control, etc. I only run sucker_punch
if I am constrained in rather specific ways.
Local
Local dev work does have me use :async
, but I'd be hard pressed to tie myself into SQS/Lambdakiq, and thusly Lambda Extensions, for local development.
In relation to Rollbar, log collection, I do agree now that extensions is much more viable though because it replaces sucker_punch as a background threading system. Anytime where I have heavy data && passing to SQS/Lambdakiq isn't viable, background threading rules.
from lamby.
In relation to Rollbar, log collection, I do agree now that extensions is much more viable though because it replaces sucker_punch as a background threading system. Anytime where I have heavy data && passing to SQS/Lambdakiq isn't viable, background threading rules.
Cool, that's the use case I'm thinking is the strongest too.
from lamby.
I got pretty far on this today. Lots of reading on Lambda Extensions.
- https://aws.amazon.com/blogs/aws/getting-started-with-using-your-favorite-operational-tools-on-aws-lambda-extensions-are-now-generally-available/
- https://aws.amazon.com/blogs/compute/building-extensions-for-aws-lambda-in-preview/
- https://docs.aws.amazon.com/lambda/latest/dg/runtimes-extensions-api.html
- https://docs.aws.amazon.com/lambda/latest/dg/runtimes-modify.html
- https://docs.aws.amazon.com/lambda/latest/dg/images-create.html
- https://github.com/aws-samples/aws-lambda-extensions/tree/main/ruby-example-wrapper
An "internal" extension is just not going to work. It was easy to create one but in order to keep the same process you will have to use load
for the runtime. Seems you can not register for any of the events either. Not sure we need them anyway. The benefit of the internal would have been the same process. So a background unit of work would not need another instance of Rails. The $FOOBAR
variable was me debugging the shared process in the handler. Worked great. But a shared process will be just like normal Lambda, frozen and not useful.
#!/var/lang/bin/ruby
$FOOBAR = 'HELLO THERE'
load "/var/runtime/lib/runtime.rb"
I think "external" is the way to go. I got something basic in place using this.
#!/var/lang/bin/ruby
def log(msg) ; puts("['LambdaPunch'] #{msg}") ; end
log "Required"
LAMBDA_PUNCH_BASE_URI = "http://#{ENV['AWS_LAMBDA_RUNTIME_API']}/2020-01-01/extension"
require 'uri'
require 'json'
require 'net/http'
def post(path, options = {})
uri = URI.parse "#{LAMBDA_PUNCH_BASE_URI}/#{path}"
http = Net::HTTP.new uri.host, uri.port
request = Net::HTTP::Post.new uri.request_uri
request['Content-Type'] = 'application/vnd.aws.lambda.extension+json'
options[:headers].each do |header, value|
request[header] = value
end if options[:headers]
request.body = options[:body] if options[:body]
http.request(request)
end
def event_next
uri = URI.parse "#{LAMBDA_PUNCH_BASE_URI}/event/next"
http = Net::HTTP.new uri.host, uri.port
request = Net::HTTP::Get.new uri.request_uri
request['Content-Type'] = 'application/vnd.aws.lambda.extension+json'
request['Lambda-Extension-Identifier'] = LAMBDA_PUNCH_EXTENSION_ID
http.request(request)
end
log "Extension Init.."
response = post 'register',
body: %q|{"events":["SHUTDOWN"]}|,
headers: { 'Lambda-Extension-Name' => File.basename(__FILE__) }
LAMBDA_PUNCH_EXTENSION_ID = response.each_header.to_h['lambda-extension-identifier']
log "LAMBDA_PUNCH_EXTENSION_ID: #{LAMBDA_PUNCH_EXTENSION_ID}"
event_next
This worked too but in order to call random blocks from the other Rails process in this extension process we need a line of communication. I think the way to go would be to use DRb (https://wiki.c2.com/?DistributedRuby) and (https://ruby-doc.org/stdlib-2.7.1/libdoc/drb/rdoc/DRb.html). So my next step is to test this out and see if it works.
from lamby.
Fun fact, DRb is used in ActiveSupport's testing parallel stuff. https://github.com/rails/rails/blob/main/activesupport/lib/active_support/testing/parallelization.rb
from lamby.
As this diagram shows, there is no post-invoke hook. And if we are to use this as an ensure
block in the handler to communicate form the app process to the extension, we need some bi-directional IPC (https://www.sitepoint.com/forking-ipc-ruby-part-ii/). But how?
I was reading there is a Log API (https://docs.aws.amazon.com/lambda/latest/dg/runtimes-logs-api.html) as some event system, but that seems very heavy handed and I'm not sure the guarantees are there. Bi-directional IPC with DRb is not really a thing. Sure I could expose state in the app and have the extensions loop till it sees it, but I'm not sure I like that either.
This post on a Rust extension (https://dev.to/aws-builders/building-an-aws-lambda-extension-with-rust-3p81) got me thinking. It talked about inotify
as a "sophisticated" approach. I kind of really like that. Ruby has a strong gem to support this (https://github.com/guard/rb-inotify) and what I could do is establish a /tmp/lambdakiq-handled
file as an event to signal to the extension it can in turn signal back to the runtime process to process the queue. Some ideas:
- Maybe use the
requestId
in the file contents (or other data like queue length) as a way to pass data to the extension. This is how the above article used post-invoke hooks. - When implementing this we would have to add a logical timeout loop in the extension to not poll/wait for the app/runtime process past the timeout. Thankfully the invoke event data gives us
deadlineMs
.
{
"eventType": "INVOKE",
"deadlineMs": 1624399969622,
"requestId": "59358f44-900e-41f9-ba63-806e585dc418",
"invokedFunctionArn": "arn:aws:lambda:us-east-1:831702759394:function:lambdapunch-main-Lambda-5iePWQ1aaoaz:live",
"tracing": {
"type": "X-Amzn-Trace-Id",
"value": "Root=1-60d26025-01eef3a30f72afd16dfb2982;Parent=7905e3756d42aff7;Sampled=0"
}
}
from lamby.
Related Issues (20)
- qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2 for M1 HOT 6
- libmysqlclient.so.18: cannot open shared object file: No such file or directory HOT 3
- Documentation on how to Add Middleware (V4) HOT 2
- Document what exact AWS-managed policies are needed for the access key HOT 5
- SIGTERM SignalException: Init error when loading handler HOT 5
- Codespaces Share HOT 1
- Don't set percent encoded queries into Rack Environment's `QUERY_STRING` HOT 1
- cannot load such file -- config/environment HOT 3
- Too slow even with provisioned provisioned concurrency HOT 6
- Opinionated OpenID Connect HOT 2
- Postgres ? HOT 2
- Ensure VPC Lattice Events Work HOT 2
- Use Image Devcontainer Step in GitHub Actions
- Unable to run my app or the cookiecutter... getting error `materialize': Could not find HOT 14
- Performance vs traditional server HOT 4
- uninitialized constant Rack::RACK_MULTITHREAD HOT 5
- Local Development
- Access to filesystem - Errno::EACCES (Permission denied
- The "existing" link on the Quickstart page does not link to instructions on working with an existing Rails app
- Had to fix bin/deploy script due to GLIBC 2.33 not found error. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lamby.