contribsys / einhorn Goto Github PK
View Code? Open in Web Editor NEWEinhorn: the language-independent shared socket manager
License: MIT License
Einhorn: the language-independent shared socket manager
License: MIT License
Just curious.
See the Sidekiq repo for an example project which is already done.
Ran into a problem when upgrading its children process to a new version, if the new children error out immediately (due to a bug). Because they error pre-ack, the old children stick around and handle requests (yay!)
However, einhorn continuously tries to start the new children. It should possibly back off, and almost certainly notify somehow. If I hadn't happened to be doing maintenance on the server in question, who knows how long we would have gone without noticing.
This can specifically be an issue with code preloading, which can, e.g., establish database connections and such.
I'm using Einhorn installed via gem install einhorn
, but it seems new releases after 0.6.4
version are not yet available, do you know when they will be?
config
seems to be a no-op, at least for the max_unacked
and max_upgrade_additional
settings
To reproduce:
sudo einhornsh -c $SERVICE -e state
sudo einhornsh -c $SERVICE -e 'config max_unacked: 10
(where 10 is any new value)
sudo einhornsh -c $SERVICE -e state
(and see that nothing changed)
@evan-stripe pointed out that it looks like config
is merging changed values into the wrong place
When doing an upgrade using einhornsh -e upgrade
is there a way to inject enviroment variables?
I'm using workers in Go, and using the os.Getenv()
function without success after upgrading them using the einhornsh
command. And since Einhorn has already started cannot use it to pass enviroment data to its children.
We need an active CI which can test old Rubies for support purposes. See the Sidekiq repo and steal its code as necessary.
Need to make sure this doesn't interfere with preloaded code though.
Hi - I'm running einhorn via upstart and I'm seeing the following exception if there is a stop/start sequence in rapid succeession:
Mar 12 19:13:13 doughnut goliath-channel_api: [MASTER 29092] INFO: Writing PID to /srv/www/channel_api/shared/pids/einhorn.pid
Mar 12 19:13:13 doughnut goliath-channel_api: [MASTER 29092] INFO: Binding to 0.0.0.0:8080 with flags ["rn"]
Mar 12 19:13:13 doughnut goliath-channel_api: [MASTER 29092] INFO: Sending USR2 to []
Mar 12 19:13:13 doughnut goliath-channel_api: /home/deploy/.bundler/channel_api/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn.rb:205:in `bind': Address already in use - bind(2) for 0.0.0.0:8080 (Errno::EADDRINUSE)
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn.rb:205:in `bind'
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn.rb:356:in `block in socketify_env!'
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn.rb:355:in `each'
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn.rb:355:in `socketify_env!'
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn.rb:449:in `run'
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/gems/einhorn-0.6.3/bin/einhorn:327:in `<top (required)>'
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/bin/einhorn:23:in `load'
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/bin/einhorn:23:in `<main>'
When the error is reported, there are no running einhorn processes but netstat -na | grep 8080
reports a few sockets in TIME_WAIT. I thought the r
flag should let einhorn bind despite that?
I've an application that I'd like to run under Einhorn. It currently binds a UNIX socket and then an SSL-terminating nginx sitting on the same box talks to the service over that socket. This is a websocket service so it's designed to have tonnes of open sockets and using UNIX domain sockets for the nginx<->service bridge gives us a lot more headroom. Would it be possible for Einhorn to be able to bind UNIX sockets as well?
I'm happy to write the code for this (have taken a short crack at it already) but I'm a) not great at ruby, and b) would probably need some handholding on making sure the in-place upgrades are OK with the changes. If I do write it would this be something you'd want to merge?
Since Einhorn is a former Stripe project it makes sense that it still has some Stripe-isms lurking around. The phrase Welcome gdb
is one of them, since gdb is a former Stripe. Anyone reading that example in 2022 outside of Stripe, however, probably has no idea and is confused as to why a Ruby shell-like process is invoking a debugger.
"Einhorn: the lanuage-indepedent shared socket manager": "lanuage" should probably be "language".
Right now, einhornsh is almost scriptable, but the resulting output leaves something to be desired:
evan@caron:~/stripe/einhorn$ echo reload | bin/einhornsh time-srv
Welcome evan! You are speaking to Einhorn Master Process 27289 (time-srv)
Enter 'help' if you're not sure what to do.
Type "quit" or "exit" to quit at any time
> Reloading, as commanded
>
^C
In particular, it doesn't exit automatically if it gets an EOF from stdin, and it's pretty verbose.
It would be nice if einhornsh checked if stdin is a tty, and interact in a less verbose, more scripting friendly mode.
I now see this pretty consistently:
[MASTER 34636] ERROR: Could not find any config for exited child 34672! This probably indicates a bug in Einhorn.
Looks like it was introduced in: 673f5a8. Sounds like we had a race before, but I think this means we have a race still.
@ebroder, what would you think about de-racing by making the state_passer live past the SIGCHLD handler install? I'd guess that would mean either (a) installing the SIGCHLD handle before loading state — though I haven't checked if there's a chicken and an egg there — or (b) having some coordination mechanism between the state_passer and the master. Thoughts?
Hi, do you have plans to publish v0.7.5 to rubygems.org?
Thanks.
after calling einhorn -b 127.0.0.1:8000 -q -m manual /myapp
it blocks the shell. and i cant exit.
how to keep einhorn running on background
if i try to exit the einhorn will just die.
do i need to use any other program to accomplish this?
Classifying this as a bug, which maybe isn't strictly accurate, but it definitely impacted our use of einhorn.
We had an issue today using einhorn's preloading where the preloaded code would modify environment variables in a way that changed einhorn's behavior when it re-exec'd itself on upgrade (specifically, changing RUBYOPT
and BUNDLE_GEMFILE
).
I think the right thing for einhorn to do in this situation is to save its environment at startup as part of its state and then restore that environment just before it re-exec's itself.
Some companies will only use gems with a certain license.
The canonical and easy way to check is via the gemspec,
via e.g.
spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']
Even for projects that already specify a license, including a license in your gemspec is a good practice, since it is easily
discoverable there without having to check the readme or for a license file. For example, it is the field that rubygems.org uses to display a gem's license.
For example, there is a License Finder gem to help companies ensure all gems they use
meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough
issue that even Bundler now generates gems with a default 'MIT' license.
If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), github has created a license picker tool.
In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :).
I hope you'll consider specifying a license in your gemspec. If not, please just close the issue and let me know. In either case, I'll follow up. Thanks!
p.s. I've written a blog post about this project
Hi to all!
Einhorn cannot run on jruby due to a lack of fork. Can i make a patch for that? I propose replace fork + exec by posix_spawn for MRI and spoon for jruby.
If einhorn will support jruby it can be proposed for puma instead its cluster mode :)
Hello
I do not know how to achieve a seamless restart Goliath.
Please give me some advice, thank you!
einhorn -n 2 -b 127.0.0.1:3000 ruby app.rb --einhorn
[MASTER 16379] INFO: Writing PID to /var/folders/nc/2jhfmrs55fz7qkqddw5cvtkw0000gn/T/einhorn.pid
[MASTER 16379] INFO: Binding to 127.0.0.1:3000 with flags []
[MASTER 16379] INFO: Launching 1 new workers
[MASTER 16379] INFO: ===> Launched 16380 (index: 0)
[WORKER 16380] INFO: About to exec ["/Users/tumayun/.rvm/rubies/ruby-2.2.3/bin/ruby", "app.rb", "--einhorn"]
[MASTER 16379] INFO: Worker 16380 has been up for 1s, so we are considering it alive.
[MASTER 16379] INFO: Up to 1 / 1 timer ACKs
/Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/runner.rb:107:in `initialize': invalid option: --einhorn (OptionParser::InvalidOption)
from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/application.rb:107:in `new'
from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/application.rb:107:in `run!'
from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/application.rb:133:in `block in <module:Goliath>'
[MASTER 16379] INFO: ===> Exited worker 16380
[MASTER 16379] INFO: ===> Launched 16382 (index: 0)
[WORKER 16382] INFO: About to exec ["/Users/tumayun/.rvm/rubies/ruby-2.2.3/bin/ruby", "app.rb", "--einhorn"]
/Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/runner.rb:107:in `initialize': invalid option: --einhorn (OptionParser::InvalidOption)
from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/application.rb:107:in `new'
from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/application.rb:107:in `run!'
from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/application.rb:133:in `block in <module:Goliath>'
[MASTER 16379] INFO: ===> Exited worker 16382 before it was ACKed
And
einhorn -n 2 -b 127.0.0.1:3000 ruby app.rb -sv
[MASTER 16434] INFO: Writing PID to /var/folders/nc/2jhfmrs55fz7qkqddw5cvtkw0000gn/T/einhorn.pid
[MASTER 16434] INFO: Binding to 127.0.0.1:3000 with flags []
[MASTER 16434] INFO: Launching 2 new workers
[MASTER 16434] INFO: ===> Launched 16435 (index: 0)
[WORKER 16435] INFO: About to exec ["/Users/tumayun/.rvm/rubies/ruby-2.2.3/bin/ruby", "app.rb", "-sv"]
[MASTER 16434] INFO: ===> Launched 16436 (index: 1)
[WORKER 16436] INFO: About to exec ["/Users/tumayun/.rvm/rubies/ruby-2.2.3/bin/ruby", "app.rb", "-sv"]
[16435:INFO] 2015-11-30 23:27:34 :: Starting server on 0.0.0.0:9000 in development mode. Watch out for stones.
[16436:INFO] 2015-11-30 23:27:34 :: Starting server on 0.0.0.0:9000 in development mode. Watch out for stones.
/Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/eventmachine-4de7d0e00fe8/lib/eventmachine.rb:526:in `start_tcp_server': no acceptor (port is in use or requires root privileges) (RuntimeError)
from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/eventmachine-4de7d0e00fe8/lib/eventmachine.rb:526:in `start_server'
from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/server.rb:86:in `block in start'
from /Users/tumayun/.rvm/gems/ruby-2.2.3/gems/em-synchrony-1.0.4/lib/em-synchrony.rb:38:in `block (2 levels) in synchrony'
[MASTER 16434] INFO: ===> Exited worker 16436 before it was ACKed
[MASTER 16434] INFO: Worker 16435 has been up for 1s, so we are considering it alive.
[MASTER 16434] INFO: Up to 1 / 2 timer ACKs, breaking the streak of 1 consecutive unacked workers dying
^C[MASTER 16434] INFO: ===> Exited worker 16435
I haven't quite thought through how this feature should work, but something like "spin up one new worker process, let it serve requests for 30 mins, and then finish the upgrade" might be cool. As well, keeping around the old processes in a paused state could be useful for implementing rollback.
Right now reloads have the potential to eat up a lot of CPU. I believe that's due to loading code. If so, we should probably have Einhorn nice itself, and renice its children back to normal.
E.g.
[MASTER 21022] INFO: Sending TERM to []
And then hangs around for a bit.
einhorn should notice if it signals a process to die and that process doesn't done so.
Or at least it should give you enough hooks to monitor it yourself (a la #2). I'd settle for that.
I'd like to use einhorn to do seamless reloading of some processes but I'm using various programs that expect something different than a SIGUSR2 for a graceful shutdown.
It would be nice if the graceful shutdown signal could be specified on the command line.
Thoughts?
Don't understand exactly what happened here, but einhorn crashed with this exception a while back and has been breakage looping since (on account of workers being left around):
[2013-02-15 02:29:11.596843] /deploy/abba/current/vendor/bundle/ruby/1.9.1/gems/einhorn-0.4.2/lib/einhorn/worker_pool.rb:9:in `block in unsignaled_workers': undefined method `length' for nil:NilClass (NoMethodError)
[2013-02-15 02:29:11.596882] from /deploy/abba/current/vendor/bundle/ruby/1.9.1/gems/einhorn-0.4.2/lib/einhorn/worker_pool.rb:8:in `select'
[2013-02-15 02:29:11.596915] from /deploy/abba/current/vendor/bundle/ruby/1.9.1/gems/einhorn-0.4.2/lib/einhorn/worker_pool.rb:8:in `unsignaled_workers'
[2013-02-15 02:29:11.596949] from /deploy/abba/current/vendor/bundle/ruby/1.9.1/gems/einhorn-0.4.2/lib/einhorn/worker_pool.rb:57:in `old_workers'
[2013-02-15 02:29:11.596981] from /deploy/abba/current/vendor/bundle/ruby/1.9.1/gems/einhorn-0.4.2/lib/einhorn/command.rb:309:in `cull'
[2013-02-15 02:29:11.597014] from /deploy/abba/current/vendor/bundle/ruby/1.9.1/gems/einhorn-0.4.2/lib/einhorn.rb:283:in `run'
[2013-02-15 02:29:11.597046] from /deploy/abba/current/vendor/bundle/ruby/1.9.1/gems/einhorn-0.4.2/bin/einhorn:294:in `<top (required)>'
[2013-02-15 02:29:11.597077] from /deploy/abba/current/vendor/bundle/ruby/1.9.1/bin/einhorn:23:in `load'
[2013-02-15 02:29:11.597109] from /deploy/abba/current/vendor/bundle/ruby/1.9.1/bin/einhorn:23:in `<main>'
@gdb Does that make sense to you? Is this something like a bug where we received a signal before we had the handlers in place for it?
Saw the following in logs when trying to kick off ~40 einhorns at the same time.
cc @zenazn
[2014-10-16 03:26:03.563102] [WORKER 19807] INFO: About to exec ["my", "command"]
[2014-10-16 03:26:03.564304] /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event.rb:163:in `write_nonblock': closed stream (IOError)
[2014-10-16 03:26:03.564331] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event.rb:163:in `break_loop'
[2014-10-16 03:26:03.564348] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command/interface.rb:188:in `block in trap_async'
[2014-10-16 03:26:03.564364] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command/interface.rb:10:in `call'
[2014-10-16 03:26:03.564867] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command/interface.rb:10:in `command_server='
[2014-10-16 03:26:03.564892] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event/command_server.rb:54:in `deregister!'
[2014-10-16 03:26:03.564908] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event/command_server.rb:44:in `close'
[2014-10-16 03:26:03.564924] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event.rb:28:in `block (2 levels) in close_all'
[2014-10-16 03:26:03.564937] from /usr/lib/rbenv/versions/2.1/lib/ruby/2.1.0/set.rb:263:in `each_key'
[2014-10-16 03:26:03.564951] from /usr/lib/rbenv/versions/2.1/lib/ruby/2.1.0/set.rb:263:in `each'
[2014-10-16 03:26:03.564964] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event.rb:27:in `block in close_all'
[2014-10-16 03:26:03.564976] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event.rb:26:in `each'
[2014-10-16 03:26:03.564989] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event.rb:26:in `close_all'
[2014-10-16 03:26:03.565003] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event.rb:34:in `close_all_for_worker'
[2014-10-16 03:26:03.565559] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command.rb:276:in `block in spinup'
[2014-10-16 03:26:03.565785] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command.rb:265:in `fork'
[2014-10-16 03:26:03.565985] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command.rb:265:in `spinup'
[2014-10-16 03:26:03.566199] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command.rb:468:in `block in replenish_immediately'
[2014-10-16 03:26:03.566399] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command.rb:468:in `times'
[2014-10-16 03:26:03.566594] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command.rb:468:in `replenish_immediately'
[2014-10-16 03:26:03.566790] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command.rb:455:in `replenish'
[2014-10-16 03:26:03.566999] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn.rb:469:in `run'
[2014-10-16 03:26:03.567200] from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/bin/einhorn:327:in `<top (required)>'
[2014-10-16 03:26:03.567399] from /stuff/vendor/bundle/ruby/2.1.0/bin/einhorn:23:in `load'
[2014-10-16 03:26:03.567590] from /stuff/vendor/bundle/ruby/2.1.0/bin/einhorn:23:in `<main>'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.