nulogy / gorgon Goto Github PK

Distributed testing for ruby with centralized management

Ruby 99.94% Shell 0.06%

gorgon's Introduction

Gorgon

About

Gorgon provides a method for distributing the workload of running ruby test suites. It relies on amqp for message passing, and rsync for the synchronization of source code.

Installing Gorgon

This tutorial explains how to install gorgon in a sample app.

Installing listener as a Daemon process (Ubuntu 9.10 or later)

run gorgon install_listener from the directory where gorgon.json is
run gorgon ping to check if the listener is running

Gotchas

if you get cannot load such file -- qrack/qrack (LoadError), just add gem 'gorgon', '~> 0.8.4' , :group => :remote_test to your Gemfile, and run tests using bundle exec gorgon
If gorgon install_listener didn't work for you, you can try these steps

Also note that the steps in the tutorial are not meant to work on every project, they will only give you initial settings. You will probably have to modify the following files:

gorgon.json
gorgon_secret.json
{test, spec}/gorgon_callbacks/gorgon_callbacks.rb
gorgon_listener.json (located in your project root or in ~/.gorgon/)

If you modify ~/.gorgon/gorgon_listener.json, make sure you restart the listener.

Configuration

gorgon.json

This file contains project-specific settings for gorgon, such as:

The connection information for AMQP
The connection information for File Server
Information about how clients can rsync the working directory (optional). See more info here
Files that can be excluded by rsync
Callback file containing Ruby code to be used as callbacks
A glob for generating the list of test files
The file used for Originator's logs

See gorgon.json example for an example file.

gorgon_secret.json (optional)

This optional file contains sensitive information such as passwords that cannot be put in gorgon.json.

See gorgon_secret.json example for an example file.

gorgon_listener.json

This file contains the listener-specific settings, such as:

The connection information for AMQP
How many worker slots are provided by this listener
The file used for logs

See gorgon_listener.json example for more details.

Contributing

Read overview architecture

Requirements:

You only need Docker

Prepare your environment

Execute ./run_dev_environment.sh
In a new terminal tab, execute ./run_listener.sh

NOTE: If you make changes changes to listener code, you must restart ./run_listener.sh for those changes to take effect

Running all tests

./run_test.sh

Running gorgon using the `spec/dummy` project

./run_gorgon.sh

Releasing to RubyGems

Update version.rb
Run rake release

Credits

Gorgon is maintained by:

Justin Fitzsimmons
Arturo Pie

Gorgon is funded by Nulogy Corp. Thank you to all the contributors.

gorgon's People

Contributors

Stargazers

Watchers

Forkers

arturopie robertpd jonerikdsuero mmorast waseem nerakdon jmks miguelac fixdocker

gorgon's Issues

Robust source tree syncer and handle rsync errors

Rsync can easily fail since it relies on ssh. Avoid password prompts by using -e 'ssh -o "NumberOfPasswordPrompts 0"' in rsync options (http://serverfault.com/questions/39571/forcing-rsync-to-non-interactive-mode)
Investigate the use of timeouts in rsync opts
Then inform originator if rsync failed by sending a crash message

Listener cleans up temporary directory after job is complete

Allow users to 'ping' listeners

Create 'gorgon ping' command to check what listeners are running waiting for jobs and what version of gorgon each listener is running

Job worker can set up external environment specific external dependencies

We might want to configure this in a project-agnostic way. Not all projects will need a database-per-worker setup like Packmanager does. Other projects may require other things. Not sure how hydra currently handles this, but it might be worth taking a look.

Originator responds to ctrl-C and cancels the jobs before shutting down

Listener logs to somewhere

Create a readme for the project

Exceptions within Worker#run_file are not reported

I think this is related to the fact that #run_file is executed in another thread via EventMachine#defer. Execution fails silently, and deadlocks the whole app.

builds source tree path programmatically if this is not specified in gorgon.json

If originator builds source tree path programmatically, rsync will be less prone to errors.

Show number of workers and hosts beside progressbar

Job creation task assembles a rsync command for synchronizing the source directory

Store failed files somewhere so that user can use a task to run them

When cancelling a job, print files and machines still running

How do we test a test framework?

Listener can be backgrounded

Job creation task creates a job-specific queue for test result replies

When cancelling a job, add the still-running files to gorgon-failed-files.json

Use Timeouts for each callback and rsync

Allow the user to specify timeouts for each callback in gorgon.json. Use the timeout, so the processes do not hang for ever if one of the callbacks blocks for ever.

Better report of exceptions

How to reproduce: raise an exception from after_sync callback. The message shown in originator is ugly and hard to read. Output 'exception' messages in a similar way to gorgon 'crash' messages

Job creation task creates a clone of the originating machine's current repository

The job won't necessarily run immediately, so it seems to make sense to be able to continue working in the working directory. Hence the need for a cloned directory.

Job creation task blocks and reports results in realtime

Job worker can tear down environment specific external dependencies

It should make a good effort to tear down external dependencies even in an error condition that causes the worker to halt.

Job listener only processes a job if there are test files remaining for that job

Create 'gorgon install' command

'gorgon install' will:

mkdir ~/.gorgon
cd ~/.gorgon
cp src/packmanager/gorgon_listener.json.example gorgon_listener.json
(probably ask for number of worker_slots, or find out how many cores the PC has)
create Procfile
sudo foreman export to upstart
sudo start gorgon-listener
add 'start on runlevel [2345]' to '/etc/init/gorgon-listener.conf'

(http://michaelvanrooijen.com/articles/2011/06/08-managing-and-monitoring-your-ruby-application-with-foreman-and-upstart/)

NOTE: We will have to add 'foreman' dependency to gorgon

EDIT: It ended up doing:

mkdir -p ~/.gorgon
cd ~/.gorgon
creates gorgon_listener.json in this directory
creates /etc/init/gorgon.conf
sudo start gorgon

I realized we don't need foreman

Job listener daemon subscribes to job broadcasts

Job creation task creates a job message that is sent to all job subscribers

Listener terminates workers when a job is canceled

Create after_sync callback

after_sync callback will be useful to run commands like 'bundle install'

Use heuristics to run files in order

Collect and store run time for each test file so that next time it runs, it distributes tests in order from slowest to fastest.

Cancel a job

Job listener creates job workers

The worker count will need to be configured for each node. How should we do this?

Allow users to update all listeners to a new gem version

Add a gorgon command that updates all listeners to a gorgon version

Listener should delete queues and clean up before it exits

Optimize rsync by ignoring unnecessary directories and using compression

Job worker runs test file

Redirect Woker Manager and workers output to files.

Worker Manager is receiving all the output generated by all workers when they invoke task 'db:reset'. This is filling up listener-worker_manager pipe very quick, and blocks workers writes. The work around was to use 'min_messages: warning' in database.yml for remote_test environment, so the output is minimized.

The final solution should be to redirect standard output and error of worker manager and worker to a temporary file. Then, if a crash happens, sends few last lines of the file to originator

Create a logger for the Workers

Worker notifies the originator when it starts a file

Allow gorgon listeners to rsync without having passwordless ssh access to originator

Send ~/.ssh/id_rsa.pub in the job definition so rsync doesn't fail.

Originator sends ~/.ssh/id_rsa.pub, and listeners creates a temporary file with its content and use the option:

'--rsh='ssh -i /path/to/tmp/file host' ...'

Job worker reports test results to the reply queue

Remove rake and replace it with a execuatable

The current rake architecture requires us to ship a Rakefile with the source directory. It would be much easier and more generic to simply use the executable infrastructure that's included with rubygems.

Listener should restart itself if it crashes

Since the listener should never exit, it should try to recover itself from a crash and restart.

Job creation task creates a job-specific queue and populates it with test filenames

Make a gem

Job creation task replaces any originating's machine's existing task

Assuming we allow the working directory to be modified while a job is queued, there may be a need to replace an existing unprocessed job with a new one.

Job listener synchronizes the working directory

Job listener does not process job broadcasts unless workers are available

Listener reports worker crashes as soon as they happen

Add message outputter

Take the payload from the listeners and output them nicely.

Rewrite worker and listener to use polling

The eventmachine-based API currently in use adds a lot of complexity and makes Gorgon very difficult to test. It would significantly reduce the complexity of the programming to use a polling/synchronous style of API, and this will not significantly affect the performance.