makeusabrew / horde Goto Github PK

Unleash hordes of zombies to tear your test suite to shreds in record time!

CoffeeScript 75.71% Perl 4.29% Racket 1.45% JavaScript 1.14% Shell 17.42%

horde's Introduction

horde

Unleash hordes of zombies to tear your test suite to shreds in record time!

What?

Horde is designed to solve a fairly specific problem: speeding up very long running integration test suites which use the Mocha test framework to run the Zombie.js headless browser against a fairly typical LAMP powered website.

It assumes that the bottleneck in such a scenario is largely the DB layer; both in terms of priming it with test data and in terms of contention over it: parallelism is hard when multiple test suites are all squabbling over the same test database.

How?

By splitting your test suite up into multiple smaller ones and running them in parallel inside individual docker containers. This not only completely isolates each database instance but also the entire LAMP server itself. The master horde process stitches the results of each container's test results into a JUnit compatible XML file making horde perfect for use in continuous integration environments. As an added bonus, running multiple processes will make much better use of multi-core processors than a single NodeJS instance.

Why?

Because running large integration suites in parallel - in my experience at least - can yield huge speed increases. A sample suite of 1,062 tests which previously took around 9 minutes 20 seconds to run now executes at best in 1 minute 21 seconds - over 85% faster.

Getting started

PLEASE NOTE: horde is in a serious state of flux; it's not practical keeping the docs up to date so they will most likely be wrong!

Docker

install docker if you haven't already
add your user to the docker group so you don't have to keep running every docker command with sudo (and since horde spawns docker sub processes, it means you don't have to run that with sudo either)
pull down the horde docker image: docker pull makeusabrew/horde

Horde

via npm

[sudo] npm install -g horde

from source

clone this repository
run npm install
run npm install -g coffee-script if you don't already have it

Configuration (LAMP environment setup)

In order to make the makeusabrew/horde docker image reusable you need to give it a hand by creating an apache configuration file it'll look for upon initialisation. For the time being this configuration file must match a specific filename so that the horde image can find it. This directory can live anywhere but since it'll probably be specific to the project you're testing it's advisable to keep it there under a horde/ directory. This also helps when running the horde script as it'll look there first for any configuration files.

apache.conf

This is the apache configuration file needed in order to run your site. At run time it'll be linked as the only file in /etc/apache2/sites-enabled/ and as such will act as the container's default (and only) host. In my usage so far this has amounted to a single VirtualHost entry adapted from the site I'm testing.

Assumptions

Since our containers spawn a completely isolated LAMP stack, they make a few key assumptions:

the source directory you provide when running the horde script (discussed later) will be mounted as /var/www (e.g. Apache's default document root)
the schema you provide will be run against horde_test as root with no password
we don't inject any /etc/hosts entries into the container, but as your site is the only one available it'll respond to requests (from within the container) to http://localhost

These assumptions mean that:

your apache.conf file should specify any relevant directives with /var/www as the root. For example, if you have a 'public' folder which is typically your document root, instead of DocumentRoot /path/to/myproject/public, use /var/www/public
your site's test configuration should point to a database named horde_test, accessed by user root with no password (or a blank password)
if your site generates absolute URLs, the host name in test mode should be localhost

Unleashing

$ horde --help

  Usage: horde [options]

  Options:

    -h, --help           output usage information
    -p, --procs <n>      Number of containers to spawn [4]
    -o, --output [file]  XML file to write JUnit results to
    -s, --source [dir]   Source directory to mount
    -c, --config [dir]   Configuration directory to mount [--source/horde]
    -i, --image [image]  Docker image to use [makeusabrew/horde]

--source

Default: process.cwd()

An absolute path to a project which itself contains a test/ directory, i.e. one should be able to run mocha from within --source and expect it to run and find some appropriate tests. This directory will be mounted within each container as /var/www.

--config

Default: horde/ sub directory of --source option

An absolute path to a directory containing the aforementioned apache.conf apache configuration file. This directory can live anywhere but since it'll probably be specific to the project you're testing it's advisable to keep it there, hence why the horde/ sub directory is checked first.

--procs

Default: 4

This controls how many docker containers to spawn and is limited only by your host machine and the complexity of your test suite. Experiment! My sample suite seems to work well with up to 20 containers on a Quad Core, 16Gb Linux machine, but any more and tests start failing unpredictably.

--output

Default: N/A

If present this controls where the combined results of all test suites will be written to in JUnit compatible (e.g. CI friendly) XML.

--image

Default: makeusabrew/horde

If you've built your own custom horde image you can pass it here.

Sample output

Please see this showterm recording for a real life test run against the 9½ minute suite referenced in this readme.

horde's People

Contributors

Stargazers

Watchers

Forkers

outrunthewolf

horde's Issues

Reorganise test file searching and approximate test counts into adapters

At the moment the process to fetch the test files and work out their approximate number of tests is hard-coded to look for coffeescript files in mocha format. That's fine, but let's introduce the concept of 'adapters' or similar and make mocha/coffee the default (or only) one. This then adds scope to create more adapters which find their files and test counts (and indeed execute their test suites) in totally different ways.

I'm dimly aware that at the moment this concept would bleed into the actual docker image (since it runs a hard-coded ./_mocha binary) but I think it can be teased out over time.

Add support for JS test suites

Two things prevent this at the moment:

We do a hard coded search for test/*.coffee
Our pattern matching to estimate test counts assumes CS syntax

Ideally we could auto detect this, inform the user, and offer them a way to specify it if we got it wrong.

Improve communication from container up to parent process

At the moment the boot script run by each container spawns the necessary processes and attempts to prefix any output from them to make the logged information helpful to users. However, this makes it more difficult for the parent process (which ultimately is the main way to run it) to parse the output. Need to think of a way to improve this.

Docker container processes frequently being zombied

This is wholeheartedly related to #2 but worth splitting out into its own issue - we must get to the bottom of why the combination of child processes aren't naturally exiting. Forcefully calling various .exit() methods (on children and parents) is leaving some docker containers running, and/or (not sure if it's a 1:1 mapping) several zombie node processes which seem to just chew up load.

See moby/moby#1320 for something which sounds vaguely similar.

Just need to step through layer step by step and hunt this down once and for all.

Rework container boot.coffee logic

At the moment each particular docker image runs a specific boot.coffee - which in turn spawns some services, waits a bit then runs a very specific local test runner.

Haven't got this worked out by any means but does this have to be part of the image? Can't it be mounted at container run time for maximum flexibility?

If mounted at run time we need to rework the config directory logic into horde/conf/ and horde/run/ or similar.

What we don't want to do is make people specify their own runner for standard stuff.

I can't help feeling like there's a lot of meta stuff possible here - e.g. an npm install -g horde in the actual Dockerfile... just all needs working out.

Create simple PHPUnit adapter

I've been trying this using the wordpress develop test suite but it's a bit complex for a starting point - something like paynedigital.com would be easier (and have the advantage of demonstrating an open source real-world usage of horde).

This will involve pushing an enormous amount of logic down into adapters as the way PHPUnit communicates data back up to the parent process is very different, as is the way we'll need to detect test suite 'end' events etc etc. Thinking about it it'd be nice to make all adapters EventEmitters so they emit consistent events like 'start', 'test', 'end', and the means with which they generate those events are then simply private implementation details.

PHPUnit supports a log-json argument but whilst it can append to stdout, it doesn't seem to replace normal output - so this needs thinking about in terms of how that data can be captured. If we write it to a tmp file, this will need a bespoke boot.coffee script (not a problem given that it needs a custom docker image anyway).

Can be split out into several sub issues if need sbe.

Default to taking --source from CWD

As title; this would mean the app could be run at best with no arguments.

Hook into container's exit signal

At the moment for a reason I can't quite track down, the combination of child_process (in the main executable) spawning a docker container, itself spawning a child_proces is causing the exit signal not to properly be picked up in the main exec when each docker instance exits. The instances are exiting (I think) but we're never getting on "exit". This means at the moment the way we detect our containers finishing is dirty and reliant on mocha telling us when a test suite is finished (which isn't exactly the same as the docker container exiting). I'd like to sort this eventually.

Package horde as an NPM module

This would mean users could install it by simply running npm install -g horde

In tandem with #6 this means users could simply:

cd /path/to/my/project
horde

(Assuming their project followed some conventions, e.g. had a horde/ directory).