Parallel package builds about st2-packages HOT 7 CLOSED

Rake + sshkit work nice. I've managed to extend sshkit behavior to capture output of concurrent commands. This was a bit hard, since it has required to implement threaded dispatcher which consumes output of concurrent processes, mangles it and finally outputs in a nice and readable way.

Talking about batch tools and parallel ssh command executors, I can say none of them supports parallel execution on a "single host". Neither sshkit nor fabric.

So what I come with is rake with a build-in enhanced sshkit which allows a user easily define parallel jobs and get all the power of ssh connections via sshkit. Here's an example DSL:

require 'rake'
require './rake/remote'

ssh_options = {
   keys: %w(/root/.ssh/busybee),
   auth_methods: %w(publickey)
}


buildnode = ENV['BUILDNODE']
# testnodes = ENV['TESTNODES']

Remote.output_verbosity = :debug
remote = Remote.new(ssh_options)

# =============================================================================

# Run shellout dispatcher thread
ShellOut.run

# Flushes shellout when tasks are finished, finalization is required
# to write out any messages left in the queue.
multitask :build => [:first, :second] { ShellOut.finalize }
task default: :build

task :first do
  remote.ssh hostname: buildnode do
    options label: 'build st2common'
    execute :bash, '-c', %q("ls -l /bin | head -n15")
  end
end

task :second do
  remote.ssh hostname: buildnode do
    options label: 'build st2auth'
    execute :bash, '-c', %q("ls -l /bin | head -n15") 
    execute :bash, '-c', %q("echo fail it; /bin/false"), label: 'fail it'
    execute :true
  end
end

This kind of DSL makes is easy to understand and far much better then any killing bash stuff, just have a look at the picture of parallel tasks example:

from st2-packages.

^^ 👍 Really nice output!

from st2-packages.

Okay this was merged into master. Build for wheezy takes 9 min on a 2CPU instance!

from st2-packages.

Cool thing to have 👍

Probably candidate for closing or you plan to extend it like build 1 package = 1 node approach for future horizontal scaling?

from st2-packages.

Okay, we've reached the same performance on CircleCI. So far I'm satisfied with the current results, further improvements can be delayed for now.

from st2-packages.

implemented

from st2-packages.

https://stackstorm.slack.com/archives/stackstorm/p1454357997003953

from st2-packages.