The omnibus from akiernan

Introduction

omnibus helps you build self-installing, full-stack software builds. For
example, we want to distribute the Chef client pre-bundled with its
dependencies: ruby, required gems, etc. The result is a single installable
shell-script that can be executed on the client side, installing Chef, and
configuring it.

omnibus starts with the idea of a “project”, which is comprised of a name,
version, and a build order. The build order references various “software”
packages, which are compiled and installed to a specific path on the
filesystem. (For the chef-full project, this would be /opt/opscode).

Using omnibus

To start using omnibus, you need a few dependencies:

Java – either the Sun JDK, or the Open JDK
Leiningen
Your operating systems base build utilities – gcc, autoconf, etc.

To install Leiningen, follow the instructions here: http://github.com/technomancy/leiningen

Once you have Leiningen installed, run:

$ lein deps

From the top of the omnibus source tree.

To start a build:
(make sure the project includes the ‘prep’ step which will create the appropriate directories)

$ lein repl
omnibus.core=> (build-fat-binary “chef-full”)

The results will be in ./pkg when the build is complete.

Adding software to omnibus

To add new software to omnibus, you want to extract it and commit the source in the ‘./source’ directory. Then add a configuration to config/software. For example, to build gdbm:

1. Download the source for gdbm-1.8.3 2. Unpacked it in ./source 3. Removed the tarball 4. Added the following configuration in config/software/gdbm.clj


(software "gdbm" :source "gdbm-1.8.3"
                 :steps [["./configure" "--prefix=/opt/opscode/embedded"]
                         (if (is-os? "darwin") ["perl" "-pi" "-e" "s/BINOWN = bin/BINOWN = root/g" "Makefile"])
                         (if (is-os? "darwin") ["perl" "-pi" "-e" "s/BINGRP = bin/BINGRP = wheel/g" "Makefile"])
                         ["make"]
                         ["sudo" "make" "install"]])

The :source field should be the name of the directory created by unpacking the tarball. The steps are a vector of vectors, each of which corresponds to a build step. These steps are passed directly to clojure.java.sh/sh, which will execute them directly. You can use two handy functions to make the build be different based on operating system or machine architecture: is-os? and is-machine?.

Adding software to a project

In config/projects, you will find the actual project definitions themselves. These contain a name for the project, a version number, and the build order. For the chef client:

(project
  "chef-full"
  "0.9.8"
  [ "zlib" "libiconv" "db" "gdbm" "ncurses" "openssl" "libxml2" "libxslt" "ree" "rubygems" "rsync" "chef" ])

The build order will run in order.

Enjoy!

The full build process

If you want to kick off a full build, the easiest way is with:

$ sudo lein run --project-name "chef-full" --bucket-name FULL_STACK --s3-access-key "ACCESS_KEY" --s3-secret-key "SECRET_KEY"

The resulting build artifact will be dropped off in the S3 bucket named above. If you want to submit a build to us, drop it off in your own bucket, and let us know where and how to test the final product.

Operating System Prep

Fedora 15

yum update -y
yum install -y gcc gcc-c++ kernel-devel make autoconf flex bison git java-1.6.0-openjdk java-1.6.0-openjdk-devel ruby ruby-libs ruby-devel ruby-docs ruby-ri ruby-irb ruby-rdoc ruby-mode rubygems rpm-build libxml2 libxml2-devel libxslt libxslt-devel wget automake help2man libtool gettext texinfo which
gem install fpm ohai --no-rdoc --no-ri
wget https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

SL 6

yum update -y
yum install -y gcc gcc-c++ kernel-devel make autoconf flex bison git java-1.6.0-openjdk java-1.6.0-openjdk-devel ruby ruby-libs ruby-devel ruby-docs ruby-ri ruby-irb ruby-rdoc ruby-mode rubygems rpm-build libxml2 libxml2-devel libxslt libxslt-devel wget automake help2man libtool gettext texinfo
gem install fpm ohai --no-rdoc --no-ri
wget https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

CentOS 5

wget http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
rpm -Uvh epel-release-5-4.noarch.rpm
yum update -y
yum install -y gcc gcc-c++ kernel-devel make autoconf flex bison git java-1.6.0-openjdk java-1.6.0-openjdk-devel ruby ruby-libs ruby-devel ruby-docs ruby-ri ruby-irb ruby-rdoc ruby-mode rubygems rpm-build libxml2 libxml2-devel libxslt libxslt-devel wget automake help2man libtool gettext texinfo
gem install fpm ohai --no-rdoc --no-ri
wget --no-check-certificate https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

Debian 6

apt-get update -y
apt-get -y install build-essential binutils-doc autoconf flex bison git-core openjdk-6-jdk default-jdk ruby ruby1.8 ruby1.8-dev rdoc1.8 irb1.8 ri1.8 libopenssl-ruby1.8 rubygems libtool dpkg-dev libxml2 libxml2-dev libxslt1.1 libxslt1-dev help2man gettext
update-java-alternatives -s java-6-openjdk
gem install fpm ohai --no-rdoc --no-ri
ln -s /var/lib/gems/1.8/bin/* /usr/local/bin
wget https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

Ubuntu 10.04

apt-get update -y
apt-get -y install build-essential binutils-doc autoconf flex bison git-core openjdk-6-jdk default-jdk ruby ruby1.8 ruby1.8-dev rdoc1.8 irb1.8 ri1.8 libopenssl-ruby1.8 rubygems libtool dpkg-dev libxml2 libxml2-dev libxslt1.1 libxslt1-dev help2man gettext texinfo
update-java-alternatives -s java-6-openjdk
gem install fpm ohai --no-rdoc --no-ri
ln -s /var/lib/gems/1.8/bin/* /usr/local/bin
wget https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

Ubuntu 11.04

apt-get update -y
apt-get -y install build-essential binutils-doc autoconf flex bison git-core openjdk-6-jdk default-jdk ruby ruby1.8 ruby1.8-dev rdoc1.8 irb1.8 ri1.8 libopenssl-ruby1.8 rubygems libtool dpkg-dev libxml2 libxml2-dev libxslt1.1 libxslt1-dev help2man gettext texinfo
update-java-alternatives -s java-6-openjdk
gem install fpm ohai --no-rdoc --no-ri
ln -s /var/lib/gems/1.8/bin/* /usr/local/bin
wget https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

Joyent Smartmachine

pkgin up
pkgin in autoconf-2.68 gtexinfo bison flex
wget --no-check-certificate https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

OpenIndiana 148

#!/bin/bash
# Notes
# pkg:/developer/gcc-3 is required over gcc-43, as Ruby was built with gcc3.
# Ruby build requires the same compiler to build extensions as was used to build
# Ruby itself.
# Need to use /opt/opscode/embedded/bin/autoconf?
# ln -s /usr/bin/automake-110 /usr/bin/automake
pkg install pkg:/developer/build/autoconf \
pkg:/developer/build/automake-110 \
pkg:/developer/build/gnu-make \
pkg:/developer/build/libtool \
pkg:/developer/build/make \
pkg:/developer/gcc-3
pkg:/developer/java/jdk \
pkg:/developer/lexer/flex \
pkg:/developer/library/lint \
pkg:/developer/linker \
pkg:/developer/macro/cpp \
pkg:/developer/object-file \
pkg:/developer/parser/bison \
pkg:/developer/versioning/git \
pkg:/file/gnu-coreutils \
pkg:/library/libxml2 \
pkg:/library/libxslt \
pkg:/ruby-dev \
pkg:/runtime/ruby-18 \
pkg:/system/header \
pkg:/system/library/math \
pkg:/system/library/math/header-math \
pkg:/system/xopen/xcu4 \
pkg:/text/gnu-gettext \
pkg:/text/texinfo \
pkg:/web/wget \
pkg:/compatibility/ucb \
pkg:/text/gnu-sed
gem update
gem install json_pure
mv /var/ruby/1.8/gem_home/gems/json-1.4.6 /tmp ; ln -s /var/ruby/1.8/gem_home/gems/json-1.5.1 /var/ruby/1.8/gem_home/gems/json-1.4.6
gem install ohai -v=0.5.8
gem install fpm

FreeBSD 8.2

portsnap fetch update
cd /usr/ports/ports-mgmt/portupgrade && make && make install # Be sure that BATCH=yes
portupgrade -a -m BATCH=yes # Upgrade all ports
portinstall lang/python26 devel/autoconf textproc/flex devel/bison devel/git java/openjdk6 lang/ruby19 devel/gmake
gem update
gem install ohai
gem install fpm
wget --no-check-certificate https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

Windows

Run in PowerShell 2.0+. “Set-ExecutionPolicy RemoteSigned” must be set on the target machine if run as a ‘ps1’ script (vs pasted into PowerShell).

$client = new-object System.Net.WebClient

# download/install Git
$client.DownloadFile("http://msysgit.googlecode.com/files/Git-1.7.6-preview20110708.exe","$ENV:TEMP\gitinstaller.exe")
Start-Process -FilePath "$ENV:TEMP\gitinstaller.exe" '/verysilent /dir="C:\git"' -Wait
[Environment]::SetEnvironmentVariable( "Path", "$ENV:PATH;C:\git\cmd", [System.EnvironmentVariableTarget]::Machine )

# download/install Ruby
$client.DownloadFile("http://files.rubyforge.vm.bytemark.co.uk/rubyinstaller/rubyinstaller-1.8.7-p334.exe","$ENV:TEMP\rubyinstaller.exe")
Start-Process -FilePath "$ENV:TEMP\rubyinstaller.exe" '/verysilent /dir="C:\ruby" /tasks="assocfiles,modpath"' -Wait

# add Ruby and Git to the PATH and persist it
$ENV:PATH = "$ENV:PATH;C:\ruby\bin;C:\git\cmd"
[Environment]::SetEnvironmentVariable("PATH", $ENV:PATH, "Machine")

# download/install Ruby DevKit
$client.DownloadFile("http://cloud.github.com/downloads/oneclick/rubyinstaller/DevKit-tdm-32-4.5.1-20101214-1400-sfx.exe","$ENV:TEMP\rubydevkit.exe")
New-Item C:\DevKit -type directory
Copy-Item $ENV:TEMP\rubydevkit.exe c:\DevKit
Start-Process -FilePath "C:\DevKit\rubydevkit.exe" '-y' -Wait
Start-Process -FilePath "ruby.exe" 'C:\DevKit\dk.rb init' -Wait
Start-Process -FilePath "ruby.exe" 'C:\DevKit\dk.rb install' -Wait

# install required gems
Start-Process -FilePath "gem" 'install win32-api win32-service --platform=mswin32 --no-rdoc --no-ri' -Wait
Start-Process -FilePath "gem" 'install win32-open3 rdp-ruby-wmi windows-api windows-pr --no-rdoc --no-ri' -Wait
Start-Process -FilePath "gem" 'install ohai chef --no-rdoc --no-ri --verbose' -Wait

# check out omnibus code
Start-Process -FilePath "git" 'clone git://github.com/opscode/omnibus.git C:\omnibus' -Wait -NoNewWindow

# this will allow execution of local scripts (.ps1 files)
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Force

p.

Bundled software and licensing

All software under the “source” directory retains the license by which it was originally distributed.

akiernan / omnibus Goto Github PK

omnibus's Introduction