Giter Club home page Giter Club logo

pipeline-templates's Introduction

Pipeline Templates

This repository collects up all that tribal wisdom we've gained from building Concourse Pipelines for various different purposes. Each template provides a base ci/pipeline.yml that structures the pipeline, ci/settings.yml to override any parameters defined by the template, a set of scripts (in ci/scripts/) that are referenced from the pipeline, and a ci/repipe utility for putting all the pieces together into a functioning Concourse pipeline.

Setup is straightforward:

git clone https://github.com/starkandwayne/pipeline-templates
cd pipeline-templates
./setup <template> ~/bosh/my-new-boshrelease

Alternatively, you can initialize the pipeline from inside the target repository:

cd code/my-buildpack
~/code/pipeline-templates/setup <template>

The first argument to setup is the template you want to use. Currently available templates:

  • bash
  • boshrelease
  • genesis-kit
  • buildpack
  • cfpush
  • docker/base
  • docker/ext-tests
  • go
  • helm

Don't let the name fool you! ./setup can also be used to update an existing templated pipeline ci/ directory to pick up new changes made to the templates.

Once you've set up your repository, you'll need to fill in your ci/settings.yml file with any parameters the template required:

cd code/my-project
ci/repipe # Attempt to update the pipeline config,
          # spitting out errors for missing parameters
vi ci/settings.yml # fill in the missing parameters
ci/repipe # Deploy the pipeline config!

Dependencies

The ci/repipe script uses Spruce to merge ci/pipeline.yml and your bespoke ci/settings.yml.

On MacOS/Homebrew:

brew install starkandwayne/cf/spruce

On Debian/Ubuntu:

wget -q -O - https://raw.githubusercontent.com/starkandwayne/homebrew-cf/master/public.key | apt-key add -
echo "deb http://apt.starkandwayne.com stable main" | tee /etc/apt/sources.list.d/starkandwayne.list
apt-get update

apt-get install spruce

The ci/repipe for genesis-kits also requires jq, which can be installed via brew on MacOS or apt-get on Debian/Ubuntu.

And Now, The Templates!

docker/base and docker/ext-tests

For building Docker images, with tests. This template comes in two flavors: docker/base (where the unit tests are inside the Docker image) and docker/ext-tests, where the tests live outside of the image.

Docker Pipeline

go

Takes a Go software project repository, runs unit tests and (when the manual shipit job is run) releases to Github.

Go Project Pipeline

boshrelease

Tries to create a BOSH release from the repository, upload it to a hosted BOSH-lite for viability testing, and (when the manual shipit job is run) releasing it to Github with a tarball artifact, and also uploading that release tarball to S3.

BOSH Release Pipeline

genesis-kit

Builds a pipeline to create a pipeline that builds a release candidate, runs it through spec tests and spec-check, then any deployment, upgrad and acceptance tests. Passing those, it prepares the release notes based on commit messages, and then can be manually released. It also includes a manual prerelease after the initial RC is built.

There is also provision for pulling upstream dependencies and bumping semantic version components.

Read the README.md file under genesis-kit for customization options. As written, it is based on the cf-genesis-kit.

Genesis Kit Pipeline

buildpack

Runs unit and integration tests on a Cloud Foundry buildpack, and can release it to Github.

Buildpack Pipeline

helm

Builds a docker image and uploads Helm chart to an S3 bucket

Helm

ci/settings.yml

You will need to customize your pipeline with information about your CI, your Amazon AWS credentials + S3 bucket for storing assets + version file, your Slack account, etc.

You will create and maintain ci/settings.yml for this.

Try very very hard to not modify ci/pipeline.yml. Instead, use ./setup to update ci/pipeline.yml with new changes from this repo. If you do need to modify ci/pipeline.yml please feel welcome to submit PRs to this repo so that we can merge them and share them with everyone.

At Stark & Wayne we store our credentials for pipelines in Vault. We use the spruce syntax (( vault "path1" )) to dynamically fetch these values during ci/repipe. Recently Concourse CI has added native support for Vault, so we will investigate this in the future. Or you could try to use it and let us know how it goes!

Here is an example ci/settings.yml from the eden CLI project (uses the go template):

---
meta:
  name: eden
  target: sw
  url:     https://ci.starkandwayne.com

  initial_version: 0.5.0

  go:
    binary: eden
    cmd_module: .

  aws:
    access_key: (( vault "secret/aws/starkandwayne-s3:access" ))
    secret_key: (( vault "secret/aws/starkandwayne-s3:secret" ))
    region_name: eu-central-1

  slack:
    webhook: (( vault "secret/pipelines/eden/slack:webhook" ))
    channel: "#eden" # https://openservicebrokerapi.slack.com/messages/C6Y5A2N8Z/
    username: starkandwayne-ci
    icon:     https://www.starkandwayne.com/assets/images/shield-blue-50x50.png

  github:
    owner: starkandwayne
    repo: eden
    access_token: (( vault "secret/pipelines/shared/github:access_token" ))
    private_key: (( vault  "secret/pipelines/shared/github:private_key" ))

Bonus, we use https://github.com/starkandwayne/safe as our CLI to interact with Vault.

For example, to populate the (( vault "secret/pipelines/eden/slack:webhook" )) value in Vault:

safe set secret/pipelines/eden/slack webhook=https://hooks.slack.com/services/T2S1X7xxx/B6Y5A7xx/0nP7jxxx

pipeline-templates's People

Contributors

drnic avatar bgandon avatar geofffranks avatar jhunt avatar dennisjbell avatar bertrand-lupart avatar lnguyen avatar nacc avatar norman-abramovitz avatar 7hunderbird avatar starkandwayne-bot avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.