Giter Club home page Giter Club logo

captain's Introduction

Build Status Coverage Status

Introduction

Captain - Convert your Git workflow to Docker containers ready for Continuous Delivery

Define your workflow in the captain.yaml and use captain to your Continuous Delivery service to create containers for each commit, test them and push them to your registry only when tests passes.

  • Use captain build to build your Dockerfile(s) of your repository. If your repository has local changes the containers will only be tagged as latest, otherwise the containers will be tagged as latest, COMMIT_ID & BRANCH_NAME. Now your Git commit tree is reproduced in your local docker repository.
  • Use captain test to run your tests
  • Use captain push to send selected images to the remote repository

From the other side, you can now pull the feature branch you want to test, or create distribution channels (such as 'alpha', 'beta', 'stable') using git tags that are propagated to container tags.

intro

Installation

To install Captain, run:

curl -sSL https://raw.githubusercontent.com/harbur/captain/v1.1.3/install.sh | bash

You will need to add ~/.captain/bin in your PATH. E.g. in your .bashrc or .zshrc add:

export PATH=$HOME/.captain/bin:$PATH

Captain.yml Format

Captain will automatically configure itself with sane values without the need for any pre-configuration, so that it will work in most cases. When it doesn't, the captain.yml file can be used to configure it properly. This is a simple YAML file placed on the root directory of your git repository. Captain will look for it and use it to be configured.

Here is a full captain.yml example:

hello-world:
  build: Dockerfile
  image: harbur/hello-world
  pre:
    - echo "Preparing hello-world"
  post:
    - echo "Finished hello-world"
hello-world-test:
  build: Dockerfile.test
  image: harbur/hello-world-test
  pre:
    - echo "Preparing hello-world-test"
  post:
    - echo "Finished hello-world-test"
  test:
    - docker run -e NODE_ENV=TEST harbur/hello-world-test node mochaTest
    - docker run -e NODE_ENV=TEST harbur/hello-world-test node karmaTest
project-with-build-args:
  build: Dockerfile
  image: harbur/buildargs
  build_arg:
    keyname: keyvalue

image

The location of the Dockerfile to be compiled.

When auto-detecting, the image will be re-constructed by the following rules:

  • Dockerfile: username/parent_dir
  • Dockerfile.*: username/parent_dir.parsed_suffix

Where

  • username is the host's username
  • parent_dir is the Dockerfile's parent directory name
  • parsed_suffix: is the suffix of the Dockerfile parsed with the following rules:
    • Lower-cased to avoid invalid repository names (Repository names support only lowercase letters, digits and _ - . characters are allowed)
image: harbur/hello-world

build

The relative path of the Dockerfile to be used to compile the image. The Dockerfile's directory is also the build context that is sent to the Docker daemon.

When auto-detecting it will walk current directory and all subdirectories to locate Dockerfiles of the following format:

  • Dockerfile
  • Dockerfile.*

The build path will be reconstructed automatically to compile the Dockerfile. The build context will be the directory where the Dockerfile is located.

Note: If more than one Dockerfiles are needed on specific directory, suffix can be used to separate them and share the same build context.

build: Dockerfile
build: Dockerfile.custom
build: path/to/file/Dockerfile
build: path/to/file/Dockerfile.custom

test

A list of commands that are run as tests after the compilation of the specific image. If any command fail, then captain stops and reports a non-zero exit status.

test:
  - docker run -e NODE_ENV=TEST harbur/hello-world-test node mochaTest
  - docker run -e NODE_ENV=TEST harbur/hello-world-test node karmaTest

pre

A list of commands that are run as preparation before the compilation of the specific image. If any command fail, then captain stops and reports a non-zero exit status.

pre:
  - echo "Preparing compilation"

post

A list of commands that are run as post-execution after the compilation of the specific image. If any command fail, then captain stops and reports a non-zero exit status.

post:
  - echo "Reporting after compilation"

build_arg

A set of key values that are passed to docker build as --build-arg flag. For more information about build-args see here.

build_arg:
  keyname: keyvalue

CLI Commands

build

Builds the docker image(s) of your repository

It will build the docker image(s) described on captain.yml in order they appear on file

Flags:

-B, --all-branches=false: Build all branches on specific commit instead of just working branch
-f, --force=false: Force build even if image is already built

test

Runs the tests

It will execute the commands described on test section in order they appear on file

push

Pushes the images to remote registry

It will push the generated images to the remote registry

By default it pushes the 'latest' and the 'branch' docker tags.

Flags:

-B, --all-branches=false: Push all branches on specific commit instead of just working branch
-b, --branch-tags=true: Push the 'branch' docker tags
-c, --commit-tags=false: Push the 'commit' docker tags

pull

Pulls the images from remote registry

It will pull the images from the remote registry

By default it pulls the 'latest' and the 'branch' docker tags.

Flags:

-B, --all-branches=false: Pull all branches on specific commit instead of just working branch
-b, --branch-tags=true: Pull the 'branch' docker tags
-c, --commit-tags=false: Pull the 'commit' docker tags

self-update

Updates Captain to the last available version.

version

Display version

Displays the version of Captain

help

Help provides help for any command in the application.

Simply type captain help [path to command] for full details.

Global CLI Flags

-D, --debug=false: Enable debug mode
-h, --help=false: help for captain
-N, --namespace="username": Set default image namespace

Docker Tags Lifecycle

The following is the workflow of tagging Docker images according to git state.

  • If you're in non-git repository, captain will tag the built images with latest.
  • If you're in dirty-git repository, captain will tag the built images with latest.
  • If you're in pristine-git repository, captain will tag the built images with latest, commit-id, branch-name, tag-name. A maximum of one tag per commit id is supported.

Roadmap

Here are some of the features pending to be implemented:

  • Environment variables to set captain flags
  • Implementation of captain detect that outputs the generated captain.yml with auto-detected content.
  • Implementation of captain ci [travis|circle|etc.] to output configuration wrappers for each CI service
  • Configure which images are to be pushed (e.g. to exclude test images)
  • Configure which tag regex are to be pushed (e.g. to exclude development sandbox branches)

captain's People

Contributors

dgil avatar dkapanidis avatar dlozano avatar ejholmes avatar jurgenweber avatar jwliechty avatar ludovicc avatar mwildehahn avatar spiddy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

captain's Issues

Add captain pull command

With 'captain pull' it will be possible on CI servers to pull the previous images from registry in order to take advantage of cached steps and avoid pushing new layers of same steps.

captain pull will by default pull 'latest' and 'branch'.
captain push will by default push 'branch' and 'latest'.

add support for multiple git branches

Currently captain by default builds the current working branch, but in case of CI you might want to delegate all branch names of specific commit to the registry.

Take the following example of git commit:

  • abcd (HEAD, branchA, branchB) a specific commit

where branchA is the current working branch. By default it will build and push only branchA. But in case of CI it might be desired to build and push all branchnames.

This is what all_branches (-B) flag does.

-B, --all-branches=false: Build all branches on specific commit instead of just working branch

Ability to disable latest tag?

If I build push from a develop branch, my latest tag will map to that build, but when I build from my master branch (stable) my latest tag will map to that build.

This could cause confusion when you are working with something like gitflow, where latest would jump between develop and master depending on whichever branch your CI server built and pushed last.

Possible solutions would be a disable latest tag cli argument, which could in turn be set as part of a build running on a CI server, or a YAML element that mapped latest to a specific branch.

Example of problem:

charlie:foo (master)$ cat Dockerfile
FROM alpine:latest
ENTRYPOINT ["/bin/echo"]
CMD ["Hello, World"]
charlie:foo (master)$ captain build
[CAPTAIN] Skipping build of foo:c69a2b5 - image is already built
[CAPTAIN] Tagging image foo:c69a2b5 as foo:latest
[CAPTAIN] Tagging image foo:c69a2b5 as foo:master
charlie:foo (master)$ docker run --rm foo
Hello, World
charlie:foo (master)$ git checkout develop
Switched to branch 'develop'
charlie:foo (develop)$ cat Dockerfile
FROM alpine:latest
ENTRYPOINT ["/bin/echo"]
CMD ["Bye Bye, Cruel World"]
charlie:foo (develop)$ captain build
[CAPTAIN] Building image foo:latest
Step 1 : FROM alpine:latest
 ---> 4e38e38c8ce0
Step 2 : ENTRYPOINT /bin/echo
 ---> Using cache
 ---> c7525f863b54
Step 3 : CMD Bye Bye, Cruel World
 ---> Using cache
 ---> 4f246db51200
Successfully built 4f246db51200
[CAPTAIN] Tagging image foo:latest as foo:79e6028
[CAPTAIN] Tagging image foo:latest as foo:develop
charlie:foo (develop)$ docker run --rm foo
Bye Bye, Cruel World
charlie:foo (develop)$

Add support for Google Container Registry

When pushing images to the GCR the commands need to be executed using gcloud as prefix of the command instead of using directly docker.

e.g. instead of:

docker pull gcr.io/your-project-id/example-image

it should be

gcloud docker pull gcr.io/your-project-id/example-image.

Captain should identify these registries automatically and support using locally installed gcloud to pull and push images.

Improve test coverage

The test we have now only test a few simple cases. We should improve test coverage.

Also, we could add in the main page a badge with the coverage.

Support new format for captain.yml

Current format of captain.yml is limited, configuration cannot be added per image. Opting for a docker-compose.yml style format where each image gets it's own space on yml.

Example of new format:

captain:
  build: Dockerfile
  image: harbur/captain
captain-test:
  build: Dockerfile.test
  image: harbur/captain-test
  test:
    - docker run harbur/captain-test go test github.com/harbur/captain/captain

Support env variables for YAML file

We're currently exposing our AWS account number through the YAML file. It would be helpful to inject this via env variables:

Actual:

apache:
  build: Dockerfile
  image: 123456789.dkr.ecr.ap-southeast-2.amazonaws.com/foobar/apache

Expected:

apache:
  build: Dockerfile
  image: $DOCKER_HUB_DOMAIN/foobar/apache

Add purge command to captain

Purge command will remove stale images.

As definition of stale image:

"Stale image is an image that is not the latest of at least one branch."

Invalid tag format?

During tagging after a build
xxxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/ss:tags.1.0.1^0 <- no idea how this got there
API error (500): {"message":"invalid tag format"}

tag & push branch names in detached mode

Currently when 'captain push' is issued in detached mode of a commit with a branch name, captain issues the following error:

[CAPTAIN] Skipping push of app in detached mode

How to reproduce:

  • Checkout git with commit-id (detached mode)
git checkout 2225ad4

Git tree example:

* 2225ad4 (HEAD, origin/example, example) Commit example 2
* ef14e6a Commit example 1

Rational:

This is basically the way most CI servers work, they checkout in detached mode, so in order to tag docker image with the branch name we need captain to be able to identify the branches in detached mode and use them.

Add support for build time variables

Docker supports now build time variables, it would be great if captain adds support for them too.
For example,

hello-world-test:
  build: Dockerfile.test
  image: harbur/hello-world-test
  build_variables:
    NODE_ENV: test
> docker build --help
Usage:  docker build [OPTIONS] PATH | URL | -
Build an image from a Dockerfile
  --build-arg=[]                  Set build-time variables
[...]

Create user-defined tag

This is quite useful for Continuous Delivery use-cases.

In order for CI systems to propagate the BUILD_NUMBER to captain they can now use '-t BUILD_NUMBER' as parameter.
This will create a docker image with the specific version and push it to registry with 'captain push'

Add error handling

Currently captain doesn't do proper error handling. e.g. In case a docker registry fails it will continue without failing and the exit code will be 0 instead of propagating the exit code of the executed command.

captain build fails where docker build succeeds on private images in dockerhub

CAPTAIN] Building image indigobio/rules:latest
Step 0 : FROM indigobio/deps:latest
Pulling repository docker.io/indigobio/deps
Error: image indigobio/deps:latest not found

docker build -t indigobio/rules .
Sending build context to Docker daemon 8.008 MB
Step 0 : FROM indigobio/deps:latest
latest: Pulling from indigobio/deps

2fad5d366fbd: Already exists

If the image is already pulled then captain build will work fine. I can also push my private images through captain push no problem. Any thoughts?

Exit status 0 on nonexistent image

When running e.g. captain build bogus without bogus defined in captain.yml, captain exits with status 0 and without any error message. This breaks CI or anything else relying on an correct exit status.

Update installation process

Now that captain self-update command is implemented (see #37) we need an installation script that downloads and installs the captain binary in the same way that captain self-update does.

Downcase container image name to avoid invalid format

When the directory where Dockerfile resides contains upper-case characters and they are transformed to image name it is an invalid repository name.

To avoid this when inferring settings the directory name should be downcased to map it to a valid repository name.

Example below:

➜  git clone https://github.com/Runnable/dockerfiles
Cloning into 'dockerfiles'...
remote: Counting objects: 1092, done.
remote: Total 1092 (delta 0), reused 0 (delta 0), pack-reused 1092
Receiving objects: 100% (1092/1092), 9.33 MiB | 1.28 MiB/s, done.
Resolving deltas: 100% (296/296), done.
Checking connectivity... done.
➜  cd dockerfiles
➜  captain build
[CAPTAIN] No configuration found [captain.json captain.yaml captain.yml] - inferring values
[CAPTAIN] Building image spiddy/UTlPL1-f2W1TAAAW:latest
API error (500): Invalid repository name (UTlPL1-f2W1TAAAW), only [a-z0-9-_.] are allowed

Feature request: support test subcommands?

It would be nice to be able to breakout tests on a container into sub-commands, this would be useful for CI/CD integration v. Developer workflow.

Example, a python app that might need coverage reports generated in junit, but developers want console tests:

app:
  image: myorg/app
  build: Dockerfile
  test:
    default: nosetests --with-coverage --with-timer 
    ci: nosetests --with-xunit --with-coverage 

Thus, development workflow for testing would be done with captain test running the default target, and a CI server could call captain test ci to run a custom test command.

Use Git tags

If I tag the code with something like RELEASE_1_0 in Git, it would be great to have a regex that detects it in captain and tags the container with version 1.0

Add a self-update command

Currently we have no way to update captain using captain itself so if we want to have the last version we have to check Github and install the last version manually.

I'd like to propose a new command captain self-update that download the last version (if any) and update the binary in the computer.

In order to have captain self-update we should:

  • store the binaries in $HOMEDIR/.captain/binaries/captain-{captain version}
  • have a symlink $HOMEDIR/.captain/bin/captain that points to the last binary (this dir should be in the users's $PATH)
  • store in github the last version of captain. E.g. https://raw.githubusercontent.com/harbur/captain/master/VERSION
How should it work?
  1. User run captain self-update
  2. captain check the last version in https://raw.githubusercontent.com/harbur/captain/master/VERSION
  3. (if new version available) download the last version and store it in $HOMEDIR/.captain/binaries/captain-{version}
  4. Update captain symlink in $HOMEDIR/.captain/bin/

What do you think? Have I missed something?

Pre commands should be skipped if image build is about to be skipped

It is not necessary to run pre commands when the image is already built. It should be skipped in such case.

for example, if pre command is used to build a WAR file before packaging it in an image, that step is no longer necessary if the image is already built and should be skipped.

Add support for "pre" & "post" commands

In order to wrap non-docker build commands on the build life-cycle of containers preparation the following will be added on the captain.yml format:

captain-example:
  build: Dockerfile
  image: harbur/captain-example
  pre:
    - echo "Preparing build"
  post:
    - echo "Cleaning up build"

Add CI integration

Add .travis.yml to compile test and get coverage captain project using travis.

This will give us badges that we can add to the project and facilitate work with the tests.

Lint all the files!

In order to have a cleaner code it will be nice to have all the files linted.

Configure which images are to be pushed

Configure which images are to be pushed, e.g. to exclude test images.

I need this functionality soon... I think that it would be simpler if by default test images are never pushed.

Support multiple modules as space separated parameters

It could be useful to build/push/purge multiple modules, same as the single module option.
Actually if we pass a lists of modules, the filter is ignored and captain makes actions on all of them like when we don't pass any

Add pre/post command on test

In some situations it may be useful to run pre or post operations on test phase. I am thinking in next scenario.

Project A has a Project A-AT which contains the acceptance tests for project A. So before releasing a container you may want to get Project A-AT source code, run its own captain file to generate Project A-AT container (which may contain for example phantomjs) and then in test phase itself running both containers to execute the tests.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.