jamesaud / pypeline Goto Github PK

A docker pipeline built in python. The ability to execute and design workflows. It might look like:

clone from github - build docker image - run tests in docker container(s) - push to docker repository

The pipeline module and generic pipeline module allow you to generate pipelines. Please checkout the examples in 'examples' for full code.

Create complex pipelines that allow for fast and flexible docker deployments.

It is build upon docker-py, a python wrapper around the REST api of the docker daemon.

This first example is drawn out and done 'the long way', but showcases some of the important operations available.

Before being able to do any pipeline code, you have to specify where the docker daemon is. You also might need to provide TLS configuration. If using 'default=True', the TLS config looks for the default.

clientsetup(default=True, docker_base_url='https://192.168.99.100:2376')

default=True

Or on unix:

clientsetup(docker_base_url=unix://var/run/docker.sock)

Refer to the docker-py api.

from pypeline.pipeline import Pipeline
from pypeline.config import clientsetup

clientsetup(default=True, docker_base_url='https://192.168.99.100:2376')
with Pipeline() as pipe:
    pipe.clone('https://github.com/jamesaud/simplest_docker')  # clone from git
    test_image = pipe.build('justatest1232123/myawesomeimage')  # build with name
    container = test_image.container('echo "rspec"') # run commands in container
    container.run()  # creating and then running containers follows the docker daemon REST api
    container.remove()  # delete.
    test_image.tag('justatest1232123/myawesomeimage:solid')  # Tag it with a solid name
    pipe.login(username='justatest1232123', password='Justatest123')  # Login to dockerhub
    test_image.push()  # Push.
    test_image.remove()  #Remove the image, if you really want.

However, it is possible and recommended to make the code much more compact. Python's 'with' syntax provides an automatic closure at the end of the block.

with Pipeline() as pipe:
        pipe.clone('https://github.com/jamesaud/simplest_docker')
        with pipe.build() as myImage, myImage.run_container('echo "unit-tests"'):
            myImage.tag("jamesaudretsch/myawesomeimage:latest").push()
#  The image and container are automatically deleted when the block ends.

The easiest but less flexible way to build a pipeline. The goal of the Generic Pipeline is to streamline the process for testing on a single image. It also hands all of the naming for you, so this example is pushed to the repository at 'dockerhub.com/r/justatest1232123/simplest_docker'. If you use a custom registry and repository name, it will be pushed to that location instead.

from pypeline.generic_pipeline import GenericPipeline
from pypeline.config import clientsetup

clientsetup(default=True, docker_base_url='https://192.168.99.100:2376')
with GenericPipeline() as GP:
        GP.build('https://github.com/jamesaud/simplest_docker')  # Clone and build.
        GP.test('echo "first test"', 'echo "second test"', 'echo "third test"')  # Run parallel commands in separate containers
        GP.login(username='justatest1232123', password='Justatest123')  # Optional registry and repository argument
        GP.push('latest')  # Tag before it pushes.

Look in the examples folder for full examples of these two modules.

generic_cline.py is an example of a built generic pipeline that allows for command line input.

python3 generic_cline.py --username justatest1232123 --password Justatest123 --url \

https://github.com/jamesaud/simplest_docker --test 'echo "hello world"' --test 'echo "hi"'

###Problem Statement Manually doing CI/CD using docker can be a bloated process, including configuration and naming of images, maintaining and versioning containers and images, deleting old images and containers, etc. Moreover, the same process is consistently used over different images so we end up repeating ourselves.

###Goal To provide a clear structure for building CI/CD pipelines with docker. This is an Object Oriented approach to building pipelines that requires very little code to get up and running.

Make sure you have python3.

You must have git installed with command line input, because the pipeline uses the shell command 'git clone'.

How to install:

git clone https://github.com/jamesaud/pypeline

cd pypeline

pip3 install -r requirements.txt

python3 setup.py install

##API Reference

##-Pipeline-

from pypeline.pipeline import Pipeline

Clone from git repository. Just like the git clone command

Params:

• git_url (str): The url of the project to clone.

Returns None

Builds from the cloned directory.

Params:

• image_tag (str): The name to give the image. : optional

• path (str): The relative path to the dockerfile : optional

• dockerfile (str): The name of the dockerfile : optional

Returns Image: an image object representing this docker image.

Pulls an image, just like 'docker pull ...'

Params:

• image_tag (str): The url of the image to pull.

Returns Image: an image object representing the docker image.

Closes the Pipeline. Because pipeline creates a directory to keep everything separated from the rest of your files, this function must be called at the end of the pipeline. Using 'with' syntax calls this command.

Params:

Returns (generator or str): The logs or output for the image

Copys a single file into the cloned directory.

Params:

• full_file_path (str): The path to the file you wish to copy into the cloned directory.

Returns None

###login

Logs into dockerhub, or another registry, using a username and password.

Params:

• username (str): The username for the registry. : optional

• password (str): The password for the registry. : optional

• registry (str): The url of the registry to login to. : optional

Returns None

##-Image-

from pypeline.pipeline import Image

Image is a representation of a docker image. It is returned when calling 'Pipeline().build' or 'Pipeline().pull'. You could also use it independently of pipeline...

###Properties:

• id : the id of the image. : read-only

• name: the name of the image. : read-only

###container

Creates a docker container based on the image. Does not run it.

Use either <Container>.run(), or <Image>.run_container() to run a container.

Params:

• args (str): The args to run inside the container : optional

• name (str): The name to give the docker container : optional

Returns Container : a container object representing a docker container.

###run_container

Same as 'container' but runs the container immediately. Is usually advised unless you have a reason to forgo running the container.

Params:

• args (str): The args to run inside the container : optional

• name (str): The name to give the docker container : optional

Returns Container : a container object representing a docker container.

###run_parallel_containers

Runs containers in parallel.

Params:

• *commands (str): Any amount of comma separated commands to run. Each separated command will be run in a separate container.

Returns None

###remove

Deletes the image. Is called when using 'with' syntax.

Params:

Returns None

###tag

Tags the image, and returns a new image

Params:

• name (str): The new name to give the image.

Returns Image (Self): returns itself with the tagged name.

###push

Pushes to registry. Like 'docker push ...'. Follows the same rules as regular docker. If you want to push to a private registry prepend the name of the image with 'registry.com/repo/image:tag'. To push to docker just keep the image name 'repo/image:tag'

Params:

Returns None

##-Container-

from pypeline.pipeline import Pipeline

Container is a representation of a docker container.

###Properties:

• id : the id of the container. : read-only

• name: the name of the container. : read-only

###remove

Removes the container.

Params:

Returns None

###run

Runs the container.

Params:

Returns None

##-GenericPipeline-

from pypeline.generic_pipeline import GenericPipeline

###build

Builds a docker image from the git directory.

Params:

• git_url (str): The url of the github project.

• path (str): The relative path to the dockerfile : optional

• dockerfile (str): The name of the dockerfile : optional

Returns None

###test

Runs commands inside of parallel containers.

Params:

• *commands : commands to run in containers based on the built image. Any number of comma separated arguments.

Returns None

###login

Logs into a docker registry. Also provides the naming for the image this way. If using dockerhub, your image will look like: 'username/git_project_name:tag'

If logging into a private repository, you'll need to provide the repository argument to tell it where to upload.

GenericPipeline().login(registry='[email protected], repository='myname')

Your image will look like: 'registry/repo/git_project_name:tag'

Params:

• username (str) : the username for the repository : optional
• password (str) : the password for the repository : optional
• registry (str) : the registry to upload to, defaults to dockerhub.
• repository (str) : only required if providing the registry, tells the generic pipeline which repository to upload to if not provided a username : optional

Returns None

###push

Pushes the image.

Params:

Returns None

###close

Closes the Pipeline() that was opened. Is called automatically using 'with' syntax.

Params:

Returns None

Unit tests are only provided for the pipeline, not the generic pipeline. They are in the unit_tests folder. Something to note is that a side effect of testing might be that a small busybox or alpine image remains on your machine.

You'll have to specify the docker client in the 'setUpClass' methods for each class in the test files.

You just need to install this package on your jenkins machine to be able to use it.

Instructions are here: https://github.com/jamesaud/dockerized-jenkins-pypeline

James Audretsch: [email protected]