infolinks / deployster Goto Github PK

An opinionated declarative deployment tool, extensible to virtually any deployment topology.

License: Apache License 2.0

Python 95.95% Shell 4.05%

deployment deployer deploy kubernetes k8s google-deployment-manager docker docker-image docker-deployment docker-development

deployster's Introduction

Introduction

Deployster is an extensible resource-centric deployment tool that works by:

Enabling deployers (eg. You!) declare the desired state of the deployment topology
It then discovers the topology's current state
It then attempts to migrate it from the current state to the desired state.

The main differences between Deployster and other, similar, tools (such as Google Deployment Manager, Terraform, and others) are:

Docker-first philosophy: resource types (eg. VMs, IP-addresses, disks, etc) are written as Docker images, allowing them to be reused in any deployment, on any platform.
Highly extensible: since resource implementations are merely Docker images - you can create new resource types by creating your own Docker images, using any technology you want. Want to create a new resource that represents a GitHub repository using Ruby? Done. Just implement a new Docker image and write your source code in Ruby inside the image, then use that resource in any Deployster manifest. The Docker image name & tag represent the resource type.
Stateless: in contrast to similar tools, Deployster will never store your topology state (locally or remotely) which enables your to also work on your infrastructure & deployments manually or in other tools, side-by-side with Deployster. While we are aware that this might incurr a performance penalty at times (for state discovery) we believe this is preferrable to a wrong state discovery.
Reproducible: deployments are meant to be idempotent, such that running the same deployment multiple times should yield the same result as running it just once, assuming that nothing else modified the state.

Getting started & documentation

The quick start page will get you up & running in 10 minutes. For a more in-depth overview of Deployster, head over to the overview page to learn more about the core concepts in Deployster.

And for the lazy...

Deployster is distributed as a Docker image available for running on your machine or servers. Since the image requires a certain set of Docker flags, a simple bash script is provided which makes running Deployster a breeze. Here's how:

curl -sSL -O "https://raw.githubusercontent.com/infolinks/deployster/master/deployster.sh"
./deployster.sh --var my_var="some-value" \
                --var another="some-value" \
                --var-file ./my-variables-file.yaml \
                ./manifest1.yaml ./manifest2.yaml

Here's the full run-down of command-line flags available:

[arik@arik ~]$ curl -sSL -O "https://raw.githubusercontent.com/infolinks/deployster/master/deployster.sh" && chmod +x ./deployster.sh
[arik@arik ~]$ ./deployster.sh --help

✔ Deployster v18.0.1

      😄 Deploy with pleasure!
      
usage: deployster.py [-h] [-c {NO,ONCE,RESOURCE,ACTION}] [--var NAME=VALUE]
                     [--var-file FILE] [-p] [-v]
                     manifests [manifests ...]

Deployment automation tool, v18.0.1.

positional arguments:
  manifests             the deployment manifest to execute.

optional arguments:
  -h, --help            show this help message and exit
  -c {NO,ONCE,RESOURCE,ACTION}, --confirm {NO,ONCE,RESOURCE,ACTION}
                        confirmation mode
  --var NAME=VALUE      makes the given variable available to the deployment
                        manifest
  --var-file FILE       makes the variables in the given file available to the
                        deployment manifest
  -v, --verbose         increase verbosity

Written by Infolinks @ https://github.com/infolinks/deployster

The documentation is still in its early phases. Some of the information may be a bit out of date or inaccurate. We are sorry for this temporary state and hope to finish documenting Deployster soon!

Credits

Deployster was written by the team @ Infolinks as a means to improve our deployment process. We plan to gradually open source more & more components of that pipeline in the coming months.

deployster's People

Contributors

Stargazers

Watchers

Forkers

alexprem

deployster's Issues

Allow resources to ask for resource dependency list

For example, a Kubernetes role binding can ask for a list of Kubernetes user resources (currently it only permits one).

This is probably useful in many contexts:

Role bindings requesting a list of subjects
Others?

Add GCP IAM binding resource

Add a gcp-iam-binding resource.

This will allow ensuring that certain users/service-accounts have IAM roles.
This requires #34 to be implemented.

Cache results for "init" actions when possible

The output for resource "init" actions, given the same resource details & type should usually remain the same, since its output mainly reflects the capabilities & requirements of the resource type, not the resource itself.

Therefor, it would probably provide a nice performance boost to be able to cache such results - perhaps using JSON files in the work directory.

Consistent resource configuration property naming

Some properties use underscope between words in the property name, and some properties use dashes instead.

Whatever we choose, it should be consistent.

Print dependency chain when circular dependencies are detected

Support using other variables in variable files

So assuming the variable foo was given the value bar via the command-line (eg. --var foo=bar), support this in a variable file:

chu: chuku-{{ foo }}

Parallelize resource initialization

Resource init actions have no side-effects, and as such can be invoked in parallel. We might want to throttle it though - perhaps around 3 resources at a time (and add a commandline flag to configure it).

Support creating a Plan from a previously-saved plan file

Allow Deployster to calculate a deployment plan and just save it to a file. That file can later be fed back to Deployster.

Notes:

When receiving a previously-saved plan file, Deployster should probably re-calculate the plan anyway, and just verify that it still gets the same plan, something like a fingerprinting or an etag mechanism to ensure reproducability.

Add prudent mode

Prudent mode should ask for confirmation for every action.

K8s Ingress resource should wait for Kube-Lego to generate certificate

When an Ingress resource is created/updated, and kube-lego is installed, the Kubernetes Ingress resource type should wait until kube-lego detects the Ingress and generates its TLS certificate.

Refresh resource state after every action

Currently, the resource is state is refreshed once all its actions have been executed. It can be beneficial to refresh the state after each action, to enable subsequent actions to get the most up-to-date state.

Updating Cloud SQL labels does not remove old labels

Currently if an instance contains labels A and B (whatever their values), and the manifest updates its flags to B & C, the end result is that the instance has labels A, B & C.

Support marking resources as deleted

Marking a resource (in a manifest) as deleted would enable users to verify that certain resources do not exist (their status must be MISSING). If they do exist, Deployster should perform a removal action.

This provides the capacity to safely delete resources through Deployster, and ensure that they don't accidentally get created again later on.

Support "--validate-only" flag

To allow requesting Deployster just to validate all resource configurations according to their configuration schemas.

Add GCP service-account resource

Create a new gcp-service-account resource.

Support both specifying the resource's name for a specific service account, as well as referencing the default service account of a GCP project.

Print list of manifest files on startup

Print the list of manifest files to be executed as part of the startup printout, which can be useful for logging and audit purposes.

Generate rollback plans

It would be useful to generate a rollback plan that negates the effects of every executed action.

Parallel resource actions execution

For unrelated resources whose actions are not related, it would be much more efficient to invoke them in parallel. This needs to consider dependencies and unseen consequences, but in general quite doable.

Support pull-if-not-present resource image pull policy

Currently, Deployster supports either:

always pull resource images (default)
relying on Docker to pull the image if not already present (includes latest which is unfortunate)

It would be beneficial to support a new --pull-policy argument with the following values:

always which would always [re]pull resource images regardless if they are present or not (esp. latest)
if-not-present which would always pull the latest tag, but avoid pulling tags if they are already present
never which would never pull new tags (failing if a tag is not already present in the system)
- is this useful?

The default IMO should be if-not-present.

Each resource has to implement them
Inaccurate: they incorporate everything the resource does (so if an action performs a long action, and then some kind of a polling, the resource needs to calculate a timeout for everything together, which is much harder)

Support loading JSON manifests & variable files

Should be easy to implement.

We currently use yaml.load(..) which is similar to json.loads(...).