These pipeline configuration files are designed to be used with Concourse to deploy components of the MOJ Analytical Platform.

Setting a pipeline in Concourse is done using the fly CLI tool

We currently have two Concourse team:

• admin with the pipelines to release Control Panel, update Helm repository, and update webapp pipelines from GH repositories
• main with the pipelines to release users' webapps and the airflow pipeline

Example invocations of fly are listed alongside the documentation of each pipeline below. You will need to fly login to a Concourse server before you can set a pipeline, and on first login, you will name the "target", which is referred to as default in the examples. Eg:

Most of these pipelines are in the admin team, so an example login command would look like:

fly login -c https://concourse.services.alpha.mojanalytics.xyz -t alpha-admin -n admin

However, airflow pipeline is in main team so you'll have to login passing -n main:

fly login -c https://concourse.services.alpha.mojanalytics.xyz -t alpha-main -n main

NOTE: You can find the Concourse URL by getting its Kubernetes Ingress resource:

kubectl -n default get ingress -lapp=concourse-web

Values can be passed to fly to interpolate into the pipeline YAML before setting the pipeline config in Concourse. These values are marked in the YAML as ((value-name)). If a value is not passed on the fly command line, the value will be looked up in the Kubernetes secrets in the Concourse team's namespace

The following pipeline configs are provided in this repo:

fly -t alpha-admin set-pipeline -p cpanel-api -c dev/cpanel-api.yaml

A pipeline for deploying the Control Panel API to the dev cluster.

fly -t alpha-admin set-pipeline -p cpanel-api -c alpha/cpanel-api.yaml

A pipeline for deploying the Control Panel API to the alpha cluster.

fly -t alpha-admin set-pipeline -p update-helm-repo -c alpha/update-helm-repo.yaml -v s3-bucket=moj-analytics-helm-repo -v aws-region=eu-west-1

A pipeline to keep our Helm repository up-to-date.

NB The Data Engineering team are responsible for this pipeline

fly -t alpha-main set-pipeline -p airflow -c airflow.yaml -v repo=moj-analytical-services/airflow-dags

A pipeline to lint user contributed airflow DAGs in the environment specific repo.

NOTE: This pipeline is in the main team so you'll have to login in that team to update it, see "fly login" section.