A repo for showcasing a basic developer platform on an OpenStack Magnum Kubernetes Cluster, provisioned through Terraform.

• Provision using Terraform from a custom Magnum Cluster Template
• Configure Persistent Storage Class for OpenStack
• Setup cert-manager for X.509 certificate management using Terraform
• Setup internal NGINX Ingress Controller using Terraform, with auto provisioning cluster wide DNS hostname
• Optional setup of a Rook Ceph storage cluster, see rook-ceph.md
• Actions Runner Controller (ARC) for self-hosted GitHub workflow runners, providing an internal CI and code build platform
• Argo CD (continuous delivery) for the cluster (see /argo-cd/argo/commands.sh for installation instructions)
• Deploy an example app with valid SSL certificate using Terraform

• Have appropriate OpenStack allocation (Clusters / Networks / FloatingIPs etc.), you can configure the resources used for cluster in the Cluster Template definition, here: main.tf
  • The supplied config needs following main resources:
    • 1 Cluster
    • 1 Network
    • 1 Router
    • 2 Floating IPs (one for cluster's main load-balancer, and one for NGINX Ingress Controller)
    • Appropriate compute and storage allocations as defined in the cluster template
• Be authenticated with OpenStack using Application/Personal credentials file
• Install Terraform
• Install kubectl (this may already be installed if you have Docker Desktop installed), I have an alias for it shortened to k
• Setup secrets directory with an already created/configured ssh key (password less), set or pass its path to /cluster/variables.tf file

The main terraform module creates a custom Magnum ClusterTemplate which is supported by the provider and configured to work with upcoming use-cases. See https://docs.openstack.org/magnum/latest/user/#clustertemplate for details of all parameters and labels.

cd cluster
terraform init
terraform apply

Wait for 5-15 minutes for cluster to be deployed

After the cluster is provisioned the kubeconfig file will be saved at ./cluster/secret/config. Load this config into env variable:

export KUBECONFIG=<PATH_TO_REPO>/cluster/secret/config

After the cluster is created we will need to configure it for our use cases like:

• Set up the cinder StorageClass as default, for all PersistentStorageClaims in the cluster
• Install CRDs for utility packages like cert-manger and argo-cd or more.

To configure cluster, check/update the path to kubeconfig in the ./cluster/provision/variables.tf file. Then:

cd cluster/configure
terraform init
terraform apply

After this complete a StorageClass named default will be setup and be available to be used by Kubernetes cluster.

Now that we have a working cluster setup, we can install and configure some essential cluster addons. Following addons will be installed:

• NGINX Ingress Controller, will be default Ingress Controller for our cluster, a DNS hostname for whole cluster will be provisioned from OpenStack using designate API, configure this in variables.tf
• cert-manager, will automatically manage X.509 certificates in our cluster.
• Argo CD, will be setup to enable continuous deployment of user workloads

There are multiple ways to install packages into our cluster here are few options:

• Manual install using yaml files
• Use Helm charts
• Use Kubernetes and Helm Terraform providers to setup packages

cd addons-terraform
terraform init
terraform apply

See Manually configure cert-man

Following repo has an example application (Quarkus based API with PostgreSQL db), with the instructions to deploy to this kubernetes cluster: Deploy Example App.