Kubernetes is a highly extensible platform. It supports many extension points to extend and customize your Kubernetes deployment without touching the core kubernetes source code. Here I focus on Custom Resource Definitions, what they are, why they are useful, and how to implement them using Kubebuilder and Operator SDK. I will explain CRDs using the excellent example from the book Kubernetes in Action by Marko Lukša.

Note, this is an EXCELLENT book on Kubernetes and I highly recommend you read it!

We will first dive into Marko's example as it is simple and clear. However, his implementation is intentionally very simple and as he notes "barely working". So we will re-implement his example using Kubebuilder and Operator SDK which enable you to build production grade CRDs and Controllers quickly.

Note, this is pretty advanced material so if new to Kubernetes, you should start with my Bootcamp

Kubernetes provides a rich set of API primitives to specify the desired state of your system. However, when it comes to deploying complex infrastructure such as Prometheus, ELK stack, etc that have multiple components that interact with each other and need to be configured, this becomes very complex. CustomResourceDefinitions are a means to create new API resources that model your problem/solution space and abstract away this complexity. For example, the Prometheus Operator from CoreOS defines the following CRDs:

• Prometheus defines the desired state of the Prometheus deployment
• AlertManager describes an Alertmanager cluster
• PrometheusRule defines alerting rules for a Prometheus instance
• ServiceMonitor defines the set of targets to be monitored by Prometheus. You select Services (and the underlying Pods) to be monitored using labels and label selectors.

Anyone that understands Prometheus, would understand these resources. Instead of thinking in terms of Deployments, Statefulsets, Services etc, you can now think in concepts you are familiar with. This is the value of Custom Resource Definitions. Of course, the Kubernetes API primitives are still there. To deploy Prometheus, Alertmanager, Exporters, Grafana etc, the Prometheus Operator deploys Deployments, StatefulSets, Services and of course, Pods, to bring up the components in a resilient manner. This is done through an implementation of a custom controller that consumes the CRDs and creates the Kubernetes resources to deploy and configure the components.

To deploy the Prometheus Operator, follow the instructions here

To demonstrate the process of creating a CRD, let's first deploy Marko Lukša's example.

The scenario is as follows:

We want to create a new resource called WebSite. When we create this resource, a new WebSite based on the source code located at the specified git repo will be deployed and exposed in Kubernetes. This will require a Deployment and Service resources to be created.

1. First step is to define our Website Custom Resource. You do so by creating a CustomResourceDefinition resource.

#We have a CRD defined in website-crd.yaml, create it like any other resource
kubectl create -f website-crd.yaml

#Verify it has been created
kubectl get customresourcedefinitions

NAME                              AGE
websites.extensions.example.com   16s

2. Now that the CRD has been created, create an instance of our Website resource.

kubectl create -f website.yaml

#Verify it has been created
kubectl get ws

NAME      AGE
kubia     4s

3. At this point, nothing happens because there is no controller that watches for the Website resource. So we need to deploy a custom controller. The source code for the controller is here

#The website-controller will need a serviceAccount with sufficient privileges to access the kube-apiserver
kubectl create serviceaccount website-controller
kubectl create clusterrolebinding website-controller --clusterrole=cluster-admin --serviceaccount=default:website-controller

#create the website-controller as a deployment
kubectl create -f website-controller.yaml

#see what pods get deployed
kubectl get pods

#Notice the kubia-website pod has been deployed
NAME                                  READY     STATUS    RESTARTS   AGE
kubia-website-5645d5dc9-np68m         2/2       Running   0          3m
website-controller-84f9785c68-lzgmn   2/2       Running   1          3m

Below is a diagram from the Kubernetes in Action book that describes the series of events that occur when the Website custom resource is deployed.

The website-controller pod contains 2 containers.

1. The controller itself that watches for 'Website' resources and deploys the website pod (which is just an nginx container that serves a static page)
2. A kubectl-proxy container as a 'side-car' container.

The controller needs to communicate with the kube-apiserver in an authenticated manner. The simplest way is to start up kubectl in proxy mode. Since containers within the same pod share the same network namespace, the controller container can access the proxy via 'localhost'. On start up, the controller gets a list of all 'Website' resource by sending a GET request like so:

#Notice the api path is group/version/resource as defined in the CRD
http://localhost:8001/apis/extensions.example.com/v1/websites?watch=true

See the following diagram from the Kubernetes in Action book.

Kubebuilder is part of the Kubernetes api-machinery SIG and is fast on track for being the framework for building custom CRDs, Controllers and Webhooks.

The Kubebuilder docs are excellent. See the installation and setup docs then see here for instructions on how to implement our custom CRD/Controller, debug, and deploy to Kubernetes.

Note that Operator SDK is alpha software! Proceed with caution!

Operator SDK is part of CoreOS' Operator Framework and is the basis for many Operator implementations. It differs from Kubebuilder in that in addition to Golang, you can create Operators using Ansible.

See the user guide to install Operator SDK locally on your machine. Note the dependencies! After you have installed the sdk, see here for instructions on how to implement the Website CRD and custom controller, debug and to deploy to Kubernetes.

• Kuberenetes Deep Dive Series by RedHat. Excellent overview of the kube-apiserver and how state is stored in etcd.
• Kubernetes Code Generation Deep Dive by RedHat
• Writing Kubernetes Custom Controllers.Describes how to implement custom controllers using client-go. Read this prior to diving into the sample-controller as it describes an established pattern for implementing controllers.
• sample-controller. Example implementing a customer controller using client-go library.
• controller-runtime library docs. Both Kubebuilder and Operator SDK depends on this library. It primarily provides the lower-level types and functions to integrate with the api server, watch for events, provide a internal work queue to decouple the Informer from the Reconciler and more.
• Kubebuilder docs
• Operator SDK docs