After a conversation with my colleague Tero we decided to prepare a session to highlight the benefits a developer can get from the Operators in general and from the Operator Framework in particular. For that session we depicted two demos: one to show how easy is to have a Kafka cluster on OpenShift (Red Hat's kubernetes distribution) and another one showing a custom operator that deployed and updated an application called Gramola.
This is the reposiory for the Operator created for that demo and effectively the demo itself.
With this guide you will learn how we created the operator to deploy version 0.0.1 of Gramola (a Java based application including an Angular UI, a Gateway and Events API and a PostgreSQL database). More importantly you will also learn how we evolved the operator moving from version 0.0.1 to 0.0.2 (this included update the database schema and migrating data to the new schema). And even more importantly how we used the Operator Lifecycle Manager to do all this automatically.
The 2nd part of the guide explains how to run the demo, no need to code just enjoy deploying and upgrading our target application.
You need basic understanding of what an operator is to understand this guide Additionally if you want to run the demo or create your own Operator you also need:
- Go 1.13.5+
- Operator SDK v0.15.1+
- Free account in Quay (this is needed to store the manifests that describe channels and versions of your operators)
Golang Based Operator SDK Installation Follow the steps in the installation guide to learn how to install the Operator SDK CLI tool.
Additional Prerequisites https://sdk.operatorframework.io/docs/golang/installation/ git go version v1.12+. mercurial version 3.9+ docker version 17.03+. kubectl version v1.11.3+. Access to a Kubernetes v1.11.3+ cluster.
Parts of the code of this operator were borrowed from another operator coded by my colleague Madou
I have divided the guide in two parts:
- The first part explains end to end how to create an Operator taking this operator as a starting point.
- The second one explains how to run the demo consisting on deploying version 0.0.1 and then upgrade to 0.0.2 and see how the database schema is modified and data is migrated.
In order to simplify and speed up the development of an operator we're going to install the Operator SDK.
export RELEASE_VERSION=v0.19.2
export OS=apple-darwin
mkdir ./bin
curl -LO https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-${OS}
mv operator-sdk-${RELEASE_VERSION}-x86_64-${OS} ./bin/operator-sdk
chmod u+x ./bin/operator-sdk
export PATH=$(pwd)/bin:$PATH
You need the free account to use Quay as the repository for your operator manifests bundle. Then use those credentials to log in using docker/podman.
docker login quay.io
Let's define some environment variables that will come handy later. Special attention to GO111MODULE.
export GO111MODULE=on
export OPERATOR_VENDOR="redhat"
export APP_NAME="gramola"
export OPERATOR_NAME="${APP_NAME}-operator"
export OPERATOR_IMAGE="${APP_NAME}-operator-image"
export API_VERSION="${APP_NAME}.${OPERATOR_VENDOR}.com/v1alpha1"
export API_VERSION="v1alpha1"
export PROJECT_NAME=${OPERATOR_NAME}-project
We're going to generate the Golang scaffold for our operator, just do as follows.
NOTE: operator-sdk init generates a go.mod file to be used with Go modules. The --repo= flag is required when creating a project outside of $GOPATH/src, as scaffolded files require a valid module path. Ensure you activate module support by running export GO111MODULE=on before using the SDK.
mkdir -p $GOPATH/src/github.com/${OPERATOR_VENDOR}
or
mkdir -p ./${OPERATOR_VENDOR}/${OPERATOR_NAME}
cd $GOPATH/src/github.com/${OPERATOR_VENDOR}
or
cd ./${OPERATOR_VENDOR}/${OPERATOR_NAME}
operator-sdk new ${OPERATOR_NAME} --type=go --repo github.com/${OPERATOR_VENDOR}/${OPERATOR_NAME}
or
operator-sdk init --domain=${OPERATOR_VENDOR}.com --repo=github.com/${OPERATOR_VENDOR}/${OPERATOR_NAME}
NOTE: IF error creating API ==> export GOROOT=$(go env GOROOT)
cd ./${OPERATOR_NAME}
operator-sdk add api --api-version=${API_VERSION} --kind=AppService
or
operator-sdk create api --group=${APP_NAME} --version=${API_VERSION} --kind=AppService
Create Resource [y/n]
y
Create Controller [y/n]
y
$ operator-sdk add controller --api-version=${API_VERSION} --kind=AppService
code ./pkg/apis/gramola//_types.go
In this case:
code ./pkg/apis/gramola/v1alpha1/appservice_types.go
// AppServiceSpec defines the desired state of AppService type AppServiceSpec struct { // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster // Important: Run "operator-sdk generate k8s" to regenerate code after modifying this file // Add custom validation using kubebuilder tags: https://book-v1.book.kubebuilder.io/beyond_basics/generating_crd.html
// Flags if the the AppService object is enabled or not
Enabled bool `json:"enabled"`
// Flags if the object has been initialized or not
Initialized bool `json:"initialized"`
// +kubebuilder:validation:Enum=Gramola,Gramophone,Phonograph
Alias string `json:"alias,omitempty"``
}
Go to ./pkg/apis/v1alpha1/appservice_types.go
Find this:
// Foo is an example field of AppService. Edit AppService_types.go to remove/update
Foo string `json:"foo,omitempty"`
And subsitute it with:
// +kubebuilder:validation:Minimum=0
// Size is the size of the memcached deployment
Size int32 `json:"size"
Go to
// Nodes are the names of the memcached pods
Nodes []string `json:"nodes"`
operator-sdk generate k8s
operator-sdk generate openapi
operator-sdk generate crds
Build the latest openapi-gen from source!
which ./bin/openapi-gen > /dev/null || go build -o ./bin/openapi-gen k8s.io/kube-openapi/cmd/openapi-gen
./bin/openapi-gen --logtostderr=true -o "" -i ./pkg/apis/gramola/v1alpha1 -O zz_generated.openapi -p ./pkg/apis/gramola/v1alpha1 -h ./hack/boilerplate.go.txt -r "-"
go mod vendor
go list -m -versions gopkg.in/src-d/go-git.v4
You should have an account in quay.io, if you don't please create it and use it here.
export USERNAME=<username>
export OPERATOR_VERSION=0.0.1
operator-sdk build quay.io/${USERNAME}/${OPERATOR_IMAGE}:${OPERATOR_VERSION}
docker login quay.io
docker push quay.io/${USERNAME}/${OPERATOR_IMAGE}:${OPERATOR_VERSION}
Update the operator manifest to use the built image name (if you are performing these steps on OSX, see note below)
The operator-sdk
has generated a default deployment
descriptor in deploy/operator.yaml
but instead of pointing to a real image it contains a placeholder REPLACE_IMAGE
, let's substitute it with the image we just built for image 0.0.1
.
sed -i "s|REPLACE_IMAGE|quay.io/${USERNAME}/${OPERATOR_IMAGE}\:${OPERATOR_VERSION}|g" deploy/operator.yaml
On OSX use:
sed -i "" "s|REPLACE_IMAGE|quay.io/${USERNAME}/${OPERATOR_IMAGE}\:${OPERATOR_VERSION}|g" deploy/operator.yaml
$ oc new-project ${PROJECT_NAME}
or
$ oc project ${PROJECT_NAME}
Before we use OLM to deliver our operator we need to develop and test it locally.
- Setup Service Account and RBAC
oc apply -f deploy/service_account.yaml
oc apply -f deploy/role.yaml
oc apply -f deploy/role_binding.yaml
- Setup the CRD
oc apply -f deploy/crds/gramola.redhat.com_appservices_crd.yaml
The operator-sdk
is prepared to run the code of our locally but within our kuberenetes cluster.
operator-sdk run --local --namespace ${PROJECT_NAME}
If there are no errors you should see something like this.
INFO[0000] Running the operator locally in namespace gramola-operator-project.
{"level":"info","ts":1584114880.591337,"logger":"cmd","msg":"Operator Version: 0.0.1"}
{"level":"info","ts":1584114880.591379,"logger":"cmd","msg":"Go Version: go1.13.5"}
{"level":"info","ts":1584114880.591383,"logger":"cmd","msg":"Go OS/Arch: darwin/amd64"}
{"level":"info","ts":1584114880.5913868,"logger":"cmd","msg":"Version of operator-sdk: v0.15.1"}
{"level":"info","ts":1584114880.602292,"logger":"leader","msg":"Trying to become the leader."}
...
{"level":"info","ts":1584114899.0024478,"logger":"controller_appservice","msg":"Reconciling AppService","Request.Namespace":"gramola-operator-project","Request.Name":"example-appservice"}
You could also, alternativele, deploy the operator. But we're not going to do that for now.
oc apply -f deploy/operator.yaml
The default controller pkg/controller/appservice/appservice_controller.go
will watch for AppService
objects and create a pod for each CR.
Now in a different terminal window but in the same path let's create an example AppService
object.
oc apply -f deploy/crds/gramola.redhat.com_v1alpha1_appservice_cr.yaml
You should get this error:
The AppService "example-appservice" is invalid:
* spec.enabled: Required value
* spec.initialized: Required value
Change the type AppService
, so that initialized is not required
Initialized bool `json:"initialized,omitempty"`
Let's change the CR so that enabled
is defined:
apiVersion: gramola.redhat.com/v1alpha1
kind: AppService
metadata:
name: example-appservice
spec:
enabled: true
Now if you try again... is thould work.
$ oc apply -f deploy/crds/gramola.redhat.com_v1alpha1_appservice_cr.yaml
appservice.gramola.redhat.com/example-appservice created
$ oc get pod -l app=example-appservice NAME READY STATUS RESTARTS AGE example-appservice-pod 1/1 Running 0 1m
$ oc describe appservice example-appservice Name: example-appservice Namespace: gramola-operator-project Labels: Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"gramola.redhat.com/v1alpha1","kind":"AppService","metadata":{"annotations":{},"name":"example-appservice","namespace":"gram... API Version: gramola.redhat.com/v1alpha1 Kind: AppService Metadata: Creation Timestamp: 2020-03-12T10:30:09Z Generation: 1 Resource Version: 920134 Self Link: /apis/gramola.redhat.com/v1alpha1/namespaces/gramola-operator-project/appservices/example-appservice UID: 21dac31a-fc2f-4fb4-9d78-bc8d72531281 Spec: Enabled: true Events:
The operator-sdk
tool will help us to create the CSV for our ClusterService Gramola Operator
if you run it now, as follows, you'll see a WARNING.
NOTE 1: Flag
--update-crds
commandsoperator-sdk
to generate properties, definitions, etc. related to the CRDs your operator manages
NOTE 2: go here to find out which properties are mandatory, optional, etc. in a CSV
$ operator-sdk generate csv --csv-version 0.0.1 --update-crds
INFO[0000] Generating CSV manifest version 0.0.1
WARN[0000] Required csv fields not filled in file deploy/olm-catalog/gramola-operator/0.0.1/gramola-operator.v0.0.1.clusterserviceversion.yaml:
spec.keywords
spec.maintainers
spec.provider
We need to fix this WARNING but at the same time we'll also fix some others you would find along the way and why not add some useful information and even an icon.
We have to provide keywords, maintainers, provider... and also an icon ;-)
export CSV_PATH=./deploy/olm-catalog/gramola-operator/0.0.1/gramola-operator.v0.0.1.clusterserviceversion.yaml
yq w -i -s update_csv_instructions.yaml ${CSV_PATH}
All these changes (user defined) are reflected directly in the CSV generated for version 0.0.1 and will be kept if you re-generate the CSV for the same version over and over.
But there're still some changes to the CSV that need to be done in the type associated with our CRD AppService
. This can be done adding some comments to the type code, comments starting with +operator-sdk:gen-csv
.
For instance to add spec.customresourcedefinitions.owned[].displayName
to the CSV indirectly we add +operator-sdk:gen-csv:customresourcedefinitions.displayName="AppService"
as in this excerpt.
...
// AppService is the Schema for the appservices API defines Gramola Backend Services
// +operator-sdk:gen-csv:customresourcedefinitions.displayName="AppService"
// +kubebuilder:subresource:status
// +kubebuilder:resource:path=appservices,scope=Namespaced
type AppService struct {
...
Another example, imagine you want to add some UI related comments to surface some CRD realated data. For instance if the AppService
is enabled and also the Alias
chosen. Next example has this into account.
...
// Flags if the the AppService object is enabled or not
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors=true
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors.displayName="Enabled"
Enabled bool `json:"enabled"`
...
// Different names for Gramola Service
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors=true
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors.displayName="Alias"
// +kubebuilder:validation:Enum=Gramola;Gramophone;Phonograph
Alias string `json:"alias,omitempty"`
...
We have put all this together, so please substitute ./pkg/apis/gramola/v1alpha1/appservice_types.go
with these.
package v1alpha1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized.
// AppServiceSpec defines the desired state of AppService
type AppServiceSpec struct {
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
// Important: Run "operator-sdk generate k8s" to regenerate code after modifying this file
// Add custom validation using kubebuilder tags: https://book-v1.book.kubebuilder.io/beyond_basics/generating_crd.html
// Flags if the the AppService object is enabled or not
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors=true
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors.displayName="Enabled"
Enabled bool `json:"enabled"`
// Flags if the object has been initialized or not
Initialized bool `json:"initialized,omitempty"`
// Different names for Gramola Service
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors=true
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors.displayName="Alias"
// +kubebuilder:validation:Enum=Gramola;Gramophone;Phonograph
Alias string `json:"alias,omitempty"`
}
// AppServiceStatus defines the observed state of AppService
type AppServiceStatus struct {
// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
// Important: Run "operator-sdk generate k8s" to regenerate code after modifying this file
// Add custom validation using kubebuilder tags: https://book-v1.book.kubebuilder.io/beyond_basics/generating_crd.html
}
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// AppService is the Schema for the appservices API defines Gramola Backend Services
// +operator-sdk:gen-csv:customresourcedefinitions.displayName="AppService"
// +kubebuilder:subresource:status
// +kubebuilder:resource:path=appservices,scope=Namespaced
type AppService struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec AppServiceSpec `json:"spec,omitempty"`
Status AppServiceStatus `json:"status,omitempty"`
}
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// AppServiceList contains a list of AppService
type AppServiceList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []AppService `json:"items"`
}
func init() {
SchemeBuilder.Register(&AppService{}, &AppServiceList{})
}
##``` ##cat << EOF > update_instructions.yaml ##- command: update
##EOF ##yq w -i -s update_instructions.yaml ${CSV_PATH} ##```
TODO
export DB_SCRIPTS_BASE_DIR=$(pwd)
git add . git commit -a -m "new" git push origin master git checkout -b 0.0.1 git push origin 0.0.1 git checkout master git tag -a v0.0.1 -m "version 0.0.1"
The easiest way to build, validate and push Operator Artifacts is using the operator-courier
.
You can follow the instructions given here or use virtualenv
to install it locally in your project:
- Install
virtualenv
(unless already installed)
sudo pip install virtualenv
virtualenv -p python3 venv
printf "\nvenv/\n" >> .gitignore
source venv/bin/activate
- Install
operator-courier
(venv) $ pip3 install operator-courier
NOTE: To leave the virtual env just deactivate it...
(venv) $ deactivate
We provide a script to generate the authetication token used later by operator-courier
to push artifacts to quay.io.
$ sh gen_quayio_auth_token.sh
Username: jmanning
Password:
basic am1hbm5pbmc6ZXhhbXBsZXB3
Time to push your operator artifacts corresponding to CSV 0.0.1 to quay.io.
Substitute <AUTH_TOKEN> with the token generated
operator-courier push ./deploy/olm-catalog/gramola-operator ${USERNAME} gramola-operator 0.0.1 "basic <AUTH_TOKEN>"
Test it
curl https://quay.io/cnr/api/v1/packages?namespace=cvicensa
[{"channels":null,"created_at":"2020-03-12T18:32:37","default":"0.0.1","manifests":["helm"],"name":"cvicensa/gramola-operator","namespace":"cvicensa","releases":["0.0.1"],"updated_at":"2020-03-12T18:32:37","visibility":"public"}]
Make your gramola-operator application public
For OpenShift to become aware of the Quay application repository, an OperatorSource CR needs to be added to the cluster. Login to your OpenShift cluster as an admin (such as kubeadmin) and change to the openshift-marketplace project:
oc get opsrc -n openshift-marketplace
Update the gramola-operatorsource to use your Quay USERNAME (if you are performing these steps on OSX, see note below)
$ sed -i "s|USERNAME|${USERNAME}|g" ./deploy/operator-source.yaml
On OSX use:
$ sed -i "" "s|USERNAME|${USERNAME}|g" ./deploy/operator-source.yaml
Now create your catalog resource:
oc apply -n openshift-marketplace -f ./deploy/operator-source.yaml
And check it has been created and refreshed properly.
oc get opsrc -n openshift-marketplace
NAME TYPE ENDPOINT REGISTRY DISPLAYNAME PUBLISHER STATUS MESSAGE AGE
acme-operators appregistry https://quay.io/cnr cvicensa ACME Operators ACME Succeeded The object has been successfully reconciled 64s
certified-operators appregistry https://quay.io/cnr certified-operators Certified Operators Red Hat Succeeded The object has been successfully reconciled 2d6h
community-operators appregistry https://quay.io/cnr community-operators Community Operators Red Hat Succeeded The object has been successfully reconciled 2d6h
redhat-operators appregistry https://quay.io/cnr redhat-operators Red Hat Operators Red Hat Succeeded The object has been successfully reconciled 2d6h
If you go to the Operator Hub you should see Other
category... etc.
https://operatorhub.io/preview
Have a look here
oc logs -f acme-operators-85cf48968d-9mgcg -n openshift-marketplace
Or here
oc logs -f catalog-operator-7fccd6877f-phh9p -n openshift-operator-lifecycle-manager
https://redhat-connect.gitbook.io/certified-operator-guide/ocp-deployment/openshift-deployment