Karmada-Operator
A Karmada Operator based on the K8s Operator model built by DaoCloud, Karmada Operator(Kubernetes Armada operator) simplifies the deployment, operation and maintenance of Karmada.
TL;DR
Switch to the root
directory of the repo.
helm install karmada-operator -n karmada-operator-system --create-namespace --dependency-update ./charts/karmada-operator
Prerequisites
- Kubernetes 1.16+
- helm v3+
Demo
Installing the Chart
To install the chart with the release name karmada-operator
in namespace karmada-operator-system
:
- local installation
Switch to the root
directory of the repo.
helm install karmada-operator -n karmada-operator-system --create-namespace --dependency-update ./charts/karmada-operator
Tip: If you need to modify values.yaml, please refer to Configuration
- remote installation
First, add the Karmada-Operator chart repo to your local repository.
$ helm repo add karmada https://release.daocloud.io/chartrepo/karmada
$ helm repo list
NAME URL
karmada https://release.daocloud.io/chartrepo/karmada
With the repo added, available charts and versions can be viewed.
$ helm search repo karmada
NAME CHART VERSION APP VERSION DESCRIPTION
karmada/karmada-operator 0.0.5 0.0.5 A Helm chart for karmada-operator
Install the chart and specify the version to install with the --version argument. Replace --version=<x.x.x>
with your desired version. such as the above karmada/karmada-operator
chart version 0.0.5 , so you can use --version=0.0.5
.
$ helm --namespace karmada-operator-system upgrade -i karmada-operator karmada/karmada-operator --version=<x.x.x> --create-namespace
Release "karmada-operator" does not exist. Installing it now.
NAME: karmada-operator
LAST DEPLOYED: Thu Aug 11 15:24:01 2022
NAMESPACE: karmada-operator-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing karmada-operator.
Your release is named karmada-operator.
To learn more about the release, try:
$ helm status karmada-operator -n karmada-operator-system
$ helm get all karmada-operator -n karmada-operator-system
- example there is an example to install a karmada instance on current cluster.
Tip: List all releases using
helm list
Create a KarmadaDeployment
After installing karmada-operator, use cr
to create karmadaDeployment
Create cr
using below command in the cluster where the karmada-operator install
Tip:
- If no
secretRef
is specified incr
, the karmada instance is installed in the cluster where karmada-operator is located- If no
namespace
is specified incr
, the default install namespace is the same as the karmadaDeployment name
#deploy KarmadaDeployment demo-kmd in the cluster where the karmada-operator install
$ kubectl apply -f https://github.com/DaoCloud/karmada-operator/blob/main/examples/karmadaDeployment.yaml
The process of deploying karmada may take a few minutes, check the operation of karmada's components
$ kubectl -n demo-kmd get pod
NAME READY STATUS RESTARTS AGE
etcd-0 1/1 Running 0 19m
karmada-demo-kmd-aggregated-apiserver-6fd745748-hjbq7 1/1 Running 0 19m
karmada-demo-kmd-apiserver-429bfb7c1-6fkrk 1/1 Running 0 19m
karmada-demo-kmd-controller-manager-7b8f5699c-mvx8s 1/1 Running 3 (21m ago) 21m
karmada-demo-kmd-controller-manager-7b8f5699c-7fc8c 1/1 Running 2 (19m ago) 19m
karmada-demo-kmd-kube-controller-manager-9b56855fdc-699c5 1/1 Running 0 19m
karmada-demo-kmd-scheduler-685b997cb7-9467w 1/1 Running 0 19m
karmada-demo-kmd-scheduler-685b997cb7-85c84 1/1 Running 0 20m
karmada-demo-kmd-webhook-85c84d84c7-g6t9n 1/1 Running 0 19m
ok, karmada installation is complete.
Create multiple KarmadaDeployments
karmada-operator supports creating multiple karmadaDeployment instances in the same cluster,You can deploy multiple karmada instances multiple times using the cr
of KarmadaDeployment with different content.
List multiple KarmadaDeployments:
$ kubectl get kmd
NAME MODE PHASE STATUS AGE
demo-kmd Helm Completed true 2m42s
demo-kmd-1 Helm Completed true 2m57s
$ kubectl get pod -n demo-kmd-1
NAME READY STATUS RESTARTS AGE
etcd-0 1/1 Running 0 14m
karmada-demo-kmd-1-aggregated-apiserver-9b56855fd-gwrmf 1/1 Running 0 14m
karmada-demo-kmd-1-apiserver-6f9bbcb9c-vmxbq 1/1 Running 0 14m
karmada-demo-kmd-1-controller-manager-5d6777b75-7rmq8 1/1 Running 2 (17m ago) 17m
karmada-demo-kmd-1-controller-manager-5d6777b75-g6xzd 1/1 Running 2 (16m ago) 16m
karmada-demo-kmd-1-kube-controller-manager-76b94fb97c-6lqwv 1/1 Running 0 14m
karmada-demo-kmd-1-scheduler-797b7b787b-klcs4 1/1 Running 0 14m
karmada-demo-kmd-1-scheduler-797b7b787b-r829t 1/1 Running 0 15m
karmada-demo-kmd-1-webhook-755d649cb4-wbx5h 1/1 Running 0 15m
$ kubectl get pod -n demo-kmd
NAME READY STATUS RESTARTS AGE
etcd-0 1/1 Running 0 19m
karmada-demo-kmd-aggregated-apiserver-6fd745748-hjbq7 1/1 Running 0 19m
karmada-demo-kmd-apiserver-429bfb7c1-6fkrk 1/1 Running 0 19m
karmada-demo-kmd-controller-manager-7b8f5699c-mvx8s 1/1 Running 3 (21m ago) 21m
karmada-demo-kmd-controller-manager-7b8f5699c-7fc8c 1/1 Running 2 (19m ago) 19m
karmada-demo-kmd-kube-controller-manager-9b56855fdc-699c5 1/1 Running 0 19m
karmada-demo-kmd-scheduler-685b997cb7-9467w 1/1 Running 0 19m
karmada-demo-kmd-scheduler-685b997cb7-85c84 1/1 Running 0 20m
karmada-demo-kmd-webhook-85c84d84c7-g6t9n 1/1 Running 0 19m
Export Karmada KubeConfig
You can get karmada kubeconfig to view karmada instance information
$ KARMADA_CONFIG=$(kubectl get secret -n ${your_karmada_operator_namespace} $(kubectl get kmd demo-kmd -o jsonpath='{.status.secretRef.name}') -o jsonpath='{.data.kubeconfig}')
$ echo $KARMADA_CONFIG | base64 -d > karmada-apiserver.config
You can export KUBECONFIG for easy login the karmada instance
export KUBECONFIG=./karmada-apiserver.config
We can simply test karmada-apiserver kubeconfig ,such as List namespaces
$ kubectl --kubeconfig karmada-apiserver.config get ns
NAME STATUS AGE
default Active 124m
demo-kmd Active 89m
karmada-cluster Active 89m
kube-node-lease Active 124m
kube-public Active 124m
kube-system Active 124m
Simple Karmada Demo
We can test karmada's multi-cloud capabilities with a simple demo
Join Clusters
1. Use kubectl karmada join cluster to Karmada.
Currently you need to download karmadactl to join cluster
Karmada provides kubectl-karmada
plug-in download service since v1.2.1. You can choose proper plug-in version which fits your operator system form karmada release.
Take v1.2.1 that working with linux-arm64 os as an example:
$ wget https://github.com/karmada-io/karmada/releases/download/v1.2.1/karmadactl-linux-arm64.tgz
$ tar -zxf karmadactl-linux-arm64.tgz
$ cp karmadactl /usr/local/bin
then we join member1
and member2
cluster into karmada
--cluster-kubeconfig
is your members kubeconfig.
--kubeconfig
is your karmada apiserver kubeconfig
$ karmadactl --kubeconfig ${your_karmada_apiserver_config} join member1 --cluster-kubeconfig=${your_member1_kubeconfig}
$ karmadactl --kubeconfig ${your_karmada_apiserver_config} join member2 --cluster-kubeconfig=${your_member2_kubeconfig}
2. Show members of karmada
$ kubectl --kubeconfig ${your_karmada_apiserver_config} get clusters
NAME VERSION MODE READY AGE
member1 v1.21.12 Push True 2m53s
member2 v1.22.6 Push True 21s
Propagate a deployment by Karmada
1. Create nginx deployment in Karmada
First, create a deployment named nginx
:
# create a deployment name nginx
$ kubectl create -f https://github.com/DaoCloud/karmada-operator/blob/main/examples/nginx/deployment.yaml --kubeconfig ${your_karmada_apiserver_config}
2. Create PropagationPolicy that will propagate nginx to member cluster
Then, we need to create a policy to propagate the deployment to our member cluster.
# create a PropagationPolicy
$ kubectl create -f https://github.com/DaoCloud/karmada-operator/blob/main/examples/nginx/propagationpolicy.yaml --kubeconfig ${your_karmada_apiserver_config}
3. Check the deployment status from Karmada
You can check deployment status from Karmada, don't need to access member cluster:
$ kubectl get deployment --kubeconfig ${your_karmada_apiserver_config}
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 2/2 2 2 20s
Configuration
Global parameters
Name | Description | Value |
---|---|---|
global.imageRegistry |
Global Docker image registry | "" |
global.imagePullSecrets |
Global Docker registry secret names as an array | [] |
Common parameters
Name | Description | Value |
---|---|---|
controllerManager.labels |
deployment labels of the karmada-operator-controller-manager | "app: karmada-operator-controller-manager" |
controllerManager.replicaCount |
replicas of the karmada-operator-controller-manager deployment | 2 |
controllerManager.podAnnotations |
pod Annotations of the karmada-operator-controller-manager | {} |
controllerManager.podLabels |
pod Labels of the karmada-operator-controller-manager | {} |
controllerManager.image.registry |
Image registry of the karmada-operator-controller-manager | release.daocloud.io |
controllerManager.image.repository |
Image repository of the karmada-operator-controller-manager | kairship/karmada-operator-controller-manager |
controllerManager.image.tag |
Image tag of the karmada-operator-controller-manager | "v0.0.1" |
controllerManager.image.pullPolicy |
Image registry of the karmada-operator-controller-manager | "IfNotPresent" |
controllerManager.image.pullSecrets |
Image pullSecrets of the karmada-operator-controller-manager | [] |
controllerManager.resources |
resources of the karmada-operator-controller-manager | {} |
controllerManager.nodeSelector |
nodeSelector of the karmada-operator-controller-manager | {} |
controllerManager.affinity |
affinity of the karmada-operator-controller-manager | {} |
controllerManager.tolerations |
tolerations of the karmada-operator-controller-manager | key: node-role.kubernetes.io/master operator: Exists |
controllerManager.kubeconfigPath |
kubeconfig Path of the karmada-operator-controller-manager | "/root/.kube" |
controllerManager.localKubeconfig |
Image registry of the karmada-operator-controller-manager | "true" |
Uninstalling
Uninstalling the KarmadaDeployment
Deleting karmadaDeployment is a dangerous operation and needs to be deleted carefully. Please delete it according to the following process.
Only Delete KarmadaDeployment CR
When you want to remove CR but keep the instance on the environment,you can execute the following command
# patch disable-cascading-deletion label
$ kubectl label --overwrite kmd demo-kmd karmada.install.io/disable-cascading-deletion="true"
# disable cascade delete KarmadaDeployment
$ kubectl delete kmd demo-kmd -n demo-kmd
Delete KarmadaDeployment CR and Instance
When you want to remove CR and the instance in your environment,you can execute the following command
# cascade delete KarmadaDeployment
$ kubectl delete kmd demo-kmd -n demo-kmd
Uninstalling the Karmada-Operator
To uninstall/delete the karmada-operator
helm release in namespace karmada-operator-system
:
helm uninstall karmada-operator -n karmada-operator-system
The command removes all the Kubernetes components associated with the chart and deletes the release.
kubectl delete ns karmada-operator-system