Sveltos is tool for policy driven management of kubernetes resources and helm charts in ClusterAPI powered Kubernetes clusters. Sveltos provides declarative APIs to provision features like Helm charts, ingress controllers, CNIs, storage classes and other resources in a given set of Kubernetes clusters. Sveltos is a freely available and open source. Sveltos is very lightweight and can be installed onto any Kubernetes clusters in minutes.
The project follows the Kubernetes Operator pattern and it uses Controllers which provides a reconcile function responsible for synchronizing resources until the desired state is reached on the cluster.
The project requires ClusterAPI to be installed in such cluster. ClusterAPI is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters.
The idea is simple:
- from the management cluster, selects one or more
clusters
with a Kubernetes label selector; - lists which
features
need to be deployed on such clusters.
where term:
clusters
represents CAPI cluster;features
represents either an helm release or a Kubernetes resource.
Here is an example of how to require that any CAPI Cluster with label env: prod has following features deployed:
- Kyverno helm chart (version v2.5.0)
- kubernetes resource(s) contained in the referenced Secret: default/storage-class
- kubernetes resource(s) contained in the referenced ConfigMap: default/contour.
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
name: demo
spec:
clusterSelector: env=prod
syncMode: Continuous
helmCharts:
- repositoryURL: https://kyverno.github.io/kyverno/
repositoryName: kyverno
chartName: kyverno/kyverno
chartVersion: v2.5.0
releaseName: kyverno-latest
releaseNamespace: kyverno
helmChartAction: Install
policyRefs:
- name: storage-class
namespace: default
kind: Secret
- name: contour-gateway
namespace: default
kind: ConfigMap
As soon as a CAPI cluster is a match for above ClusterProfile instance, all referenced features are automatically deployed in such cluster.
To see the full demo, have a look at this youtube video
- Flexible cluster selection (see video)
- Sync Modes: OneTime, Continuous or DryRun (see video)
- Snapshotting (see video)
- Rollback (see video)
- Conflict detection
- Declaritive API and CLI sveltosctl
Just execute make create-cluster
The command will:
- create a KIND cluster on your laptop;
- install ClusterAPI;
- create a CAPI Cluster with Docker as infrastructure provider;
- install CRD and the Deployment from this project;
- create a ClusterProfile instance;
- modify CAPI Cluster labels so to match ClusterProfile selector.
First, enable Kubernetes in Docker Desktop
1. from the Docker Dashboard, select the Setting icon, or Preferences icon if you use a macOS.
2. select Kubernetes from the left sidebar.
3. next to Enable Kubernetes, select the checkbox.
4. select Apply & Restart to save the settings and then click Install to confirm.
Second, install ClusterAPI
https://cluster-api.sigs.k8s.io/user/quick-start.html#install-clusterctl
Third, initialize the management cluster
clusterctl init --infrastructure docker
Fourth, install Sveltos applying this manifest YAML
kubectl create -f https://raw.githubusercontent.com/projectsveltos/cluster-api-feature-manager/main/manifest/manifest.yaml
Finally, if you have prometheus operator installed in your management cluster, you can apply following ServiceMonitor
kubectl create -f https://raw.githubusercontent.com/projectsveltos/cluster-api-feature-manager/main/manifest/service_monitor.yaml
First, install ClusterAPI
https://cluster-api.sigs.k8s.io/user/quick-start.html#install-clusterctl
Second, initialize the management cluster
https://cluster-api.sigs.k8s.io/user/quick-start.html#initialize-the-management-cluster
Third, install Sveltos applying this manifest YAML
kubectl create -f https://raw.githubusercontent.com/projectsveltos/cluster-api-feature-manager/main/manifest/manifest.yaml
The clusterSelector field is a Kubernetes label selector that matches against labels on CAPI clusters.
SyncMode has three possible options: Continuous
, OneTime
and DryRun
.
OneTime means that when a CAPI Cluster matches a ClusterProfile instance, all the ClusterProfile's helm charts and Kubernetes resources at that point in time will be installed into the CAPI Cluster. Any change to ClusterProfile (for instance adding one more helm chart or referencing a new ConfigMap/Secret) will not be deployed into the already matching CAPI Clusters.
Continuous means that any change to ClusterProfiles (referencing a new helm chart or a new ConfigMap) will be immediately reconciled into the matching CAPI Clusters. Reconciled can mean three things:
- deploy (when CAPI Cluster matches a ClusterProfile)
- update (when ClusterProfile configuration changes and/or any of the referenced ConfigMap/Secret changes);
- withdraw (when ClusterProfile stops listing an helm release and/or referencing to a ConfigMap/Secret or any of the referenced ConfigMap/Secret is deleted).
Let's take an example. ClusterProfile is referencing, in the policyRefs section, following ConfigMap containing a Kyverno policy preventing any ServiceAccount (except cluster-admin) to create/delete a Gateway instance.
apiVersion: v1
kind: ConfigMap
metadata:
name: kyverno-disallow-gateway
namespace: default
data:
kyverno.yaml: |
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: no-gateway
annotations:
policies.kyverno.io/title: Block Create,Update,Delete of Gateway instances
policies.kyverno.io/severity: medium
policies.kyverno.io/subject: Gateway
policies.kyverno.io/description: >-
Management cluster admin controls Gateway's configurations.
spec:
validationFailureAction: enforce
background: false
rules:
- name: block-gateway-updates
match:
any:
- resources:
kinds:
- Gateway
exclude:
any:
- clusterRoles:
- cluster-admin
validate:
message: "Gateway's configurations is managed by management cluster admin."
deny:
conditions:
- key: "{{request.operation}}"
operator: In
value:
- CREATE
- DELETE
Because ClusterProfile is referencing above ConfigMap, such Kyverno ClusterPolicy no-gateway will be deployed in any matching CAPI Clusters. Because the ClusterProfile syncMode is set to Continuous, any modifications to:
- ClusterProfiles;
- the content of the referenced ConfigMaps/Secret will result in an update in any of the matching CAPI Clusters.
For instance, continuing with above example, if we modify the content of the ConfigMap by changing the validate part to (note we are adding UPDATE on top of CREATE/DELETE):
apiVersion: v1
kind: ConfigMap
metadata:
name: kyverno-disallow-gateway
namespace: default
...
validate:
message: "Gateway's configurations is managed by management cluster admin."
deny:
conditions:
- key: "{{request.operation}}"
operator: In
value:
- CREATE
- DELETE
- UPDATE
the Kyverno policy no-gateway in each matching CAPI clusters will be update accordingly.
If we modify the ClusterProfile to not reference this policy anymore:
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
name: demo
spec:
...
policyRefs:
- name: storage-class
namespace: default
that will cause the Kyverno ClusterPolicy no-gateway to be withdrawn from any matching CAPI cluster.
In a dry-run execution, you can execute a workflow so that the entire flow of the execution (all the operations that are executed in an actual run) is shown, but no actual code is executed and there are no side effects.
Same applies here. Sometimes it is useful to see what will happen if a ClusterProfile is added/modified/deleted. So potential changes in matching CAPI clusters can be seen, validated and only then eventually applied.
A ClusterProfile in DryRun mode will make no changes to matching clusters. It will though generate a report indicating, per CAPI Cluster, what those changes would be.
There is a CRD for that, called ClusterReport. But using sveltosctl is much easier to see what those changes would be.
./bin/sveltosctl show dryrun
+-------------------------------------+--------------------------+-----------+----------------+-----------+--------------------------------+------------------+
| CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | ACTION | MESSAGE | CLUSTER FEATURES |
+-------------------------------------+--------------------------+-----------+----------------+-----------+--------------------------------+------------------+
| default/sveltos-management-workload | helm release | kyverno | kyverno-latest | Install | | dryrun |
| default/sveltos-management-workload | helm release | nginx | nginx-latest | Install | | dryrun |
| default/sveltos-management-workload | :Pod | default | nginx | No Action | Object already deployed. | dryrun |
| | | | | | And policy referenced by | |
| | | | | | ClusterProfile has not changed | |
| | | | | | since last deployment. | |
| default/sveltos-management-workload | kyverno.io:ClusterPolicy | | no-gateway | Create | | dryrun |
+-------------------------------------+--------------------------+-----------+----------------+-----------+--------------------------------+------------------+
To see DryRun mode in action, have a look at this video
When, for instance, we want to deploy a StorageClass
in a subset of CAPI Clusters, we create a ClusterProfile
instance, with an appropriate cluster selector and then we need to have this ClusterProfile reference the kubernetes resource we want to deploy in each matching cluster.
In this example, we need the StorageClass instance deployed in matching CAPI cluster. We don't need such resource in the management cluster. So where do we put the StorageClass instance in the management class so that ClusterProfile can reference it? We use Secrets
and ConfigMaps
.
Both Secrets and ConfigMaps data fields can be a list of key-value pairs. Any key is acceptable, and as value, there can be multiple objects in yaml or json format.
Secrets are preferred if the data includes sensitive information.
ClusterProfile can reference a ConfigMap and/or Secret, and the ConfigMap/Secret contains the Kubernetes resources we want to deploy in matching CAPI Clusters.
ClusterProfile's PolicyRefs section, is then a list of ConfigMaps/Secrets (identified by name and namespace and kind).
For instance by referecing following contour-gateway ConfigMap, a GatewayClass and Gateway instance will be automatically deployed in each matching CAPI clusters.
apiVersion: v1
kind: ConfigMap
metadata:
name: contour-gateway
namespace: default
data:
gatewayclass.yaml: |
kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: contour
spec:
controllerName: projectcontour.io/projectcontour/contour
gateway.yaml: |
kind: Namespace
apiVersion: v1
metadata:
name: projectcontour
---
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: contour
namespace: projectcontour
spec:
gatewayClassName: contour
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
HelmCharts section in above CRD instance, is a list of Helm charts we want to deploy in each CAPI matching clusters.
Helm is a CNCF graduated project that serves as a package manager widely used in the community. This project uses Helm golang SDK to deploy all helm charts listed in above ClusterProfile instance.
There is many-to-many mapping between Clusters and ClusterProfile:
- Multiple ClusterProfiles can match with a CAPI cluster;
- Multiple CAPI clusters can match with a single ClusterProfile.
As for DryRun, sveltosctl can be used to properly list all deployed features per CAPI cluster:
./bin/sveltosctl show features
+-------------------------------------+--------------------------+-----------+----------------+---------+-------------------------------+------------------+
| CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | CLUSTER PROFILES |
+-------------------------------------+--------------------------+-----------+----------------+---------+-------------------------------+------------------+
| default/sveltos-management-workload | helm chart | kyverno | kyverno-latest | v2.5.0 | 2022-10-11 20:59:18 -0700 PDT | mgianluc |
| default/sveltos-management-workload | helm chart | nginx | nginx-latest | 0.14.0 | 2022-10-11 20:59:25 -0700 PDT | mgianluc |
| default/sveltos-management-workload | helm chart | mysql | mysql | 9.3.3 | 2022-10-11 20:43:41 -0700 PDT | mgianluc |
| default/sveltos-management-workload | :Pod | default | nginx | N/A | 2022-10-12 09:33:25 -0700 PDT | mgianluc |
| default/sveltos-management-workload | kyverno.io:ClusterPolicy | | no-gateway | N/A | 2022-10-12 09:33:25 -0700 PDT | mgianluc |
+-------------------------------------+--------------------------+-----------+----------------+---------+-------------------------------+------------------+
Otherwise, a new CRD is introduced to easily summarize which features (either helm charts or kubernetes resources) are deployed in a given CAPI cluster because of one or more ClusterProfiles. Such CRD is called ClusterConfiguration.
There is exactly only one ClusterConfiguration for each CAPI Cluster.
Following example shows us that because of ClusterProfile demo three helm charts and one Kyverno ClusterPolicy were deployed.
apiVersion: v1
items:
- apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterConfiguration
metadata:
creationTimestamp: "2022-09-19T21:14:15Z"
generation: 1
name: sveltos-management-workload
namespace: default
ownerReferences:
- apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
name: demo2
uid: f0f93440-95ae-4663-aff7-3b27c1135dfc
resourceVersion: "45488"
uid: efa326cb-a7e0-401c-918f-0a9252033856
status:
clusterProfileResources:
- Features:
- charts:
- appVersion: v1.7.0
chartName: kyverno-latest
chartVersion: v2.5.0
lastAppliedTime: "2022-09-19T21:14:16Z"
namespace: kyverno
repoURL: https://kyverno.github.io/kyverno/
- appVersion: 2.3.0
chartName: nginx-latest
chartVersion: 0.14.0
lastAppliedTime: "2022-09-19T21:14:23Z"
namespace: nginx
repoURL: https://helm.nginx.com/stable
- appVersion: 1.22.1
chartName: contour
chartVersion: 9.1.2
lastAppliedTime: "2022-09-19T21:17:31Z"
namespace: projectcontour
repoURL: https://charts.bitnami.com/bitnami
featureID: Helm
- featureID: Resources
resources:
- group: kyverno.io
kind: ClusterPolicy
lastAppliedTime: "2022-09-19T21:20:18Z"
name: no-gateway
owner:
kind: ConfigMap
name: kyverno-disallow-gateway
namespace: default
clusterProfileName: demo
Multiple ClusterProfiles can match same CAPI cluster. Because of that misconfiguration can happen and need to be detected.
For instance:
- ClusterProfile A references ConfigMap A containing a Kyverno ClusterPolicy called *no-gateway"
- ClusterProfile B references ConfigMap B containing a Kyverno ClusterPolicy called *no-gateway"
In such case, ClusterProfile A will be allowed to deploy the Kyverno ClusterPolicy, while ClusterProfile B will report a conflict.
Please note that in following example there is no conflict since both ClusterProfiles are referencing same ConfigMap:
- ClusterProfile A references ConfigMap A containing a Kyverno ClusterPolicy called *no-gateway"
- ClusterProfile B references ConfigMap A containing a Kyverno ClusterPolicy called *no-gateway"
Another example of misconfiguration is when two different ClusterProfiles match same CAPI Cluster(s) and both want to deploy same Helm chart in the same namespace.
In such a case, only one ClusterProfile will be elected and given permission to manage a specific helm release in a given CAPI cluster. Other ClusterProfiles will report such misconfiguration.
Sveltos exposes following metrics:
- projectsveltos_program_resources_time_seconds: time to deploy resources in a CAPI clusters (all ClusterProfiles and all CAPI clusters considered);
- projectsveltos_program_charts_time_seconds: time to deploy Helm charts in a CAPI clusters (all ClusterProfiles and all CAPI clusters considered);
- clusterNamespace_clusterName_program_resources_time_seconds: time to deploy resources in a specific CAPI clusters (all ClusterProfiles considered);
- clusterNamespace_clusterName_program_charts_time_seconds: time to deploy Helm charts in a specific CAPI clusters (all ClusterProfiles considered);
To delete the CRDs from the cluster:
make uninstall
UnDeploy the controller to the cluster:
make undeploy
Sveltos is compatible with the following versions of Cluster API:
Cluster API v1beta1 (v1.0.x ) |
Cluster API v1beta1 (v1.1.x ) |
Cluster API v1beta1 (v1.2.x ) |
|
---|---|---|---|
Sveltos v0.1.0 |
✓ | ✓ | ✓ |
This controller version can install and manage the following versions of Kubernetes:
Sveltos v0.1.0 |
|
---|---|
Kubernetes 1.23.x | ✓ |
Kubernetes 1.24.x | ✓ |
Kubernetes 1.25.x | ✓ |
Test status:
✓
tested
If you have questions, noticed any bug or want to get the latest project news, you can connect with us in the following ways:
- Open a bug/feature enhancement on github;
- Chat with us on the Slack in the #projectsveltos channel
Copyright 2022.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.