cluster-api-provider-proxmox is a Cluster API infrastructure provider implementation for Proxmox VE.

cluster-api-provider-proxmox provides only infrastructure controller (ProxmoxCluster and ProxmoxMachine). To bootstrap your cluster/machine you need to provide Control Plane provider and Bootstrap provider. For example KubeadmControlPlane and KubeadmBootstrap.

You’ll need a Kubernetes cluster to run against. You can use KIND to get a local cluster for testing, or run against a remote cluster. Note: Your controller will automatically use the current context in your kubeconfig file (i.e. whatever cluster kubectl cluster-info shows).

1. Initialize Management cluster

for more information : https://cluster-api.sigs.k8s.io/user/quick-start.html#initialize-the-management-cluster

# install cluster-api components
export EXP_CLUSTER_RESOURCE_SET=true
clusterctl init --infrastructure=proxmox:v0.3.3 --config https://raw.githubusercontent.com/sp-yduck/cluster-api-provider-proxmox/main/clusterctl.yaml

Note: container images are available at ghcr.io/sp-yduck/cluster-api-provider-proxmox:<tag>

2. Create your first workload cluster

# export env variables
export CONTROLPLANE_HOST=X.X.X.X                   # control-plane vip
export PROXMOX_URL=https://X.X.X.X:8006/api2/json
export PROXMOX_PASSWORD=password
export PROXMOX_USER=user@pam

# generate manifests (available flags: --target-namespace, --kubernetes-version, --control-plane-machine-count, --worker-machine-count)
clusterctl generate cluster cappx-test --control-plane-machine-count=3 --infrastructure=proxmox:v0.3.3 --config https://raw.githubusercontent.com/sp-yduck/cluster-api-provider-proxmox/main/clusterctl.yaml > cappx-test.yaml

# inspect and edit
vi cappx-test.yaml

# apply manifests
kubectl apply -f cappx-test.yaml

3. Access your first workload cluster !!

Usually it takes 2~10 mins to complete bootstrapping the nodes.

# get workload cluster's kubeconfig
clusterctl get kubeconfig cappx-test > kubeconfig.yaml

# get node command for workload cluster
kubectl --kubeconfig=kubeconfig.yaml get node

4. Tear down your workload cluster

kubectl delete cluster cappx-test

• No need to prepare vm templates. You can specify any vm image in ProxmoxMachine.Spec.Image. CAPPX bootstrap your vm from scratch.

• Supports mutiple image format. CAPPX uses VNC websocket for downloading/installing node images so it can support multiple image format not only ISO (Proxmox API can only support ISO)

• Supports custom cloud-config (user data). CAPPX uses VNC websockert for bootstrapping nodes so it can applies custom cloud-config that can not be achieved by only Proxmox API.

• Flexible vmid/node assigning. You can flexibly assign vmid to your qemu and flexibly schedule qemus to proxmox nodes. For more details please check qemu-scheduler.

CAPPX is compatible with iso, qcow2, qed, raw, vdi, vpc, vmdk format of image. You can build your own node image and use it for ProxmoxMachine.

CAPPX relies on a few prerequisites which have to be already installed in the used operating system images, e.g. a container runtime, kubelet, kubeadm,.. .

To build your custom node image, you can use kubernetes-sigs/image-builder project.

Also there are some available out-of-box images published other communities such as Metal3. For example https://artifactory.nordix.org/ui/native/metal3/images/. Example MD can be found metal3-ubuntu2204-k8s127.yaml.

If it isn't possible to pre-install those prerequisites in the image, you can always deploy and execute some custom scripts through the ProxmoxMachine.spec.cloudInit or KubeadmConfig. Example MD can be found ubuntu2204.yaml.

CAPPX is tested with pve-manager/7.4-3/9002ab8a (running kernel: 5.15.102-1-pve).

CAPPX is tested with KubeadmControlPlane and KubeadmBootstrap.

This project aims to follow the Cluster API Provider contract.

Because Proxmox-VE does not provide LBaaS solution, CAPPX does not follow the typical infra-cluster logic. ProxmoxCluster controller reconciles only Proxmox storages used for instances. You need to prepare control plane load balancer by yourself if you creates HA control plane workload cluster. In the cluster-template.yaml, you can find HA control plane example with kube-vip.

ProxmoxMachine controller follows the typical infra-machine logic. To bootstrap your machine, CAPPX supports only cloud-config type bootstrap data secret. CAPPX is mainly tested with KubeadmControlPlane and KubeadmBootstrap.

make unit-test

export PROXMOX_URL=https://X.X.X.X:8006/api2/json
export PROXMOX_PASSWORD=password
export PROXMOX_USER=user@pam

make test

export CONTROLPLANE_HOST=X.X.X.X
export PROXMOX_URL=https://X.X.X.X:8006/api2/json
export PROXMOX_PASSWORD=password
export PROXMOX_USER=user@pam

make e2e

Are you interested in contributing to cluster-api-provider-proxmox? Do not hesitate to open GitHub issues.

Copyright 2023 Teppei Sudo.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.