jthomperoo / custom-pod-autoscaler-operator Goto Github PK

View Code? Open in Web Editor NEW

40.0 40.0 6.0 455 KB

Operator for managing Kubernetes Custom Pod Autoscalers (CPA).

License: Apache License 2.0

Makefile 1.18% Dockerfile 0.79% Go 98.03%

autoscaler autoscaling kubernetes kubernetes-operator

custom-pod-autoscaler-operator's People

Contributors

Stargazers

Watchers

Forkers

c3-e liviuchircu subrota-mondal themasterr macanmeong kanalum-crowdstrike

custom-pod-autoscaler-operator's Issues

CPA Operator has trouble with cascading foreground deletion

Hi.

We´re encountering weird/undefined/unpredictable behaviour when ArgoCD tries to delete CustomPodAutoscaler CustomeResources using foreground cascading deletion.

What we observe happening is that the Operator kills the CPA pod, and then immediately re-creates it. This seems to loop for a undetermined amount of time until it finally seems to hit some timing/conditions where it successfully deletes the pod and finished the reconcilliation. This can sometimes take a few minutes, but we've seen the operator get stuck in this loops for hours on end.

Redacted CPA Operator logs during this pod deletion/creation loop are included in the Additional Context section below.

To Reproduce
Steps to reproduce the behavior:

Deploy a CustomPodAutoscaler Custom Resource.
Delete the CustomPodAutoscaler Custom Resource you created using foreground cascading deletion.

kubectl delete custompodautoscaler <cpa-name> --cascade=foreground

The operator starts a delete/creation loop of the CPA pod for an undetermined amount of time. If it manages somehow to exit the loop then the CustomPodAutoscaler Custom Resource will finally be deleted.

Expected behavior
We expect the CPA pod to be deleted, and then immediately the CustomPodAutoscaler Custom Resource should be deleted.

Kubernetes Details (kubectl version):

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.5-eks-5e0fdde

Custom Pod Autoscaler Operator Details:
1.4.0

Additional context

...
...
...
2024-01-18T15:45:15Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:15Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:15Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:15Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:16Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:16Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:16Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:16Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:16Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:16Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:16Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:16Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:16Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:16Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:16Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:16Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:16Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:16Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:17Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    ERROR    Reconciler error    {"controller": "custompodautoscaler", "controllerGroup": "custompodautoscaler.com", "controllerKind": "CustomPodAutoscaler", "CustomPodAutoscaler": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]"}, "namespace": "[redacted]", "name": "[redacted]-latest-[redacted]-autoscaler", "reconcileID": "2c17cbdb-a4f2-4a14-89b2-6f64e610ed7b", "error": "pods \"[redacted]-latest-[redacted]-autoscaler\" is forbidden: error looking up service account [redacted]s/[redacted]-latest-[redacted]-autoscaler: serviceaccount \"[redacted]-latest-[redacted]-autoscaler\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /home/runner/work/custom-pod-autoscaler-operator/custom-pod-autoscaler-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /home/runner/work/custom-pod-autoscaler-operator/custom-pod-autoscaler-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /home/runner/work/custom-pod-autoscaler-operator/custom-pod-autoscaler-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Deleting k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Deleting k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    ERROR    Reconciler error    {"controller": "custompodautoscaler", "controllerGroup": "custompodautoscaler.com", "controllerKind": "CustomPodAutoscaler", "CustomPodAutoscaler": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "namespace": "[redacted]s", "name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler", "reconcileID": "00d5b65c-bc99-46ef-ac67-63b9a68ed01d", "error": "Operation cannot be fulfilled on rolebindings.rbac.authorization.k8s.io \"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler\": StorageError: invalid object, Code: 4, Key: /registry/rolebindings/[redacted]s/[redacted]-2a4af7aa23f0998-[redacted]-autoscaler, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 7c3d7393-e2d6-477a-b512-1023eeec24e1, UID in object meta: "}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /home/runner/work/custom-pod-autoscaler-operator/custom-pod-autoscaler-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /home/runner/work/custom-pod-autoscaler-operator/custom-pod-autoscaler-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /home/runner/work/custom-pod-autoscaler-operator/custom-pod-autoscaler-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:18Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:19Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:19Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:19Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:19Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:19Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:19Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:19Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:19Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:19Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:19Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:19Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:19Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:19Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:19Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Deleting k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:20Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Deleting k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:21Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Creating a new k8s object     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Pod currently being deleted     {"Request": {"name":"[redacted]-2a4af7aa23f0998-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-2a4af7aa23f0998-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Service Account update, retaining secrets     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    INFO    controllers.CustomPodAutoscaler    Updating k8s object     {"Request": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]s"}, "Namespace": "[redacted]s", "Name": "[redacted]-latest-[redacted]-autoscaler"}
2024-01-18T15:45:22Z    ERROR    Reconciler error    {"controller": "custompodautoscaler", "controllerGroup": "custompodautoscaler.com", "controllerKind": "CustomPodAutoscaler", "CustomPodAutoscaler": {"name":"[redacted]-latest-[redacted]-autoscaler","namespace":"[redacted]"}, "namespace": "[redacted]", "name": "[redacted]-latest-[redacted]-autoscaler", "reconcileID": "d9a6c4d0-6eb5-4d3d-bd93-2d44da86bd9a", "error": "Operation cannot be fulfilled on roles.rbac.authorization.k8s.io \"[redacted]-latest-[redacted]-autoscaler\": StorageError: invalid object, Code: 4, Key: /registry/roles/[redacted]s/[redacted]-latest-[redacted]-autoscaler, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: b705c32a-dd7e-4bad-9d59-7e6133e8bed3, UID in object meta: "}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /home/runner/work/custom-pod-autoscaler-operator/custom-pod-autoscaler-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /home/runner/work/custom-pod-autoscaler-operator/custom-pod-autoscaler-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /home/runner/work/custom-pod-autoscaler-operator/custom-pod-autoscaler-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226
...
...
...

Pausing autoscaling

Is your feature request related to a problem? Please describe.
It would be helpful if there was a way to easily pause and unpause autoscaling (ex. for cluster maintenance).

Describe the solution you'd like
*To avoid confusion, the pod(s) created by this custom-pod-autoscaler-operator repo will be referred to as the operator and the pod(s) created by the other custom-pod-autoscaler repo will be referred to as the autoscaler. The pods that are being scaled by the autoscaler (ex. a microservice) will be referred to as the target.

The proposed solution is to use a specific annotation (key/value pair) in the autoscaler's configuration file (ex. Helm chart or .yaml file). When the annotation is present, it will pause autoscaling. When it is not present, autoscaling will run as normal. See Additional context for more information on why this approach is good (tldr: copy KEDA).

The annotation's key would be a special string that the operator will look for. The value would be the number of replicas you desire for the target while autoscaling is paused (let's call this number the pausedReplicasCount). For example, the presence of the annotation in the example below would pause autoscaling and set the replica count of the target to 42.

apiVersion: custompodautoscaler.com/v1
kind: CustomPodAutoscaler
metadata:
  name: custom-autoscaler
  annotations:
    "v1.custompodautoscaler.com/paused-replicas": "42"
spec:
  template:

The proposed implementation for this would mainly involve a change to the function

func (r *CustomPodAutoscalerReconciler) Reconcile(context context.Context, req ctrl.Request)

found in controllers/custompodautoscaler_controller.go.

The proposed implementation is the following:

The operator detects the pause annotation on an autoscaler
The operator deletes the autoscaler
The operator updates target to have pausedReplicasCount replicas

If the annotation is not detected, the operator will reconcile the autoscaler as usual. This includes creating an autoscaler if one does not exist

Describe alternatives you've considered
The proposed implementation is to have the operator do everything by itself.

One alternative is to have the autoscaler do everything by itself. This would be difficult because it is not trivial to have the autoscaler query for metadata information about itself (via something like the downward API). It is also not feasible for the autoscaler to delete itself while paused (and then come back when unpaused). It is possible to let the autoscaler remain intact and simply disable its autoscaling functionality while paused, but this would be more trouble than it's worth. Seeing that the autoscaler has been deleted will be a clearer sign to cluster admins that things are going as expected.

The second and more promising alternative is to have the operator and autoscaler share the work. The workflow would look something like this:

The operator detects the annotation and sends the pausedReplicasCount to the autoscaler
The autoscaler updates target to have pausedReplicasCount replicas
The autoscaler informs the operator when this change has been successfully made
The operator deletes the autoscaler

The main advantage of this alternative approach is that the autoscaler already has the necessary infrastructure to scale pods while the operator does not. The operator also by default does not have RBAC permission to scale pods (this would require developers to allow that in the Helm chart or other configuration file)

The main disadvantage of this approach is that it would be quite complicated to implement. It would require changes to both the custom-pod-autoscaler-operator and custom-pod-autoscaler repos. This would make it much more difficult to review and understand the changes.

Overall it makes more sense to go for the simpler approach and have the operator do everything. It is not too difficult to give the operator some basic scaling capabilities (it is more or less a copy-and-paste from the custom-pod-autoscaler repo). It is also not very costly for the operator to have another client for scaling. These changes could be made with just a few additional client dependencies that the custom-pod-autoscaler repo also uses. Hopefully these changes will be easy to review and understand.

Additional context
KEDA recently implemented a feature to pause and unpause HPA autoscaling via a metadata annotation. It makes sense for CPA to have similar functionality and implementation of the feature. I mostly remained faithful to the KEDA implementation for checking for a pause annotation. However, the KEDA implementation for setting the target replicas to pausedReplicasCount is rather complicated due to their architecture being much more complex. KEDA also just uses built-in Kubernetes HPA pods for autoscaling, while this project uses its own custom pods with source code in a separate repo. I think these proposed changes to the CPA operator will be a bit easier for other developers to understand while still keeping the operator somewhat lightweight.

roleRequiresMetricsServer flag does not generate role that works for custom or external metrics

Describe the bug
The roleRequiresMetricsServer flag does not generate a role that works for custom or external metrics - it only
has permission to access the metrics.k8s.io group, but needs permission to access custom.metrics.k8s.io and external.metrics.k8s.io also.

To Reproduce
Steps to reproduce the behavior:

Deploy the CPAO.
Install a CPA that queries custom metrics and uses the roleRequiresMetricsServer flag to generate a role.
The CPA will fail with this error:

invalid metrics (1 invalid out of 1), first error is: failed to get pods metric: unable to get metric my_custom_metric: unable to fetch metrics from custom metrics API: pods.custom.metrics.k8s.io "*" is forbidden: User "system:serviceaccount:default:my-cpa" cannot get resource "pods/my_custom_metric" in API group "custom.metrics.k8s.io" in the namespace "default"

Expected behavior
The role created should have access to K8s custom and external metrics.

Kubernetes Details (kubectl version):
All K8s versions.

Custom Pod Autoscaler Operator Details:
v1.1.0

Additional context
The fix here is to create a role with the correct permissions.

Workaround
Just include the provisionRole: false flag in your CPA definition, and create a custom role that is like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: <NAME OF YOUR CUSTOM POD AUTOSCALER>
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - replicationcontrollers
  - replicationcontrollers/scale
  verbs:
  - '*'
- apiGroups:
  - apps
  resources:
  - deployments
  - deployments/scale
  - replicasets
  - replicasets/scale
  - statefulsets
  - statefulsets/scale
  verbs:
  - '*'
- apiGroups:
  - metrics.k8s.io
  resources:
  - '*'
  verbs:
  - '*'
- apiGroups:
  - custom.metrics.k8s.io
  resources:
  - '*'
  verbs:
  - '*'
- apiGroups:
  - external.metrics.k8s.io
  resources:
  - '*'
  verbs:
  - '*'

Update pod image on CRD image changes

Thanks for the handy project!

I noticed that the reconcile process doesn't seem to catch changes in the image, so I have to delete the CPA and re-create when testing.

Use scaleTargetRef rather than selector

Use scaleTargetRef rather than selector, better consistency with Horizontal Pod Autoscaler.

apiVersion: custompodautoscaler.com/v1alpha1
kind: CustomPodAutoscaler
metadata:
  name: player-count-scaler
spec:
  image: example-python-custom-pod-autoscaler:latest
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: flask-metric

Add linting

Failing linting should fail CI

Adjust env vars to be consistent with rest of YAML config

Environment variables SELECTOR and WATCH_NAMESPACE should be changed to selector and namespace respectively. This ensures that configuration via env vars and YAML are consistent.

multiple service account token being created

deployed operator through https://github.com/jthomperoo/custom-pod-autoscaler-operator/releases/download/v1.0.0/cluster.yaml

Expected: 1 secret - 1 service account - 1 cpa

Actual: multiple secrets - 1 service account(only 1 secret mountable) - 1 cpa

steps to reproduce:

create any cpa resource
kill the cpa pod (in our case, the pod was scheduled to other node)

when a cpa is being. re-span/killed, I was expecting either of them:

service account and service account token being garbage collected
reconciling service account using the previously created service account token

Looks like neither of them happened, thus resulting in multiple unused service account token being created?

Deployment YAML for operator should be version tagged

The deployment YAML for an operator should use an image that is tagged to a specific release version - the release that the YAML is bundled with.

Add support for Argo Rollouts

Is your feature request related to a problem? Please describe.
The CPAO does not support Argo Rollouts, this is a requested feature for the CPA here: jthomperoo/custom-pod-autoscaler#145

Describe the solution you'd like
Add support for Argo Rollouts.

Additional context
This should be a new flag, Argo Rollouts should not be included in the Role definitions by default and only added when requested.

When a release is created, the YAML config should be bundled into the GitHub release to allow for easier installs.

Allow entire Pod specification rather than an image

Allow full use of Kubernetes Pod definition within a CPA definition, allowing use of volumes etc.

Add testing

Expose namespace to CPA container

The CPA container should be able to determine which namespace it is operating in, this should be delivered in the form of an environment variable CPA_NAMESPACE.

Allow operator to modify k8s object owners

This would allow CPAs to modify standard resources, such as roles, and would still allow them to be managed through the operator as they would all be deleted together.

Namespaced deployment fails to watch namespaced resources

Describe the bug
It seems that a namespaced deploy no longer works, since the operator uses some APIs to watch for resource changes but it relies on having cluster wide access to do so.

https://github.com/jthomperoo/custom-pod-autoscaler-operator/blob/master/controllers/custompodautoscaler_controller.go#L316-L323

This results in this issue logged out in the operator:

E0417 16:02:12.786046       1 reflector.go:127] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:225: Failed to watch *v1.CustomPodAutoscaler: failed to list *v1.CustomPodAutoscaler: custompodautoscalers.custompodautoscaler.com is forbidden: User "system:serviceaccount:test:custom-pod-autoscaler-operator" cannot list resource "custompodautoscalers" in API group "custompodautoscaler.com" at the cluster scope

To Reproduce
Steps to reproduce the behavior:

Create a namespace, e.g. test
Do a namespaced deploy into that namespace (see here https://github.com/jthomperoo/custom-pod-autoscaler-operator/blob/master/INSTALL.md#namespace-scoped-install)
Deploy a CPA into the namespace.
Check the operator logs, it will print the message above and fail to set up any resources (pod/role/etc.) for the CPA.

Expected behavior
When you are using a namespaced operator and you deploy a namespaced CPA then it should provision all the required resources.

Kubernetes Details (kubectl version):

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:38:05Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6+k3s1", GitCommit:"3228d9cb9a4727d48f60de4f1ab472f7c50df904", GitTreeState:"clean", BuildDate:"2022-01-25T01:27:44Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}

Custom Pod Autoscaler Operator Details:

v1.2.0

Additional context
Found by @liviuchircu in #88

Missing WATCH_NAMESPACE variable

Hi Jamie,

I ran a cluster scoped install, but the operator is in CrashLoopBackOff state probably due to the missing WATCH_NAMESPACE variable:

{"level":"info","ts":1589173446.6930842,"logger":"cmd","msg":"Go Version: go1.13.6"}
{"level":"info","ts":1589173446.6931605,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1589173446.6931858,"logger":"cmd","msg":"Version of operator-sdk: v0.10.0"}
{"level":"error","ts":1589173446.6932368,"logger":"cmd","msg":"Failed to get watch namespace","error":"WATCH_NAMESPACE must be set","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nmain.main\n\tcustom-pod-autoscaler-operator/cmd/manager/main.go:89\nruntime.main\n\t/opt/hostedtoolcache/go/1.13.6/x64/src/runtime/proc.go:203"}

The initial value in dist/cluster/operator.yaml is set to "". Seeting it to "default" yields the same error. Any ideas?

Create easier install guide

Install guide should outline how to do both a cluster-scoped install and a namespaced install.

Handle service account/role/role binding creation for CPAs

At the moment when a CPA is created it is just a deployment, however this means by default it does not have the permissions required to manage deployments/pods. On CPA creation these should be created:

Role (namespace specific)
Service Account
Role Binding (to tie role and service account together)
On CPA deletion these should also be removed.
This will require the Operator role to have permission to create these resources.
https://kubernetes.io/docs/reference/access-authn-authz/rbac/

The Custom Pod Autoscaler Operator should include an option for provisioning a role with access to the metrics server

Is your feature request related to a problem? Please describe.
With jthomperoo/custom-pod-autoscaler/issues/126 Custom Pod Autoscalers will be able to automatically query the K8s metrics server if configured to do so. At the minute this requires a custom role to be included, alongside the provisionRole:false flag.

Describe the solution you'd like
A roleRequiresMetricsServer flag, defaulted to false would be useful, allowing the autoscalers described above to simply include that flag and then they would have a role provisioned that would allow them access to the metrics server.

Add Helm chart for deploying

Add Helm chart for publishing, allow install via Helm.

Creating a release doesn't work, image not tagged correctly

Related to #5 - when a release is created on GitHub it should result in a new image to be tagged in the format custompodautoscaler/operator:<RELEASE_NAME> at the minute this fails as the full release reference is used, e.g. custompodautoscaler/operator:refs/tags/0.1.0 rather than custompodautoscaler/operator:0.1.0.

CustomPodAutoscaler/v1alpha1

Hi,

I need to CustomPodAutoscaler/v1alpha1 version. I need to add initialReadinessDelay metric like in below but v1 couldn't allow add this metric. Could you please share v1alpha1 version?

apiVersion: custompodautoscaler.com/v1alpha1
kind: CustomPodAutoscaler
metadata:
name: horizontal-pod-autoscaler-example
spec:
template:
spec:
containers:
- name: horizontal-pod-autoscaler-example
image: horizontal-pod-autoscaler:latest
imagePullPolicy: Always
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: php-apache
config:
- name: minReplicas
value: "1"
- name: maxReplicas
value: "3"
- name: interval
value: "30000"
- name: initialReadinessDelay
value: "45"

Thanks :)

Can i use CPA will work for predication and sudden spike autoscaling on EKS ?

@jthomperoo

Can you please let me know CPA will work for predication and sudden spike autoscaling on EKS ?

Thanks !

Unable to apply CustomPodAutoscaler

I was able to install the CustomPodAutoscaler operator via helm (cluster wide) on our AKS Cluster and the pod started without any issues . But when I try to install an CustomPodAutoscaler via the below YAML it throws me an error saying resource mapping not found for name.

error: resource mapping not found for name: "telegraf-operator" namespace: "" from ".\\Custom HPA.yml": no matches for kind "CustomPodAutoscaler" in version "custompodautoscaler.com/v1alpha1"
ensure CRDs are installed first

Below is the YAML I used :

apiVersion: custompodautoscaler.com/v1alpha1
kind: CustomPodAutoscaler
metadata:
  name: telegraf-operator
spec:
  template:
    spec:
      containers:
      - name: horizontal-pod-autoscaler-example
        image: horizontal-pod-autoscaler:latest
        imagePullPolicy: Always
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: telegraf-operator
  config: 
    - name: minReplicas
      value: "2"
    - name: maxReplicas
      value: "5"
    - name: interval
      value: "30000"
    - name: tolerance
      value: "0.2"
    - name: downscaleStabilization
      value: "120"
    - name: metrics
      value: |
        - type: Resource
          resource:
            name: cpu
            target:
              type: Utilization
              averageUtilization: 50
---

The reference deployment does exist on the namespace and all that was verified . Unsure if I am missing some thing here and need help figuring this out.

Helm chart only works if deployed in default namespace

Just trying to deploy this in its own namespace on my cluster. The ClusterRoleBinding is hardcoded to look for the service account in the "default" namespace, where this should be templated out to look in {{ .Release.Namespace }}.

reconcile would delete existing pod during Update event

Describe the bug
On my local environment (docker-desktop) a phpa pod is being deleted every 10 minutes or so.

To Reproduce
Steps to reproduce the behavior:

Deploy operator and phpa
Run one of the examples coming with https://github.com/jthomperoo/predictive-horizontal-pod-autoscaler
Watch for an UpdateEvent event sent to operator. This may take as long as 10+ min.
Once event is consumed by operator compare between old and new states attached to an event object. Even if old and new states appear identical the pod will be scheduled for deletion inside Reconcile method.

Expected behavior
There should be evaluation of pod state differences before arriving at decision to delete the pod.

Kubernetes Details (kubectl version):
kubectl version:
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:32:32Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
docker-desktop version:
4.5.0 (74594)
docker version:
Client:
Cloud integration: v1.0.22
Version: 20.10.12
API version: 1.41
Go version: go1.16.12
Git commit: e91ed57
Built: Mon Dec 13 11:46:56 2021
OS/Arch: darwin/amd64
Context: default
Experimental: true

Server: Docker Desktop 4.5.0 (74594)
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
Go version: go1.16.12
Git commit: 459d0df
Built: Mon Dec 13 11:43:56 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.12
GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0

Custom Pod Autoscaler Operator Details:
operator version: v1.2.0

Additional context
Probably being caused by #53

Upgrade Operator SDK to stable version v1.1.0 to get future backwards compatible bug fixes

Add unit tests for controller

Add unit tests for k8s reconciling

Deploy built images to Docker hub

Each commit should result in a new image being pushed to Docker Hub.

Add CI

GitHub Actions or Travis?

CI should build image.

Operator for cluster scope refers to incorrect service account

Operator refers to custom-pod-autoscaler-operator when it should refer to custom-pod-autoscaler-operator-cluster.

Update CPA roles to include new resources to manage

Following jthomperoo/custom-pod-autoscaler#50

New permissions should be added to roles to handle using the scaling API, alongside managing these resources:

deployments
replicationcontrollers
replicasets
statefulsets

Ability to specify the serviceAccount to use

I'm running this in GKE using their workload identity support to provide a google service account with proper permissions to access their metrics API. In order to make that happen, there's an annotation on the service account that needs to be set. As such, it'd be handy to able to specify the SA to use for the Pod.

Thanks!

Selector not being delivered into container

Selector not currently being delivered as an environment variable into the container, so the deployment selection isn't working.

Operator doesn't apply pod's ObjectMeta defined on the CustomPodAutoscaler CR.

Describe the bug
Operator doesn't apply pod's ObjectMeta defined on the CustomPodAutoscaler CR.

To Reproduce
Steps to reproduce the behavior:

Deploy this CPA

apiVersion: custompodautoscaler.com/v1
kind: CustomPodAutoscaler
metadata:
  name: cpa-test
spec:
  template:
    metadata:
      name: cpa-test-pod
      labels:
        test: "true"
...

kubectl get po
See pod with name cpa-test insead of cpa-test-pod

Expected behavior
The pod should have the name specified in the PodTemplateSpec.

Kubernetes Details (kubectl version):
1.21

Custom Pod Autoscaler Operator Details:
v1.2.0

Additional context
The issue I think is that the CPA custom resource openapi schema doesn't define the values for PodSpecTemplate's metadata, thus the CPA CR is stored incompletely by the kube api-server.

❯ k get cpa -o yaml cpa-test
apiVersion: custompodautoscaler.com/v1
kind: CustomPodAutoscaler
metadata:
  name: cpa-test
spec:
  config:
   ...
  roleRequiresMetricsServer: true
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: hello-kubernetes
  template:
    metadata: {}
    spec:
      containers:
      - image: ...
        imagePullPolicy: IfNotPresent
        name: cpa-test
      terminationGracePeriodSeconds: 5

notice the metadata: {} although I've defined it on the applied CPA resource

Validate CPA on creation/update

CPAs should be validated when they are created/edited.

Two CPAs should not exist with the same selector and namespace.
CPA requires selector.
CPA requires image.
CPA requires name.

Deploy release images to Docker hub

If a new release is built on GitHub, it should result in a new image being pushed to Docker Hub and that image should be tagged as latest.

Could not create ServiceMonitor object

I installed it with the following command:
VERSION=v0.5.0
curl -L "https://github.com/jthomperoo/custom-pod-autoscaler-operator/releases/download/${VERSION}/cluster.tar.gz" | tar xvz --to-command 'kubectl apply -f -'

and Check the pod log and find the following error:
{"level":"info","ts":1591752793.199578,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"an empty namespace may not be set during creation"}
How should i solve it

Add unit tests for add custompodautoscaler controller

Check that the code-gen run by the CI it is the same as what is currently committed

Add a Changelog

Changing Pod name through metadata can cause 2 pods to be created

Describe the bug
Change introduced in #93 means that you can modify the pod name independently of the CPA name, this causes issues because you can change the name of the pod, so the old pod won't get any reconcile event, meaning it wont be deleted and you will end up with two pods.

To Reproduce
Steps to reproduce the behavior:

Deploy a CPA with some metadata for the Pod name:

spec:
  template:
    metadata:
      name: test-pod
...

Wait and see the Pod is created.
Update the metadata to change the Pod name:

spec:
  template:
    metadata:
      name: test-pod-2
...

Wait and see that there are now 2 pods.

Expected behavior
The old pod should be deleted as the new one is created.

Kubernetes Details (kubectl version):

v1.22.7

Custom Pod Autoscaler Operator Details:
Unreleased CPA based off master (commit 1efee1a5076b20aaccb467b39314fcc2fae68ec4)

Additional context
N/A

Add status field information

Is your feature request related to a problem? Please describe.
To help with debugging and to provide more context the CPA should give better information when reporting its status.

Describe the solution you'd like
The status should include these fields:

lastScaleTime is the last time the CustomPodAutoscaler scaled the number of pods, used by the autoscaler to keep track of downscale stabilization between restarts.
currentReplicas is the current number of replicas of pods managed by this autoscaler, as last seen by the autoscaler.
desiredReplicas is the desired number of replicas of pods managed by this autoscaler, as last calculated by the autoscaler.
currentMetrics is the last set of metrics gathered by this autoscaler, the last metrics gathered in the metric gathering stage
currentEvaluation is the last evaluation decision retrieved by this autoscaler, the last evaluation decision retrieved in the evaluation stage
reference is the identifier for the resource being scaled in the format <api-version>/<api-kind/<name>.

Of these fields only the reference could be populated by the operator, the rest would rely on the CPA itself updating them.

Along with these new status fields the CPA should also report some of this information as printer columns:

Last Scale Time is the last time the CustomPodAutoscaler scaled the number of pods.
Current Replicas is the current number of replicas of pods managed by this autoscaler.
Desired Replicas is the desired number of replicas of pods managed by this autoscaler.
Reference is the identifier for the resource being scaled in the format <api-version>/<api-kind/<name>.

Describe alternatives you've considered
N/A

Additional context
Would allow jthomperoo/custom-pod-autoscaler#168 to be implemented.

Add Pull Policy for creating a Custom Pod Autoscaler

Pull Policy required to allow local development of CPAs.
Default Pull Policy should be IfNotPresent.

Operator fails to install on K8s >= v1.18.x

Describe the bug
The CPAO fails to install on K8s >= v1.18.x

To Reproduce
Steps to reproduce the behavior:

Install the operator on a K8s cluster v1.18.x:

VERSION=v1.0.2
HELM_CHART=custom-pod-autoscaler-operator
helm install ${HELM_CHART} https://github.com/jthomperoo/custom-pod-autoscaler-operator/releases/download/${VERSION}/custom-pod-autoscaler-operator-${VERSION}.tgz

Error reported, install fails:

Error: CustomResourceDefinition.apiextensions.k8s.io "custompodautoscalers.custompodautoscaler.com" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[template].properties[spec].properties[containers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[template].properties[spec].properties[initContainers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property]

Expected behavior
CPAO should install correctly.

Kubernetes Details (kubectl version):
K8s v1.18.x

Custom Pod Autoscaler Operator Details:
`v1.0.2

Additional context
This is blocked until operator-framework/operator-sdk#3235 is resolved. If this is required urgently a temporary fix could be added to manually modify the CRD YAML, to make the protocol property required.

WORKAROUND
A workaround for this is to install using the following steps:

Download the helm release to install:

VERSION=v1.0.2
wget https://github.com/jthomperoo/custom-pod-autoscaler-operator/releases/download/${VERSION}/custom-pod-autoscaler-operator-${VERSION}.tgz

Untar the helm package and then delete it:

tar -xzf custom-pod-autoscaler-operator-${VERSION}.tgz && rm custom-pod-autoscaler-operator-${VERSION}.tgz

Edit custom-pod-autoscaler-operator/templates/crd/custompodautoscaler.com_custompodautoscalers.yaml, replacing:

                              required:
                              - containerPort
                              type: object
                            type: array
                            x-kubernetes-list-map-keys:
                            - containerPort
                            - protocol

with:

                              required:
                              - containerPort
                              - protocol
                              type: object
                            type: array
                            x-kubernetes-list-map-keys:
                            - containerPort
                            - protocol

There should be two instances of this to replace.

Tar the edited helm package back up:

tar -czvf custom-pod-autoscaler-operator-${VERSION}.tgz custom-pod-autoscaler-operator/

Install using the modified helm package:

HELM_CHART=custom-pod-autoscaler-operator
helm install ${HELM_CHART} custom-pod-autoscaler-operator-${VERSION}.tgz

The CPAO should install successfully, check with helm ls and kubectl get pods.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.