CPA Operator has trouble with cascading foreground deletion about custom-pod-autoscaler-operator HOT 3 CLOSED

Thanks for raising this issue and including detailed steps to reproduce and error logs, very useful.

I've been able to recreate this.

I think what's happening here is with a foreground cascading deletion it first updates some metadata values on the object being deleted (the CPA resource in this instance):

The Kubernetes API server sets the object's metadata.deletionTimestamp field to the time the object was marked for deletion.
The Kubernetes API server also sets the metadata.finalizers field to foregroundDeletion.

This will cause reconciles to occur within the operator.

Then the dependent resources are deleted:

After the owner object enters the deletion in progress state, the controller deletes the dependents.

When this occurs at the same time as the update reconciles the operator will try to recreate the resources that have just been deleted.

I think this is what causes the infinite loop, the operator is trying to recreate the dependent resources which Kubernetes is deleting at the same time.

I think if I check if the resource is marked for deletion and skip the reconcile stages if that is occurring it should fix this.

I'll keep this issue updated on my progress on this.

from custom-pod-autoscaler-operator.

Created a pull request with a fix for this in it. It appeared to fix it for my setup.

Once I finish testing this I will merge this in and cut a new version of the operator.

Feel free to use the branch with the fix on it if you want to test this out/need it fixed before then.

It's available with the image custompodautoscaler/operator:73793a3, you can manually edit your helm charts to use this image instead of the templated value if you want to install this.

from custom-pod-autoscaler-operator.

This is now available in version v1.4.1.

from custom-pod-autoscaler-operator.