Volumes are not removed after PV is deleted with Delete reclaimPolicy about zfs-localpv HOT 7 CLOSED

4 days have passed and the garbage collector (?) still hasn't run. Can I safely delete the dangling volumes directly with zfs destroy... ? Or is this going to break my openebs-zfs/k8s storage ?

from zfs-localpv.

You can delete the ZFSVolume resource and the volume should be deleted.

On the other one, unfortunately the delete pathway is closely linked with the PVC delete event. the Kubernetes PV delete is a handled by the kubernetes-csi components, and not the ZFS drivers. The driver doesn't actually watch for PV's and doesn't logically map ZFSVolumes to PV's.

from zfs-localpv.

Duplicate of #404.

from zfs-localpv.

Thanks for the answer.

It is not a duplicate of #404. It is the opposite. If you read my original post at point 4, I changed the reclaimPolicy to "Delete" on an unbound volume and the PV object was instantly garbage-collected and gone from the tree -- but the volume is still there.

NB: The Pod/PVC which were bound to this PV were deleted long ago and the PV was showing status Released before I did the reclaimPolicy change to delete through a patch operation.

Anyway, here is the current status :

1. I tried to delete the the dangling volumes with k delete -n openebs zv ..., like you suggested, and the delete operation hangs forever.
2. I assumed it was a similar issue on the finalizers than the one you usually see on PVs but with ZVs instead. So I tried to remove the finalizers there (since there is no PV associated with this ZV anyway, I dont have any other option) with k -n openebs patch zv $zv --type=merge -p '{"metadata": {"finalizers": null}}':

The results are not very conclusive: The delete operation at point 1. succeeded. The zv object is gone from the tree too, but zfs list still shows the volume present in the storage pool. Obviously those finalizers were needed... Maybe the delete operation hangs because you need to be root to actually run zfs destroy on the machine ? I dont know.

I assume the next step to get rid of the volume would be to manually delete the volume with zfs destroy -- which is what I did in this case -- but I have to do this for 150+ ZFSvolumes that were previously bound in my cluster. Is there no way to automate the deletion of the zfs volumes when the PVs are deleted ? From your answer I assume "NO". But what about when I delete the ZVs ?

From your anwer :

The driver doesn't actually watch for PV's and doesn't logically map ZFSVolumes to PV's.

Does that mean the reclaimPolicy field on the PVs objects has absolutely no impact on ZFS storage behavior ? Is the behavior I observed in my original post the default behavior regardless of what you put in the PV/StorageClass reclaimPolicy fields ?

There is no reclaimPolicy field on ZFSVolumes so I am confused. I just want to understand how it works.

thanks
Bernard

from zfs-localpv.

Does that mean the reclaimPolicy field on the PVs objects has absolutely no impact on ZFS storage behavior

@bernardgut
That is correct. This will be become clearer when we take a look at the design of a kubernetes CSI volume provisioner plugin. The dynamic provisioning scheme limits user interactions to the PVC level. The PV (along with the underlying volume) is managed by the CSI provisioner plugin.

Once the retainPolicy is set to 'Retain' the PV and the volume are to be left behind after the PVC delete. This plugin makes use of the CSI external provisioner helper sidecar to interface with PVs and PVCs via gRPC CSI calls. The sidecar doesn't request for a 'DeleteVolume' call when the PV is deleted. The 'DeleteVolume' call is issued only for PVC delete. A 'Retain' configured PVC cannot follow any other workflow unless the configuration was altered at the PVC level. A 'Delete' patch on the PV would make the csi sidecar look for a PVC attached to it. When it can't find one, it will remove the PV object. However, a PV delete is not grounds for a CSI call. A 'Retain' PVC's artifacts are for the administrator to deal with. This is expected behavior for a CSI plugin.

However, you can raise a feature-request for a volume garbage collector, or better yet, you can contribute the feature yourself. This feature is beyond the scope of the simple CSI plugin behaviour set.

from zfs-localpv.

This is super interesting. Thanks for the explanation.

I think I understand what you mean. However, according tho this : https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/ :

You can use finalizers to control garbage collection of resources. For example, you can define a finalizer to clean up related resources or infrastructure before the controller deletes the target resource.

I can use the Finalizers to check dangling resources and delete them. So should I create a PR on the CSI plugin repo that updates the finalizer so that delete the volumes on the node when the reclaimPolicy is set to DELETE at deletion time ? Is that what you mean ? (Or is that something you would be interested in ?)

thanks
B.

from zfs-localpv.

This is super interesting. Thanks for the explanation.

I think I understand what you mean. However, according tho this : https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/ :

You can use finalizers to control garbage collection of resources. For example, you can define a finalizer to clean up related resources or infrastructure before the controller deletes the target resource.

I can use the Finalizers to check dangling resources and delete them. So should I create a PR on the CSI plugin repo that updates the finalizer so that delete the volumes on the node when the reclaimPolicy is set to DELETE at deletion time ? Is that what you mean ? (Or is that something you would be interested in ?)

thanks B.

We'd have to create a garbage collector which would run once in a while, and would try to find ZFSvolume resourcses without a PVC attached to it, and then try to find a PV attached to it. If no PV exists, it would delete the ZFSVolume, and that would trigger a delete for the underlying dataset.

from zfs-localpv.