Handle `image`, `env` and `args` fields updates in DaskCluster in k8s operator about dask-kubernetes HOT 3 OPEN

Permanent clusters are far less common than ephemeral clusters, so I'm not surprised this hasn't come up before.

I would be happy to propogate other changes if that's valuable to you. Do you have any interest in raising a PR to handle this?

Alternatively you could just delete and recreate the resource?

from dask-kubernetes.

Dropping daskcluster.kubernetes.dask.org/dask-primary is what I do now. I am concerned about graceful shutdown because scheduler and workers might have pending tasks. Is there a way to do this?

from dask-kubernetes.

When you delete the cluster all the Pods will be sent a SIGTERM. At this point the Dask scheduler and workers should gracefully shutdown. If they take too long to shut down then Kubernetes will send a SIGTERM, but this timeout is configurable via terminationGracePeriodSeconds.

In a long lived deployment I expect you have some application that runs work on the Dask cluster. If the Dask cluster restarts without completing a computation then it should be the job of the application to resubmit the work.

from dask-kubernetes.