Reset Dask worker to use TCP even if it was configured to use TLS in yaml file about dask-kubernetes HOT 8 OPEN

Thanks for raising this @weiwang217. I've opened #837 to resolve this. Would you mind testing that PR out and letting me know if it solves your problem?

from dask-kubernetes.

from dask-kubernetes.

We have documentation on how to do this here https://kubernetes.dask.org/en/latest/testing.html#testing-operator-controller-prs

from dask-kubernetes.

from dask-kubernetes.

Hi Jacob,

I have a suspicion that change may have caused a regression when working with replicas > 1. When I start a new DaskJob, all but one replica fails to connect to the scheduler because of duplicate names. Indeed when I run
kubectl describe pod <worker_pod>

I see:
Worker 1:

    Environment:
      DASK_WORKER_NAME:                                  simple-job-default-worker-a10a25ac26
      DASK_SCHEDULER_ADDRESS:                      tcp://simple-job-scheduler.join.svc.cluster.local:8786
      ...

Worker 2:

    Environment:
      DASK_WORKER_NAME:                                  simple-job-default-worker-00add84cde
      DASK_SCHEDULER_ADDRESS:                      tcp://simple-job-scheduler.join.svc.cluster.local:8786
      DASK_WORKER_NAME:                                   simple-job-default-worker-a10a25ac26
      DASK_SCHEDULER_ADDRESS:                      tcp://simple-job-scheduler.join.svc.cluster.local:8786

Because the last defined environment variable is the first replica, all replicas share the same name.

Do you mind taking a look?

(Context: I'm on the same team as weiwang217 and we just noticed this change recently)

from dask-kubernetes.

Thanks for reporting this @kjleftin. Why are you setting the DASK_WORKER_NAME in your config?

from dask-kubernetes.

Hi Jacob,

I'm following the example code in https://kubernetes.dask.org/en/latest/operator_resources.html#daskjob

Specifically, passing the DASK_WORKER_NAME env. variable to the dask worker CLI:

            - name: worker
              image: "ghcr.io/dask/dask:latest"
              imagePullPolicy: "IfNotPresent"
              args:
                - dask-worker
                - --name
                - $(DASK_WORKER_NAME)
                - --dashboard
                - --dashboard-address
                - "8788"

Note that I'm not setting DASK_WORKER_NAME explicitly. That is handled by the Dask Operator. (Before this change, each worker would have a different value for DASK_WORKER_NAME, but after this change, each worker has the same value).

from dask-kubernetes.

@kjleftin ok thanks for the clarification. I expect we may need to use copy to avoid this. I'll take a look at the PR and update it.

from dask-kubernetes.