dask / helm-chart Goto Github PK

View Code? Open in Web Editor NEW

90.0 8.0 91.0 5.48 MB

Helm charts for Dask

Home Page: https://helm.dask.org/

Mustache 2.48% YAML 97.52%

helm-chart's Introduction

Dask Helm Charts

This repository contains Dask's two helm charts.

dask: Install Dask on Kubernetes for a single user with Jupyter and dask-kubernetes.
daskhub: Install Dask on Kubernetes for multiple users with JupyterHub and dask-gateway.

Single-user Quickstart

Users deploying Dask for a single user should use the dask/dask helm chart.

helm repo add dask https://helm.dask.org/
helm repo update
helm install my-release dask/dask

See dask for more.

Multi-user Quickstart

Users deploying Dask for multiple users should use the dask/daskhub helm chart.

helm repo add dask https://helm.dask.org/
helm repo update
helm install --name my-release dask/daskhub

See daskhub for more.

helm-chart's People

Contributors

Stargazers

Watchers

Forkers

jacobtomlinson informatics-lab tomgrek stevededalus zonca dsaiztc hdoupe gridl junfugithub danielchalef jgerardsimcock michaelwinnicki datarevenue-berlin yutiansut raybellwaves learningdymyr ndebrabander ianferreira supersom prasadph marov translucentcomputing sensibleweather stefanvangastel rohitrsh jakub-d francoisserra kazimuth wabscale ykale amcnicho tomaugspurger climacell edennuriel meoril kyprifog jcscottiii roarjn georgianaelena consideratio cheunhong srikiz mwengren arokem 0xifis dedmari jasonthedeveloper jmorton matrixmanatyrservice tonyc-blastpoint andrew-nascimento ishais123 cyrilcros kerossin fojo12 ramzi-hosisey3 lakret hercules261188 junyittcxt alexrogalskiy yuhuishi-convect oshadura matt711 rigzba21 lumy desy-hh guillaumeeb manasjain0699 vulturm dertiedemann dmtrs ericdeflaminis chadsr dxma abhiramum isvoid prantoshdas prantosh-mindtree dioguerra dashanji paulbroek cta-epfl carlos-a-enriquez complexsplit phyyou ph0tonic sangamesh-ts

helm-chart's Issues

support distributed XGBoost training

Seems like xgboost depends on rabit for distributed training with dask, and port 9091-9999 will be randomly assigned. https://github.com/dmlc/xgboost/blob/6848d0426f138c12bd7d371792fbace37991202d/python-package/xgboost/tracker.py#L144

Those configurations are not reflected in the helm chart. I run into socket error when doing distributed XGBoost training. cuDF is fine.

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<timed exec> in <module>

/opt/conda/envs/rapids/lib/python3.6/site-packages/xgboost/dask.py in train(client, params, dtrain, evals, *args, **kwargs)
    379     workers = list(_get_client_workers(client).keys())
    380 
--> 381     rabit_args = _get_rabit_args(workers, client)
    382 
    383     def dispatched_train(worker_addr):

/opt/conda/envs/rapids/lib/python3.6/site-packages/xgboost/dask.py in _get_rabit_args(worker_map, client)
    333 
    334     env = client.run_on_scheduler(_start_tracker, host.strip('/:'),
--> 335                                   len(worker_map))
    336     rabit_args = [('%s=%s' % item).encode() for item in env.items()]
    337     return rabit_args

/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/client.py in run_on_scheduler(self, function, *args, **kwargs)
   2299         Client.start_ipython_scheduler: Start an IPython session on scheduler
   2300         """
-> 2301         return self.sync(self._run_on_scheduler, function, *args, **kwargs)
   2302 
   2303     async def _run(

/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    765         else:
    766             return sync(
--> 767                 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    768             )
    769 

/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    343     if error[0]:
    344         typ, exc, tb = error[0]
--> 345         raise exc.with_traceback(tb)
    346     else:
    347         return result[0]

/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/utils.py in f()
    327             if callback_timeout is not None:
    328                 future = asyncio.wait_for(future, callback_timeout)
--> 329             result[0] = yield future
    330         except Exception as exc:
    331             error[0] = sys.exc_info()

/opt/conda/envs/rapids/lib/python3.6/site-packages/tornado/gen.py in run(self)
    733 
    734                     try:
--> 735                         value = future.result()
    736                     except Exception:
    737                         exc_info = sys.exc_info()

/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/client.py in _run_on_scheduler(self, function, wait, *args, **kwargs)
   2265         if response["status"] == "error":
   2266             typ, exc, tb = clean_exception(**response)
-> 2267             raise exc.with_traceback(tb)
   2268         else:
   2269             return response["result"]

/opt/conda/envs/rapids/lib/python3.6/site-packages/xgboost/dask.py in _start_tracker()
     46     """Start Rabit tracker """
     47     env = {'DMLC_NUM_WORKER': n_workers}
---> 48     rabit_context = RabitTracker(hostIP=host, nslave=n_workers)
     49     env.update(rabit_context.slave_envs())
     50 

/opt/conda/envs/rapids/lib/python3.6/site-packages/xgboost/tracker.py in __init__()
    143         for port in range(port, port_end):
    144             try:
--> 145                 sock.bind((hostIP, port))
    146                 self.port = port
    147                 break

OSError: [Errno 99] Cannot assign requested address

Automatically deploy new chart to gh-pages on merge to master

We should develop a CI mechanism that avoids having to manually push new charts into the gh-pages branch.

Does it make sense to have workers on the same node?

Hello,

I don't if this is a feature request or bug. Feel free to change it.

Sometimes it happens that kubernetes schedules workers on the same node, but leaves out others. I would like to know if it makes sense to schedule workers on the same node? Wouldn`t they just steal each others ressources?

If so I would like to propose to add a default affinity to the workers similar to this:

{
   "podAntiAffinity":{
      "requiredDuringSchedulingIgnoredDuringExecution":[
         {
            "topologyKey":"kubernetes.io/hostname",
            "labelSelector":{
               "matchLabels":{
                  "app":"dask",
                  "component":"worker"
               }
            }
         }
      ]
   }
}

Adding ExtraConfig in JupyterHub Results in Error

I tried adding extraConfig options to the Jupyterhub config using the Daskhub helm chart yaml file, but it resulted in the error shown below.

What happened:

I got this error:

coalesce.go:196: warning: cannot overwrite table with non table for extraConfig (map[])

and the jupyterhub public proxy service wouldn't start

What you expected to happen:
I expected the service to start when I add extraConfig options to the Jupyterhub config

Minimal Complete Verifiable Example:

jupyterhub:
  hub:
    extraConfig: |
      c.JupyterHub.hub_connect_ip = '0.0.0.0'
      c.JupyterHub.bind_url = 'http://127.0.0.1:8000'

helm upgrade --install \
    --namespace  ${NAMESPACE} \
    --values config.yaml \
    --values=secrets.yaml \
    --version=4.4.1 \
    ${RELEASE} \
    dask/daskhub

Unable to access loadbalancer url with installation on AWS

I tried deploying on AWS using helm charts using the command below:
helm install dask dask/dask --set scheduler.serviceType=LoadBalancer --set jupyter.serviceType=LoadBalancer

mentioned as part of https://docs.dask.org/en/latest/setup/kubernetes-helm.html

What happened:
Able to get the External URL for dask-jupyter by executing the command: kubectl get svc
But, I am unable to access this URL from my machine.

What you expected to happen:
Loadbalancer URL should be public and accessible

Environment:

Dask version:
Python version:
Operating System: Windows
Install method (conda, pip, source):

Helm Chart freezes

Using the default set up the helm chart is freezing for me:

helm install --wait dask --name dask --debug
[debug] Created tunnel using local port: '61002'

[debug] SERVER: "127.0.0.1:61002"

[debug] Original chart version: ""
[debug] CHART PATH: charts/dask

It freezes after this point. If I kill it everything looks good except the LoadBalancers.
Changing LoadBalancer to NodePort or ClusterIP solves the issue but then obviously I dont have the external IP which I need. I assume this means that the LoadBalancers are not able to deploy.

I've tried added permissions for AWS ELB to no avail to solve the issue. It might be worth adding some documentation for LoadBalancer deployments on AWS if there is anything unique about the dask chart in how it approaches it.

hpa config

I saw in the now deprecated stable/dask-distributed chart maxReplicas config for the horizontal pod autoscaler. That doesn't seem to have made it here. Is HPA supported?

License

I noticed today that this repo doesn't have a license.

As a Dask project it should probably have the New BSD license. However at one point we merged it into helm/charts which is Apache 2.0 and we accepted contributions during that time. We then split off again as instructed by the helm maintainers.

As the Apache 2.0 license is slightly less permissive (it is more prescriptive about including the license and other info) I expect we must continue to use that license going forward.

What do folks think about me raising a PR to add the Apache license?

Build daskdev/dask locally

I've been running some experiments with the helm chart in this repo, and I'm interested in building the daskdev/dask docker image locally. However, it looks like it relies on some script, prepare.sh. Where I can find that script?

Thanks in advance!

Dashboard does not display all the workers

I received advice from the dask-gateway forum to use the dask helm chart instead.
I managed to install the helm chart on two nodes per a revised values.yaml file to have 10 dask workers total.
On the kubernetes dashboard I was able to see 10 workers installed on the two nodes, however the Dask dashboard only displays 5 workers. How do I display all the workers so I can view my resources when I perform computing?
Thanks,

Vinh

Upstream Pangeo Helm Chart here

The Pangeo community have a Helm Chart which installs both the zero2jupyterhub and dask-gateway Helm Charts as dependencies and configures them to work together.

This enables folks to set up a multi-user deployment of Jupyter and Dask for their group or organisation. It is effectively the next stage on from the single-user experience of the current chart in this repo.

There has been a discussion in that chart around consolidating it into the main Pangeo configuration to remove maintenance burden (pangeo-data/helm-chart#129).

My view is that the Pangeo chart as it stands does not have much, if any, Pangeo specific content in it. It is just a wrapper around Jupyter and Dask gateway's existing charts. Therefore I propose we upstream the chart here so that this repository contains both a single-user and multi-user chart.

Moving it here would remove the maintenance burden from the Pangeo folks and only add a small burden here as we already maintain this chart with automated PRs to update dependencies. It would also expose the chart to a wider audience.

I would be keen to get feedback from both the Dask and Pangeo communities on this proposal.

Adding volumes

Is it possible to add volumes to the workers? I need them to access files and I have been doing it using volumes with docker compose, now I am using helm and this is the only thing which is missing.

That being said, if there is another way to solve the problem:

Using dask to calculate checksums in parallel. Without using client.upload_file could I do this without a volume?

Errors in workers: ModuleNotFoundError: No module named 'joblib'

I use the following config.yaml to setup my cluster on GKE, but the workers encounter the error: ModuleNotFoundError: No module named 'joblib'. I used both the EXTRA_CONDA_PACKAGES and EXTRA_PIP_PACKAGES to install joblib, but it seems not installed.

worker:
  replicas: 4
  resources:
    limits:
      cpu: 2
      memory: 7.5G
    requests:
      cpu: 2
      memory: 7.5G
  env:
    - name: EXTRA_CONDA_PACKAGES
      value:  joblib numba xarray -c conda-forge
    - name: EXTRA_PIP_PACKAGES
      value: gcsfs dask-ml numpy pandas sklearn joblib --upgrade

jupyter:
  enabled: true
  env:
    - name: EXTRA_CONDA_PACKAGES
      value:  joblib numba xarray matplotlib -c conda-forge
    - name: EXTRA_PIP_PACKAGES
      value: gcsfs dask-ml numpy pandas sklearn joblib --upgrade

My code for executing with dask:

from dask.distributed import Client
import joblib

from sklearn.datasets import load_digits
from sklearn.manifold import MDS

# client = Client(processes=False)
client = Client('scheduler-address') 

X, _ = load_digits(return_X_y=True)

with joblib.parallel_backend('dask'):
  embedding = MDS(n_components=2, n_jobs=-1)
  X_transformed = embedding.fit_transform(X[:300])

The error I got from the workers:

Traceback (most recent call last):
  File "python/test-dask.py", line 14, in <module>
    X_transformed = embedding.fit_transform(X[:300])
  File "/home/kelvin/venv/p36/lib/python3.6/site-packages/sklearn/manifold/mds.py", line 439, in fit_transform
    return_n_iter=True)
  File "/home/kelvin/venv/p36/lib/python3.6/site-packages/sklearn/manifold/mds.py", line 266, in smacof
    for seed in seeds)
  File "/home/kelvin/venv/p36/lib/python3.6/site-packages/joblib/parallel.py", line 1016, in __call__
    self.retrieve()
  File "/home/kelvin/venv/p36/lib/python3.6/site-packages/joblib/parallel.py", line 910, in retrieve
    self._output.extend(job.get())
  File "/home/kelvin/venv/p36/lib/python3.6/site-packages/joblib/_dask.py", line 270, in get
    return ref().result()
  File "/home/kelvin/venv/p36/lib/python3.6/site-packages/distributed/client.py", line 218, in result
    raise exc.with_traceback(tb)
  File "/opt/conda/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 59, in loads
    # The protocol we write by default.  May be less than HIGHEST_PROTOCOL.
ModuleNotFoundError: No module named 'joblib'

Exposed load balancer port

Is the Dask task schduler service exposed as a public IP / port by using a Kubernetes load balancer service? Possible security implications of this on (say cloud) deployments needs investigation.

Please expose the --no-nanny and --nprocs of dask-worker

Hi dask team,
Could you expose the --no-nanny and --nprocs options of dask-worker ?

Thanks a lot
Matthieu

Adding TLS/SSL to Dask Scheduler, Workers, and Jupyter

What happened:
I am trying to add TLS/SSL Security to my Dask Cluster hosted on Google Kubernetes Engine, installed using helm. I added the fields to the config yaml file and then applied the changes to the Dask Cluster, but I'm still able to freely access my Dask cluster without having to add a security object.

I'm trying to accomplish this in the documentation [1]

I also reviewed this similar issue [2]

[1] https://distributed.dask.org/en/latest/tls.html
[2] dask/dask#5079

While it is explained how to access a secured Dask cluster, there isn't much comprehensive information in the docs on how to add a security layer

What you expected to happen:
I was expecting to be prohibited from accessing the Dask cluster without invoking a security object, and then to be able to access the cluster when providing a security object

Minimal Complete Verifiable Example:

helm repo add dask https://helm.dask.org
Access k8s cluster
gcloud container clusters get-credentials ${RELEASE} --region ${REGION}
Install Dask workers
helm install ${RELEASE} dask/dask --set scheduler.serviceType=LoadBalancer
Create Security Files
openssl req -newkey rsa:4096 -nodes -sha256 -x509 -days 3650 -nodes -out myca.pem -keyout mykey.pem
Yaml File w/ Config Changes:

distributed:
  comm:
    default-scheme: tls
    require-encryption: True
    tls:
      ca-file: myca.pem
      scheduler:
        key: mykey.pem
        cert: myca.pem
      worker:
        key: mykey.pem
        cert: myca.pem
      client:
        key: mykey.pem
        cert: myca.pem

worker:
  name: worker
  allowed-failures: 5
  replicas: 10
  resources:
    limits:
      cpu: 2
      memory: 10G
    requests:
      cpu: 2
      memory: 10G
  env:
    - name: EXTRA_CONDA_PACKAGES
      value: numba xarray joblib -c conda-forge
    - name: EXTRA_PIP_PACKAGES
      value: >
        s3fs dask-ml pandas lz4 numpy dask distributed gcsfs SQLAlchemy google-cloud-storage pyarrow mlflow keras
        tensorflow prefect aiofile aiohttp asyncio gcloud-aio-storage pyspark --upgrade

# We want to keep the same packages on the worker and jupyter environments
jupyter:
  enabled: true
  env:
    - name: EXTRA_CONDA_PACKAGES
      value: numba xarray matplotlib joblib -c conda-forge
    - name: EXTRA_PIP_PACKAGES
      value: >
        s3fs dask-ml pandas lz4 numpy dask distributed gcsfs SQLAlchemy google-cloud-storage pyarrow mlflow keras
        tensorflow prefect aiofile aiohttp asyncio gcloud-aio-storage pyspark --upgrade

scheduler:
  allowed-failures: 5
  env:
    - name: EXTRA_PIP_PACKAGES
      value: >
        s3fs dask-ml pandas lz4 numpy dask distributed gcsfs SQLAlchemy google-cloud-storage pyarrow mlflow keras
        tensorflow prefect aiofile aiohttp asyncio gcloud-aio-storage pyspark --upgrade

Apply Config Changes
helm upgrade ${RELEASE} dask/dask -f dask_config.yaml --set scheduler.serviceType=LoadBalancer --version 4.1.0

Anything else we need to know?:
N/A

Environment:

Dask version: 2.21
Python version: 3.7.4
Operating System: macOSX
Install method (conda, pip, source): pip

Allow Jupyter pod to scale workers

Currently the Jupyter Notebook instance which is deployed as part of the helm chart can connect to the Dask cluster in order to use it, but it has no permissions on the Kubernetes cluster it is deployed on.

In dask/dask-kubernetes#255 we are adding a HelmCluster cluster manager which allows users to access logs and scale the workers deployed via the Helm Chart.

In order to use the cluster manager, you must have permissions to modify Deployment resources and access Pod log resources. It is assumed that those able to install the helm chart will have those permissions.

However if you deploy the helm chart and then make use of the provided Jupyter session then your credentials will not be available without some manual copying. Therefore the HelmCluster manager requires some additional configuration steps before it can work.

It could be helpful to include an optional service account in the helm chart which is made available to the Jupyter pod which has permission to scale the deployment and access logs.

cc @gforsyth @raybellwaves

Suggestion: "chart: dask" and "chart: daskhub" label for issues/PRs

Since there are two helm charts and one repo with issues associated with one or the either, it would be nice to have a label to use in issues / prs to separate these from each other.

When I visit this repo to consider existing issues etc for one of the helm charts, it would help me a bit.

Automate new versions

Currently we have docs on releasing the chart, and a bash script to simplify the process.

When a new docker image is created for Dask we get a Pull Request into this repo to update the chart. When we merge this we still have to manually update the Chart.yaml file with a new version and push a git tag. Travis will then handle releasing that tag.

To improve this we could also automate bumping the chart version in Chart.yaml, perhaps with another pull request. And then have an action which checks the chart vesion against the git tags and if the tag doesn't exist it creates it, triggering the rest of the release workflow.

Steps implement:

Add GitHub Actions workflow to bump the Chart patch version and raise a PR when the chart is modified.
Add GitHub Actions workflow to create and push a git tag when the version in the Chart.yaml is updated.

After that the release process would look like:

🤖 GHA automatically opens PR when Docker image is updated
👩‍💻 Maintainer merges PR
🤖 GHA automatically opens PR to bump chart patch version
👩‍💻 Maintainer merges PR
🤖 GHA creates git tag corresponding to new version
🤖 Travis CI builds new tag and releases to gh-pages branch

link to values.yaml not working on the front page README

When you scroll down to the README and click on any of the values.yaml files it brings up https://github.com/dask/helm-chart/blob/master/values.yaml. It works when clicking the links on https://github.com/dask/helm-chart/tree/master/dask

JupyterHub Helm chart dependency bump to 0.10.0-beta.1 or 0.10.0 in not too long

JupyterHub Helm chart 0.10.0-beta.1 is now released, with a Changelog. I think bumping to that will help @tjcrone for example in jupyterhub/kubespawner#282 (comment) by getting a more recent version of KubeSpawner.

Feature request: Allow separately specifying --nthreads and resource limits

I'm using Dask to coordinate some ML jobs, which have internal parallelism. I'd like to restrict my cluster so that only one job can run on a node at a time, even though nodes have multiple CPUs. However, this is currently impossible, since the Kubernetes worker resource limit is used to customize --nthreads:
https://github.com/dask/helm-chart/blob/master/dask/templates/dask-worker-deployment.yaml#L37-L38

This also prevents giving nodes e.g. 0.5 or 1.5 cpus, because even though Kubernetes allows it Dask will get confused and crash (i.e. --nthreads fails to parse as an integer.)

This is moderately esoteric so if I should just fork the chart and use that I don't mind.

Link this repo on the website?

The "source" link on the website goes to dask/dask, not this repo.

Clarify updated location of helm chart in repo description

It took me a bit to find that the helm chart is at https://github.com/helm/charts/tree/master/stable/dask. Can that be linked to in the description?

Setup ingress for dask WebUI

I am deploying this on Jetstream, I would like to expose the dask WebUI via Ingress.

In the same Kubernetes deployment I also have JupyterHub configured with zero-to-jupyterhub (with TLS provided by kube-lego.

In ingress, if I just replace proxy-public from jupyterhub with dask-scheduler, it works fine and I can browse the webUI, so the pod and the LoadBalancer work fine:

spec:
  rules:
  - host: js-xxx-xxx.jetstream-cloud.org
    http:
      paths:
      - backend:
          serviceName: dask-scheduler
          servicePort: 80
        path: /
  tls:
  - hosts:
    - js-xxx-xxx.jetstream-cloud.org
    secretName: kubelego-tls-jupyterhub

I would like however to have Jupyterhub on / and dask on /dask, I have tried with no success several variations of this, I always get 404 on /dask:

spec:
  rules:
  - host: js-xxx-xxx.jetstream-cloud.org
    http:
      paths:
      - backend:
          serviceName: proxy-public
          servicePort: 80
        path: /
      - backend:
          serviceName: dask-scheduler
          servicePort: 80
        path: /dask
  tls:
  - hosts:
    - js-xxx-xxx.jetstream-cloud.org
    secretName: kubelego-tls-jupyterhub

Would you have any suggestion? or do you know where I can ask for help?

Client can't connect to scheduler in minikube

I'm running dask in minikube using a helm release. The only change that I made to the default configuration was to set the serviceType on the scheduler to NodePort, since I believe that minikube doesn't support LoadBalancer services (and no external IPs are published). This is what my helm status output looks like:

RESOURCES:
==> v1/Pod(related)
NAME                                      READY  STATUS   RESTARTS  AGE
solemn-garfish-scheduler-fff8d4cfc-gqrbk  1/1    Running  0         39m
solemn-garfish-worker-86d9b6765d-5drwk    1/1    Running  0         39m
solemn-garfish-worker-86d9b6765d-9xnr2    1/1    Running  0         39m
solemn-garfish-worker-86d9b6765d-znj9c    1/1    Running  0         39m

==> v1/ConfigMap
NAME                           DATA  AGE
solemn-garfish-jupyter-config  1     39m

==> v1/Service
NAME                      TYPE          CLUSTER-IP    EXTERNAL-IP  PORT(S)                      AGE
solemn-garfish-jupyter    LoadBalancer  10.110.50.76  <pending>    80:32229/TCP                 39m
solemn-garfish-scheduler  NodePort      10.98.5.177   <none>       8786:32595/TCP,80:30254/TCP  39m

==> v1beta1/Deployment
NAME                      DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
solemn-garfish-jupyter    0        0        0           0          39m
solemn-garfish-scheduler  1        1        1           1          39m
solemn-garfish-worker     3        3        3           3          39m```

My minikube VM is running at http://192.168.99.101/

When I connect to http://192.168.99.101:30254/ via my browser, the bokeh dashboard appears. However, when I attempt to connect to the scheduler via the python client, I get the following...

In [1]: from dask.distributed import Client

In [2]: c = Client("tcp://192.168.99.101:32595")
distributed.utils - ERROR - in <closed TCP>: Stream is closed: while trying to call remote method 'identity'
Traceback (most recent call last):
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/distributed/utils.py", line 238, in f
    result[0] = yield make_coro()
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/distributed/client.py", line 741, in _start
    yield self._ensure_connected(timeout=timeout)
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/distributed/client.py", line 779, in _ensure_connected
    yield self._update_scheduler_info()
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/distributed/client.py", line 804, in _update_scheduler_info
    self._scheduler_identity = yield self.scheduler.identity()
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/distributed/core.py", line 467, in send_recv_from_rpc
    % (e, key,))
CommClosedError: in <closed TCP>: Stream is closed: while trying to call remote method 'identity'
---------------------------------------------------------------------------
CommClosedError                           Traceback (most recent call last)
<ipython-input-2-5c2e8e76598c> in <module>()
----> 1 c = Client("tcp://192.168.99.101:32595")

/Users/steven/predata/venv/lib/python2.7/site-packages/distributed/client.pyc in __init__(self, address, loop, timeout, set_as_default, scheduler_file, security, asynchronous, name, **kwargs)
    553                                      io_loop=self.loop)
    554
--> 555         self.start(timeout=timeout)
    556
    557         from distributed.recreate_exceptions import ReplayExceptionClient

/Users/steven/predata/venv/lib/python2.7/site-packages/distributed/client.pyc in start(self, **kwargs)
    668             self._started = self._start(**kwargs)
    669         else:
--> 670             sync(self.loop, self._start, **kwargs)
    671
    672     def __await__(self):

/Users/steven/predata/venv/lib/python2.7/site-packages/distributed/utils.pyc in sync(loop, func, *args, **kwargs)
    252             e.wait(1000000)
    253     if error[0]:
--> 254         six.reraise(*error[0])
    255     else:
    256         return result[0]

/Users/steven/predata/venv/lib/python2.7/site-packages/distributed/utils.pyc in f()
    236             yield gen.moment
    237             thread_state.asynchronous = True
--> 238             result[0] = yield make_coro()
    239         except Exception as exc:
    240             logger.exception(exc)

/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/Users/steven/predata/venv/lib/python2.7/site-packages/distributed/client.pyc in _start(self, timeout, **kwargs)
    739         self.scheduler_comm = None
    740
--> 741         yield self._ensure_connected(timeout=timeout)
    742
    743         for pc in self._periodic_callbacks:

/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/Users/steven/predata/venv/lib/python2.7/site-packages/distributed/client.pyc in _ensure_connected(self, timeout)
    777             comm = yield connect(self.scheduler.address, timeout=timeout,
    778                                  connection_args=self.connection_args)
--> 779             yield self._update_scheduler_info()
    780             yield comm.write({'op': 'register-client',
    781                               'client': self.id,

/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/Users/steven/predata/venv/lib/python2.7/site-packages/distributed/client.pyc in _update_scheduler_info(self)
    802     @gen.coroutine
    803     def _update_scheduler_info(self):
--> 804         self._scheduler_identity = yield self.scheduler.identity()
    805
    806     def __enter__(self):

/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/Users/steven/predata/venv/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/Users/steven/predata/venv/lib/python2.7/site-packages/distributed/core.pyc in send_recv_from_rpc(**kwargs)
    465             except (RPCClosed, CommClosedError) as e:
    466                 raise e.__class__("%s: while trying to call remote method %r"
--> 467                                   % (e, key,))
    468
    469             self.comms[comm] = True  # mark as open

CommClosedError: in <closed TCP>: Stream is closed: while trying to call remote method 'identity'

I'm completely stumped. Has anyone experienced something like this before?

The repo I'm using is dask https://dask.github.io/helm-chart and I just ran helm repo update

Thanks!!!

values.yaml to install packages to run all the notebook examples

I would like use a values.yaml that has all the packages needed to run the examples in https://github.com/dask/dask-docker/tree/master/notebook/examples

Perhaps it is suited for an examples page like here: https://jobqueue.dask.org/en/latest/examples.html

The example documentation on Matching the User Environment on this article is not right

What happened:
The example documentation on Matching the User Environment on this article is not right
https://docs.dask.org/en/latest/setup/kubernetes-helm.html#matching-the-user-environment

I have deployed twice and generates this error:
ValueError: Unknown fields ['image']

What you expected to happen:
The whole extraConfig: optionHandler: | should go inside a parent level called gateway like it correctly shows on the example prior to this on file: secrets.yaml

add gcsfs in EXTRA_PIP_PACKAGES

It is work adding gcsfs to

https://github.com/dask/helm-chart/blob/master/dask/values.yaml#L62
https://github.com/dask/helm-chart/blob/master/dask/values.yaml#L97
https://github.com/dask/helm-chart#customizing-python-environment

I am suggesting this as the one of the examples shipped in the docker uses gcsfs
https://github.com/dask/dask-docker/blob/master/notebook/examples/06-nyc-parquet.ipynb

Pods won't start from helm chart - Permission denied errors

What happened:

I installed the helm chart into a kubernetes cluster (Red Hat Openshift 4.4), but neither the jupyter pod nor the worker pod are starting successfully.

% oc version
Client Version: 4.3.3
Server Version: 4.4.8
Kubernetes Version: v1.17.1+3f6f40d

% helm repo add dask https://helm.dask.org/
% helm repo update
% helm install dask --set worker.replicas=1 dask/dask

% helm list
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
dask    elo             1               2020-07-14 15:17:42.137373 -0500 CDT    deployed        dask-4.1.12     2.20.0

% oc get pods
NAME                              READY   STATUS             RESTARTS   AGE
dask-jupyter-7bb66c97db-cgdjh     0/1     CrashLoopBackOff   3          97s
dask-scheduler-78494dcfc8-bpxxf   1/1     Running            0          97s
dask-worker-584f6fc66c-8kh7n      0/1     CrashLoopBackOff   3          97s

Here's the logs of the jupyter pod:

% oc logs dask-jupyter-7bb66c97db-cgdjh
+ '[' '' ']'
no environment.yml
+ '[' -e /opt/app/environment.yml ']'
+ echo 'no environment.yml'
+ '[' '' ']'
+ '[' '' ']'
+ exec start.sh jupyter lab
Adding passwd file entry for 1000580000
Container must be run with group "users" to update files
Executing the command: jupyter lab
/opt/conda/lib/python3.8/site-packages/traitlets/config/loader.py:795: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if len(key) is 1:
/opt/conda/lib/python3.8/site-packages/traitlets/config/loader.py:804: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if len(key) is 1:
Fail to get yarn configuration. {"type":"error","data":"Could not write file \"/opt/conda/lib/python3.8/site-packages/jupyterlab/yarn-error.log\": \"EACCES: permission denied, open '/opt/conda/lib/python3.8/site-packages/jupyterlab/yarn-error.log'\""}
{"type":"error","data":"An unexpected error occurred: \"EACCES: permission denied, scandir '/home/jovyan/.config/yarn/link'\"."}
{"type":"info","data":"Visit https://yarnpkg.com/en/docs/cli/config for documentation about this command."}

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/traitlets/traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'runtime_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyter-lab", line 10, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.8/site-packages/jupyter_core/application.py", line 270, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/traitlets/config/application.py", line 663, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "/opt/conda/lib/python3.8/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/notebook/notebookapp.py", line 1766, in initialize
    self.init_configurables()
  File "/opt/conda/lib/python3.8/site-packages/notebook/notebookapp.py", line 1380, in init_configurables
    connection_dir=self.runtime_dir,
  File "/opt/conda/lib/python3.8/site-packages/traitlets/traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "/opt/conda/lib/python3.8/site-packages/traitlets/traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "/opt/conda/lib/python3.8/site-packages/jupyter_core/application.py", line 100, in _runtime_dir_default
    ensure_dir_exists(rd, mode=0o700)
  File "/opt/conda/lib/python3.8/site-packages/jupyter_core/utils/__init__.py", line 13, in ensure_dir_exists
    os.makedirs(path, mode=mode)
  File "/opt/conda/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/opt/conda/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/opt/conda/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/opt/conda/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.local'

And here's the log of the worker pod:

% oc logs dask-worker-584f6fc66c-8kh7n
no environment.yml
+ '[' '' ']'
+ '[' -e /opt/app/environment.yml ']'
+ echo 'no environment.yml'
+ '[' '' ']'
+ '[' '' ']'
+ exec dask-worker dask-scheduler:8786 --no-dashboard
distributed.nanny - INFO -         Start Nanny at: 'tcp://10.128.0.84:44265'
Process Dask Worker process (from Nanny):
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.8/site-packages/distributed/process.py", line 191, in _run
    target(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/distributed/nanny.py", line 728, in _run
    worker = Worker(**worker_kwargs)
  File "/opt/conda/lib/python3.8/site-packages/distributed/worker.py", line 489, in __init__
    os.makedirs(local_directory)
  File "/opt/conda/lib/python3.8/os.py", line 221, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/dask-worker-space'
distributed.nanny - INFO - Worker process 14 exited with status 1
distributed.nanny - INFO - Closing Nanny at 'tcp://10.128.0.84:44265'
distributed.dask_worker - INFO - End worker

What you expected to happen:
Dask pods to start succcessfully.

Make service type ClusterIP and port forwarding the default

Hello all,

why is the default service type not ClusterIP? This has the advantage over LoadBalancer to not expose the dask cluster to the open internet (with a default passwort).

Then you can connect to the Jupyter Lab and the dashboard by port forwarding:
kubectl port-forward --namespace default svc/dask-jupyter 8011:80
kubectl port-forward --namespace default svc/dask-scheduler 8012:80

If this is a good idea, I would create a pull request :)

Greetings

Add SecurityContext to the scheduler deployment

Currently, the jupyter and worker deployments both have a securityContext section in the spec. To be consistent, it would be nice to have the same section in the scheduler as well.

Usage of github-activity - a tool to generate changelogs

I have come to appreciate github-activity a lot, it is automatically able to summarize changes since a point in time or since a git reference like a tag. The summary also includes a list of contributors and such.

For an example and some more notes about it, see jupyterhub/jupyterhub#3192.

It may need some updates to manage a situation where a repo contain two separate Helm charts like this, but I think it could be a sensible feature - for example filtering on issues/prs perhaps.

Revive Dask Helm ?

@beberg and I were chatting about reviving the helm-chat repo here rather than rely on the official helm repo. Currently there is a lingering PR to update the dask helm chart but it's not getting much attention from helm devs to merge in : helm/charts#12552

It's also worth noting that Jupyterhub maintains it's own helm chart:
https://github.com/jupyterhub/zero-to-jupyterhub-k8s

cc @jacobtomlinson @mrocklin

Add option for volume mounts

It would be great to be able to configure volume mounts on the notebook and workers. This way if they need to access some network resource or other Kubernetes volume type they can.

Inspired by #75

Announcement: Now on Helm Hub

The chart has now been registered on Helm Hub in helm/hub#171.

https://hub.helm.sh/charts/dask/dask

Migrate CI to GitHub Actions

Due to changes in the Travis CI billing, the Dask org is migrating CI to GitHub Actions.

This repo contains a .travis.yml file which needs to be replaced with an equivalent .github/workflows/ci.yml file.

See dask/community#107 for more details.

Kernel not starting on EKS

Hello,

I deployed this EKS cluster with eksctl:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: test-cluster-01
  region: eu-central-1
  version: "1.14"

availabilityZones: ["eu-central-1a", "eu-central-1b","eu-central-1c"]

nodeGroups:
  - name: test-nodes
    instanceType: c5.large
    volumeSize: 30
    desiredCapacity: 2
    privateNetworking: true
    maxPodsPerNode: 10
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy

and then I installed dask via helm on the cluster. I've only changed the the passwort in the configuration file.

CHART: dask-4.1.6
APP VERSION: 2.11.0

I can connect to the jupyter lab, create a notebook, save it, but no kernel is able to start (no execution or syntax highlighting).

I'd be glad for any help :)

Usage of GitHub discussions for Q&A and such

I suggest to request the beta of GitHub discussions to ask that all questions and answers etc go there by default rather than GitHub issues, which I suggest would be focused to confirmed bugs and accepted feature proposals.

Adding GitHub discussions and using it alongside GitHub Issues in this repo can reasonably be considered a premature optimization, but I frame it as a experience worth piloting for maintainer sustainability in this and other projects.

So, what is GitHub discussions? It is a beta feature that GitHub projects can try opt-in into. I think a good example of how it can be used is here: https://github.com/aquasecurity/starboard/discussions

Support aws IAM role in the chart.

For access control, dask workers should be able to use IAM role instead of security credentials added as env variables like so

apiVersion: v1
kind: Pod
metadata:
  name: aws-cli
  labels:
    name: aws-cli
  annotations:
    iam.amazonaws.com/role: arn:aws:iam::ACCOUNTID:role/S3-BUCKET-Role
spec:
  containers:
  - image: fstab/aws-cli
    command:
      - "/home/aws/aws/env/bin/aws"
      - "s3"
      - "ls"
      - "some-bucket"
    name: aws-cli

Automatically bump JupyterHub's version of pangeo/base-notebook

The daskhub helm chart requires dask-gateway and jupyterhub. These are provided in the pangeo/base-notebook docker image.

Right now the version is pinned. We should either unpin it, or add a Github Action to update it based on releases of pangeo/base-notebook.

JupyterHub: Error Starting Kernel

I deployed the daskhub helm chart on my GKE cluster as follows:

helm upgrade --install \
    --namespace  ${NAMESPACE} \
    --values config.yaml \
    --values=secrets.yaml \
    ${RELEASE} \
    dask/daskhub

I got the following error:

After logging into Jupyter Hub I'm experiencing trouble connecting to the kernel. This is only an issue when I'm on my company VPN, but is a non-issue when I'm not on my company VPN.

Here's the error message from the browser's developer console:

Some help would be greatly appreciated.

dask worker / scheduler preload script

There is no way to include preload path / script to the existing chart as an option or through values.yaml. If I needed to add custom scripts or path to my workers to preload, there should be an option. Is this something that is requested or is there a work around?

Pin docker container versions

This came up in the stable issue tracker

Unable to install dask chart

Hi I keep getting the following error trying to install helm chart. I had a script and it was working fine until yesterday in GCP. it fails when I enter the following command:

helm install --name my-release dask/dask

Error: validation failed: [configmaps "my-release-dask-jupyter-config" not found, services "my-release-dask-jupyter" not found, services "my-release-dask-scheduler" not found, deployments.apps "my-release-dask-jupyter" not found, deployments.apps "my-release-dask-scheduler" not found, deployments.apps "my-release-dask-worker" not found]

error on helm upgrade

Running helm upgrade {deployment_name} dask/dask -f worker-config.yml returns following error

Error: UPGRADE FAILED: render error in "dask/templates/dask-worker-deployment.yaml": template: dask/templates/dask-worker-deployment.yaml:36:58: executing "dask/templates/dask-worker-deployment.yaml" at <.Values.workers.defa...>: can't evaluate field default_resources in type interface {}

This looks like its pointing to line 33 in the values.yml file.

my helm version is 2.8.1. I am not sure if that matters.

Perhaps related to this

Dask Cuda Worker

I have two machines with NVIDIA GPUS and a iMAC in a kubernetes cluster. I'm trying to spread out the workers so I can have a mix of dask cuda workers and dask workers, but I'm not getting the dask cuda workers on any of the nodes.
I modified the values.yaml file as follows to get a dask cuda worker

worker:
  name: worker
  image:
    repository: "daskdev/dask"
    tag: 2.19.0
    pullPolicy: IfNotPresent
    dask_worker: "dask-cuda-worker"
    dask_worker: "dask-worker"
    pullSecrets:
    #  - name: regcred
  replicas: 10
  default_resources:  # overwritten by resource limits if they exist
    cpu: 1
    memory: "4GiB"
  env:
  #  - name: EXTRA_APT_PACKAGES
  #    value: build-essential openssl
  #  - name: EXTRA_CONDA_PACKAGES
  #    value: numba xarray -c conda-forge
  #  - name: EXTRA_PIP_PACKAGES
  #    value: s3fs dask-ml --upgrade
  resources: {}
  #  limits:
  #    cpu: 1
  #    memory: 3G
  #    nvidia.com/gpu: 1
  #  requests:
  #    cpu: 1
  #    memory: 3G
  #    nvidia.com/gpu: 1
  tolerations: []
  nodeSelector: {}
  affinity: {}
  securityContext: {}
  # serviceAccountName: ""
  # port: ""
  #  this option overrides "--nthreads" on workers, which defaults to resources.limits.cpu / default_resources.limits.cpu
  #  use it if you need to limit the amount of threads used by multicore workers, or to make workers with non-whole-number cpu limits
  # threads_per_worker: 1

Add default subchart versions to daskhub readme / Are subchart versions configurable?

daskhub wraps jupyterhub and dask-gateway helm chart, which are versioned:

helm-chart/daskhub/Chart.yaml

Lines 7 to 15 in c1b7f9d

 dependencies: 

 - name: jupyterhub 

 version: "0.9.1" 

 repository: 'https://jupyterhub.github.io/helm-chart/' 

 import-values: 

 - child: rbac 

 parent: rbac 

 - name: dask-gateway 

 version: "0.8.0"

Is it possible to do a helm install that overrides the default version (for example jupyterhub 0.9.0)?

Automated version bumps were previously discussed over in pangeo-data/helm-chart#124 but never fully implemented

Error: No such file or directory as /home/joyvan/<filename.csv

I'm using the jupyterlab provided by the dask Helm chart to experiment with dataframes.
I uploaded a spreadsheet I downloaded from Kaggle (NYC Parking Tickets) and renamed it Parking_2017_csv.

When I do the following everything is fine:

import dask.dataframe as dd
from dask.diagnostics import ProgressBar 
from matplotlib import pyplot as plt     
df = dd.read_csv('Parking_2017.csv')    
df

However, when I perform the following, I get an error that there is no such file as /home/joyvan/Parking_2017.csv even though the file is in the directory.

missing_values = df.isnull().sum()
missing_count = ((missing_values / df.index.size) * 100)
with ProgressBar():   
        missing_count_pct = missing_count.compute()
missing_count_pct

Also, is there a way I can use Jupyter Notebook instead of Jupyter Lab in the helm chart?

Can values.yaml be used to put privately-hosted pip packages on Dask workers?

I would like to deploy a dask-kubernetes helm chart that uses a pip package from a private pypi repo. Is this possible?

My current approach: To deploy the chart I use helm install mydask dask/dask -f values.yaml, and the relevant section of my yaml file looks like this:

worker:
  env:
    - name: EXTRA_PIP_PACKAGES
      value: s3fs dask-ml myprivatepkg --extra-index-url=https://pypi.mydomain.de/token

I am able to deploy the helm chart, connect to the cluster, and successfully perform computations. To obtain a list of packages installed on the cluster, I use a custom function whose output is similar to that of 'pip list', but as a python function that can be submitted to the cluster:

from distributed import Client
client = Client("<cluster_scheduler_ip>")
# Custom function for obtaining a list of installed packages
def list_local_packages():
    import pkg_resources
    installed_packages = pkg_resources.working_set
    installed_packages_list = sorted(["%s==%s" % (i.key, i.version) for i in installed_packages])
    return installed_packages_list
# Submit custom function to remote scheduler
client.submit(list_local_packages).result()

From this I obtain a list where I can see that s3fs and dask-ml were indeed installed, but not myprivatepkg.

If I deploy dask without the yaml, then neither s3fs nor dask-ml get installed either, so clearly the EXTRA_PIP_PACKAGES section is doing something. But it seems it cannot handle the --extra-index-url option, even though it can handle other options like --upgrade just fine.

If this feature is not implemented, is there any other approach to put privately-hosted pip packages onto the workers of a helm-deployed dask-kubernetes instance?

	dependencies:
	- name: jupyterhub
	version: "0.9.1"
	repository: 'https://jupyterhub.github.io/helm-chart/'
	import-values:
	- child: rbac
	parent: rbac
	- name: dask-gateway
	version: "0.8.0"