Can values.yaml be used to put privately-hosted pip packages on Dask workers? about helm-chart HOT 3 CLOSED

I would like to deploy a dask-kubernetes helm chart

Just a quick bit of clarification before we get into discussing your issue. The Dask Helm Chart and dask-kubernetes are two separate projects with different purposes.

The helm chart allows you to install a jupyter server and long running Dask cluster on kubernetes. The dask-kubernetes project allows you to create ephemeral ad-hoc Dask clusters from within your Python session.

Sometimes this can be a bit confusing and folks end up on the wrong set of documentation. Just wanted to give you a heads up. This project is the Dask Helm chart and has no relation to dask-kubernetes.

My current approach: To deploy the chart I use helm install mydask dask/dask -f values.yaml, and the relevant section of my yaml file looks like this:

This is the correct approach to specify extra pip packages. The Dask helm chart uses the Dask Docker image daskdev/dask by default. This is the container image that your environment variables are being passed to. If you check out the prepare.sh script you can see where this environment variable is being handled.

As you can see we are just passing the contents of this environment variable to the pip install command. Therefore it is up to you to pass valid options here.

I recommend that you get a shell into one of your workers and experiment with your pip install command and continue debugging there. You can do this quickly by running kubectl get pods to list the pods and then run kubectl exec -it <pod name> bash.

from helm-chart.

Hello Jacob,

Thanks to your suggestion to use kubectl exec -it <pod name> bash to get a shell into one of my workers, I managed to resolve the issue.

While in the shell of a worker, I discovered that calling pip install with the --extra-index-url option actually worked totally fine, and I was able to successfully download myprivatepkg from my private repo. The installation, however, failed. Turns out that what was causing the installation of my pip pkg to fail was that one of myprivatepkg's dependencies required build-essential from the debian package repo.

At that point I scratched my head for a while trying to figure out the best way to install extra apt packages, but then I noticed that the prepare.sh script (which you had linked to) had an if-statement containing the phrase EXTRA_APT_PACKAGES, which seemed to suggest that this was something I could use in my values.yaml file in a manner similar to that of EXTRA_PIP_PACKAGES. So I modified my yaml file like so:

worker:
  env:
    - name: EXTRA_APT_PACKAGES
      value: build-essential
    - name: EXTRA_PIP_PACKAGES
      value: s3fs dask-ml myprivatepkg --extra-index-url=https://pypi.mydomain.de/token

And that fixed my issue! Myprivatepkg now gets installed on all workers and I can use it :)

Thank you!

from helm-chart.

Great!

from helm-chart.