Comments (3)
I would like to deploy a dask-kubernetes helm chart
Just a quick bit of clarification before we get into discussing your issue. The Dask Helm Chart and dask-kubernetes are two separate projects with different purposes.
The helm chart allows you to install a jupyter server and long running Dask cluster on kubernetes. The dask-kubernetes project allows you to create ephemeral ad-hoc Dask clusters from within your Python session.
Sometimes this can be a bit confusing and folks end up on the wrong set of documentation. Just wanted to give you a heads up. This project is the Dask Helm chart and has no relation to dask-kubernetes.
My current approach: To deploy the chart I use helm install mydask dask/dask -f values.yaml, and the relevant section of my yaml file looks like this:
This is the correct approach to specify extra pip packages. The Dask helm chart uses the Dask Docker image daskdev/dask
by default. This is the container image that your environment variables are being passed to. If you check out the prepare.sh
script you can see where this environment variable is being handled.
As you can see we are just passing the contents of this environment variable to the pip install
command. Therefore it is up to you to pass valid options here.
I recommend that you get a shell into one of your workers and experiment with your pip install command and continue debugging there. You can do this quickly by running kubectl get pods
to list the pods and then run kubectl exec -it <pod name> bash
.
from helm-chart.
Hello Jacob,
Thanks to your suggestion to use kubectl exec -it <pod name> bash
to get a shell into one of my workers, I managed to resolve the issue.
While in the shell of a worker, I discovered that calling pip install
with the --extra-index-url
option actually worked totally fine, and I was able to successfully download myprivatepkg from my private repo. The installation, however, failed. Turns out that what was causing the installation of my pip pkg to fail was that one of myprivatepkg's dependencies required build-essential from the debian package repo.
At that point I scratched my head for a while trying to figure out the best way to install extra apt packages, but then I noticed that the prepare.sh
script (which you had linked to) had an if-statement containing the phrase EXTRA_APT_PACKAGES
, which seemed to suggest that this was something I could use in my values.yaml
file in a manner similar to that of EXTRA_PIP_PACKAGES
. So I modified my yaml file like so:
worker:
env:
- name: EXTRA_APT_PACKAGES
value: build-essential
- name: EXTRA_PIP_PACKAGES
value: s3fs dask-ml myprivatepkg --extra-index-url=https://pypi.mydomain.de/token
And that fixed my issue! Myprivatepkg now gets installed on all workers and I can use it :)
Thank you!
from helm-chart.
Great!
from helm-chart.
Related Issues (20)
- Daskhub `additional_worker_groups` configuration HOT 2
- Additional worker group quirks HOT 4
- [BUG] Using a hostname to expose Jupyter causes Helm Render to fail
- Release 2022.5.0 HOT 13
- Hourly check of upstream Docker image version failing
- Remove generated files from version control HOT 5
- Add `envFrom` in dask/dask workers deployment HOT 1
- Automatic updates have broken HOT 6
- Chart README not being packaged
- SSL error when having https enabled with manual certificates in jupyterhub side. HOT 2
- is it possible to change the scheduler.service.name ? HOT 2
- Chart is using depracted `bokeh-port` argument HOT 8
- Add RELEASE.md HOT 1
- Add Liveness and Readiness Probes to Pods
- OverrideNames failure
- DaskHub cannot be placed in a subdirectory URL
- jupyterhub helm chart update HOT 4
- client = cluster.get_client() : Timed out HOT 1
- ARM support HOT 4
- Network policies break daskhub HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from helm-chart.