Configure cluster defaults from environment variables about dask-kubernetes HOT 16 CLOSED

I propose we handle configuration redundantly with both a yaml file and a set of environment variables as follows (this is how dask does things in other projects):

1. We have a config module with a config dictionary that holds configuration

   from daskernetes import config
   config['key'] = 'value'
   

2. We look for a config file in .dask/daskernetes.yaml and set that as the base for the configuration. We might make this path configurable with a DASKERNETES_CONFIG_PATH environment variable.

3. On import we take all environment variables of the form DASKERNETES_FOO_BAR=value, remove the DASKERNETES_, lower-case and replace underscores with hyphens, resulting in

   >>> config
   {'foo-bar': 'value'}
   

I've found this approach to be useful. Operationally I find myself using the yaml file more for personal use and environment variables more for deployment. In either case it's nice for internal code to not care and just look at the centralized config dictionary.

from dask-kubernetes.

from dask-kubernetes.

Yeah, thinking about this more critically now there might not be that much. Operationally I've needed two things:

1. default worker pod spec, or perhaps a filename that holds such a spec
2. how to point users to the bokeh diagnostics dashboard (this is the one thing in the current deployment that we don't have)

Looking through our code we don't have that many global constant values, so this sort of treatment may not be necessary. Some possible examples:

1. default labels (although maybe we start leaving this to the pod spec?)

       self.pod_template.metadata.labels['dask.pydata.org/cluster-name'] = name
       self.pod_template.metadata.labels['app'] = 'dask'
       self.pod_template.metadata.labels['component'] = 'dask-worker'

2. (nope, that was all I could find)

from dask-kubernetes.

from dask-kubernetes.

There might not be one. I was looking around for hard-wired values and that is what I came up with. This lack of configuration is a decent argument to not have a serious configuration mechanism.

from dask-kubernetes.

from dask-kubernetes.

I do still need to find a solution to the following two problems:

1. Default worker configuration so that admins can create worker specs and not have users deal with this
   • Could use a magic filename
   • Could use an environment variable
2. Point users to the right location for the bokeh diagnostics server, as is proxied through the nbserverproxy

from dask-kubernetes.

from dask-kubernetes.

2. Sure, we can put other information in the ipywidget that is determined by some external mechanism. Historically the "external mechanism" here was just a config option. I'd love to see an extension as well but ...
   • I'm unlikely to do this work (unfamiliar with the Jupyter ecosystem and not too adept at JS)
   • We should probably have a crude solution even without installing the extension

from dask-kubernetes.

from dask-kubernetes.

In the case of pangeo, yes, but I'm not sure what will happen in the general case.

I'm happy to write it, shouldn't be too hard! Can you list what all it should do?

I'm not sure what's best here and would appreciate feedback on these thoughts. I'll open a new issue for discussion on this subtopic.

from dask-kubernetes.

Using an environment variable to start point to a yaml file is implemented here: #41

from dask-kubernetes.

Continuing discussion on extension here: #23 (is this what you had in mind?)

from dask-kubernetes.

What are the cases where someone would want to change the default labels?
You can always add additional labels, and getting the defaults wrong might
have bad consequences. Curious what the use case would be!

Also, it looks like this was a request from @jacobtomlinson in the original message of this issue

For example it would nice to be able to set the name of the cluster to be the JUPYTERHUB_USER.

from dask-kubernetes.

from dask-kubernetes.

This has been resolved.

from dask-kubernetes.