Update cpu and memory requests after cluster creation about dask-kubernetes HOT 15 CLOSED

We are thinking about this in dask/distributed#2118 and dask/distributed#2208 (comment).

But we didn't talk about adaptive part yet ("scale them according to what the client requests"), which would probably need modifications too.

from dask-kubernetes.

I think that this seems like a reasonable request. I think that it would require additional logic to the Adaptive class that looked at resources when making requests. It's non-trivial work, but seems reasonably in-scope.

from dask-kubernetes.

from dask-kubernetes.

I'd be happy to help here, but I don't really know how to... I've never used KubeCluster yet, but I'm planning to give it a go in the next few months. From my limited point of view, I'd like to be able to easily specify my worker size when initializing a cluster, so if this is not easily possible yet, an improvement would be welcome.

One thought: does or may this affect Cloud VM flavor also? Are these hard coded in another place when defining the Kubernetes cluster? It will be good if by specifying cpu, mem, and perhaps flavor, the automatically created VM are adapted to those.

from dask-kubernetes.

@guillaumeeb you currently specify the whole worker config when you initialise the cluster and can configure everything from resources to docker image.

The admin of the cluster can also specify a default config to use if the user doesn't specify one. This issue is discussing how to make tweaks to resources in the default config on the fly without having to specify a whole new config.

from dask-kubernetes.

@jacobtomlinson does Kubernetes chose by itself the machine type to use according to what is specified in the pod spec? Looking at http://dask-kubernetes.readthedocs.io/en/latest/#quickstart, I don't see anything like n1-standard-2 or another flavor.

Sorry if this is a dumb question, I still need to learn how Kubernetes is working.

from dask-kubernetes.

from dask-kubernetes.

So looking at core.py, currently the easiest way to do what you say is to use the following method:

pod_spec = make_pod_spec(image='daskdev/dask:latest',
                         memory_limit='4G', memory_request='4G',
                         cpu_limit=1, cpu_request=1,
                         env={'EXTRA_PIP_PACKAGES': 'fastparquet git+https://github.com/dask/distributed'})

Which implies that you understand all the rest.

Again I've only a limited view of all this, but it sounds like it would be a welcome functionality here. The way I would do that is to add your update methods or an equivalent, and call that in the __init__ with associated kwargs, just after pod_template is secured, somewhere here https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/core.py#L174, or below.

Updating after creation looks like and edge case to me, this is not what we are doing in dask-jobqueue. But I can understand in some situation you might want to do this.

from dask-kubernetes.

Yes I agree that updating it after creating is an edge case. Perhaps a better way would be to allow users to call make_pod_spec with missing kwargs that get filled in from the defaults.

E.g

pod_spec = make_pod_spec(memory_limit='4G', memory_request='4G',
                         cpu_limit=1, cpu_request=1)

In this example the image and extra packages would come from the default. There is no reason why a scientist should even know what a docker image is.

from dask-kubernetes.

Not sure it's such an edge case. It's possible to think about running a single dask scheduler for multiple tasks, each of which may have a slightly different set of arguments/variables.

from dask-kubernetes.

@mturok interesting point. However the example I put above would modify the scheduler which would not be great for multi-use clusters.

from dask-kubernetes.

As long as the pods get created with matching values, there should be no pathological cases.

from dask-kubernetes.

Quick question, that may be related to this issue.
Would it be possible to use multiple pod specs and scale them according to what the client requests?
Like the use case described here for the pure dask distributed.

client.submit(process, d, resources={'GPU': 1})

The main use case is to have some instances with GPUs and others without, and use them as needed.

I guess this would require another scheduling layer to fit the work in the smallest pool possible, plus some logic for when a pool is maxed out but the other one is not.

from dask-kubernetes.

That is an interesting idea but would involve upstream changes in dask and distributed. For now I would just create multiple clusters.

@mrocklin I'm sure you are very busy at the moment do you have any thoughts on this?

from dask-kubernetes.

As work is ongoing in the issues that @guillaumeeb mentioned and much of the scaling logic in dask-kubernetes has been replaced with SpecCluster from distributed I'm going to close this in favour of dask/distributed#2118 in particular.

from dask-kubernetes.