Problem when creating cluster: "suggest closing worker" about pangeo-binder HOT 8 CLOSED

Hi @lwchao-vivian. Those are log messages that can be ignored. Typically they're hidden, but something changed in Dask recently, so they're showing up in the notebook. This is being fixed in pangeo-data/pangeo-stacks#92

from pangeo-binder.

@TomAugspurger Thank you for your response! In my case, after all of the "Suggest closing workers" messages, Dask removed the workers and I lost all of them. Do you know how to fix it?

from pangeo-binder.

Loading your binder now. It's rebuilding the image so it may be a bit.

Are you using adaptive? That could just be Dask saying that the cluster is idle, so it's releasing all the workers.

from pangeo-binder.

Yes, I am using adapt.
It would be nice if there is a way to hide these messages. Can you elaborate what your meaning of "specify logging at the top-level" in pangeo-data/pangeo-stacks#92?

from pangeo-binder.

I noticed that the problem of looping messages can be avoided by specifying the number of workers:
That is,
cluster = KubeCluster()
cluster.adapt(minimum=4, maximum=40)
should be replaced with
cluster = KubeCluster(n_workers=4)

Thank you for your assistance!

from pangeo-binder.

These are actually logged at debug level today in Dask. Pangeo should probably change their base image to point to dask >= 2.7.

from pangeo-binder.

pangeo-data/pangeo-stacks#92 is updating the dask_config.yaml that's used in the base image. On master, the logging configuration is under the "distributed" key in the YAML file, but it shouldn't be.

And just to be clear, you may want to continue using adapt. This problem will solve itself once https://github.com/pangeo-data/pangeo-stacks/pull/92/files is merged and the docker images are rebuilt (probably toay or tomrrow).

from pangeo-binder.

That's clear. Now I understand. Thank you @mrocklin and @TomAugspurger !

from pangeo-binder.