No information on running tasks in Bokeh in EMR about knit HOT 6 CLOSED

Do tasks in fact run? Can you confirm that you don't have a previous scheduler process hanging around?

In the cluster-runnig process, you can investigate what port the scheduler has actually used for bokeh,

cluster.local_cluster.scheduler.services['bokeh'].port

or from the other script with the client

client.scheduler_info()['services']['bokeh']

from knit.

They do run! I have to wrap it up for the day, but my application writes JSON to s3, and it finished running and wrote JSON. I'll check the bokeh address next time I run things.

from knit.

from knit.

Looks like this problem is not related to YARN scheduler. The same thing happens when running dask-scheduler and dask-worker processes from console on a EMR cluster. The scheduler output indicates bokeh UI on port 8787:

(dvss) [hadoop@ip-10-21-0-229 ~]$ dask-scheduler
distributed.scheduler - INFO - -----------------------------------------------
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO -   Scheduler at:    tcp://10.21.0.229:8786
distributed.scheduler - INFO -       bokeh at:                     :8787
distributed.scheduler - INFO - Local Directory:    /tmp/scheduler-ytyz4of_
distributed.scheduler - INFO - -----------------------------------------------

UI is accessible, I see workers etc., but no tasks.

from knit.

Gah. User error! I failed to provide the running scheduler address to my code, and it quietly created a new LocalCluster as a fallback. So I was looking at the wrong UI. Smart me. Closing.

from knit.

from knit.