Pre-installed Python Option about knit HOT 2 CLOSED

Certainly I think this is useful.
We need to tell the scala code NOT to upload anything (there is already a path for this) and to pass the python executable path through rather than constructing one.

from knit.

Skein (https://jcrist.github.io/skein/) allows for a more flexible way of describing applications to run on YARN. Dask-Yarn has been rewritten to use skein (http://dask-yarn.readthedocs.io/), and can accept a generic skein specification to start dask. The above can be achieved by:

# spec.yaml
name: dask
queue: default
max_attempts: 1

services:
    dask.scheduler:
        resources:
            memory: 512
            vcores: 1
        commands:
            - /path/to/dask-yarn-scheduler

    dask.worker:
        instances: 0
        max_restarts: 2
        resources:
            memory: 512
            vcores: 1
        depends:
            - dask.scheduler
        commands:
            - /path/to/dask-yarn-worker 1 --memory_limit 512MB

from dask_yarn import YarnCluster

cluster = YarnCluster.from_specification('spec.yaml')
cluster.scale(10)

Closing, development has moved to https://github.com/dask/dask-yarn.

from knit.