Dask-Marathon ==========
Deploy dask-worker processes on Marathon in response to load on a Dask scheduler. This creates a Marathon application of dask-worker processes. It watches a Dask Scheduler object in the local process and, based on current requested load, scales the Marathon application up and down.
This project is a proof of concept only. No guarantee of quality or future maintenance is made.
It's not yet clear how to expose all of the necessary options to a command line interface. For now we're doing everything manually.
Make an IOLoop running in a separate thread:
from threading import Thread
from tornado.ioloop import IOLoop
loop = IOLoop()
thread = Thread(target=loop.start); thread.daemon = True
thread.start()Start a Scheduler in that IOLoop
from distributed import Scheduler
s = Scheduler(loop=loop)
s.start()Start a dask-marathon cluster. Give it the scheduler, information about the application, and the address of the Marathon master. This example assumes that dask-worker is available in the system environment, but see see marathon.MarathonApp for possible keyword args to define the application, including docker containers, etc..
from dask_marathon import AdaptiveCluster
cluster = AdaptiveCluster(s, marathon_address='http://localhost:8080',
cpus=1, mem=1000, executable='dask-worker',
**kwargs)Create a Client and submit work to the scheduler. Marathon will scale workers up and down as neccessary in response to current workload.
from distributed import Client
c = Client(s.address)
future = c.submit(lambda x: x + 1, 10)- Deploy the scheduler on the cluster
- Support a command line interface
This sets up a docker cluster of one Mesos master and two Mesos agents using docker-compose.
Requires:
- docker version >= 1.11.1
- docker-compose version >= 1.7.1
export DOCKER_IP=127.0.0.1
docker-compose upRun py.test:
py.test dask-marathon- Master and Agents have dask.distributed installed its github repository
- Mesos container names:
- mesos_master
- mesos_slave_one
- mesos_slave_two
- http://localhost:5050/ for Mesos master UI
- http://localhost:5051/ for the first Mesos agent UI
- http://localhost:5052/ for the second Mesos agent UI
- http://localhost:8080/ for Marathon UI
- http://localhost:8888/ for Chronos UI
Mesos Docker-compose solution originally forked from https://github.com/bobrik/mesos-compose
This project was then forked from dask-mesos.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent