Create a personal Dask cluster for a given Conda environment
- Work in the
env-dev folder when running terraform.
- Create your conda environment. You might want do something like:
conda create -n ml python=3.5 tensorflow keras ipykernel dask distributed -y
- In Jade you can go into this new environment by doing the above and below in the terminal:
bash
source activate ml
python -m ipykernel install --user --name ml --display-name "Py 3.5 Keras"
- You will now have a "Py 3.5 Keras" kernel available.
- To use with this tool you're environment needs to have
dask and distributed included in the conda environment.
- To use as a kernel on JADE you'll need to include ipykernel in the environment.
- Once you've set up you're environment. Export it:
conda env export -n ml > env.yaml
- This exports the 'ml' environment into the env.yaml file.
- Take the content of
env.yaml and put it in env.yaml inside the env-dev folder
- Run
python config-to-b64.py env.yaml which will convert the file to base64 and store in env.yaml.b64
- You are now ready to terraform with something like the below (where conda_env is the content of
env.yaml.b64)
- You need to pass in the AWS credentials so this can be run programatically.
terraform plan -var-file=dev.tfvars -var aws_access_key=$AWS_ACCESS_KEY_ID -var aws_secret_key=$AWS_SECRET_ACCESS_KEY -var "conda_env=bmFtZTogcHkyCmNoYW5uZWxzOiAhIXB5dGhvbi90dXBsZQotIGRlZmF1bHRzCmRlcGVuZGVuY2llczoKLSBiYWNrcG9ydHM9MS4wPXB5MjdfMAotIGJhY2twb3J0c19hYmM9MC41PXB5MjdfMAotIGJvdG8zPTEuNC4zPXB5MjdfMAotIGJvdG9jb3JlPTEuNC45MD1weTI3XzAKLSBjYWlybz0xLjE0Ljg9MAotIGN1cmw9Ny40NS4wPTAKLSBjeWNsZXI9MC4xMC4wPXB5MjdfMAotIGRlY29yYXRvcj00LjAuMTE9cHkyN18wCi0gZG9jdXRpbHM9MC4xMy4xPXB5MjdfMAotIGVudW0zND0xLjEuNj1weTI3XzAKLSBmaWxlbG9jaz0yLjAuNz1weTI3XzAKLSBmb250Y29uZmlnPTIuMTIuMT0zCi0gZnJlZXR5cGU9Mi41LjU9MgotIGZ1bmNzaWdzPTEuMC4yPXB5MjdfMAotIGZ1dHVyZXM9My4wLjU9cHkyN18wCi0gZ2V0X3Rlcm1pbmFsX3NpemU9MS4wLjA9cHkyN18wCi0gZ2xpYj0yLjUwLjI9MQotIGhhcmZidXp6PTAuOS4zOT0yCi0gaGRmNT0xLjguMTUuMT0zCi0gaXB5a2VybmVsPTQuNS4yPXB5MjdfMAotIGlweXRob249NS4xLjA9cHkyN18wCi0gaXB5dGhvbl9nZW51dGlscz0wLjEuMD1weTI3XzAKLSBqYmlnPTIuMT0wCi0gam1lc3BhdGg9MC45LjA9cHkyN18wCi0ganBlZz05Yj0wCi0ganVweXRlcl9jbGllbnQ9NC40LjA9cHkyN18wCi0ganVweXRlcl9jb3JlPTQuMi4xPXB5MjdfMAotIGxpYmZmaT0zLjIuMT0xCi0gbGliZ2ZvcnRyYW49My4wLjA9MQotIGxpYmljb252PTEuMTQ9MAotIGxpYm5ldGNkZj00LjMuMy4xPTMKLSBsaWJwbmc9MS42LjI3PTAKLSBsaWJzb2RpdW09MS4wLjEwPTAKLSBsaWJ0aWZmPTQuMC42PTMKLSBsaWJ4bWwyPTIuOS40PTAKLSBsaWJ4c2x0PTEuMS4yOT0wCi0gbHhtbD0zLjcuMj1weTI3XzAKLSBtYXRwbG90bGliPTEuNS4xPW5wMTEwcHkyN18wCi0gbWtsPTExLjMuMz0wCi0gbW9jaz0yLjAuMD1weTI3XzAKLSBuZXRjZGY0PTEuMi4yPW5wMTEwcHkyN18wCi0gbm9zZT0xLjMuNz1weTI3XzEKLSBudW1weT0xLjEwLjQ9cHkyN18yCi0gb3BlbnNzbD0xLjAuMms9MAotIHBhbmdvPTEuNDAuMz0xCi0gcGF0aC5weT0xMC4wPXB5MjdfMAotIHBhdGhsaWIyPTIuMi4wPXB5MjdfMAotIHBicj0xLjEwLjA9cHkyN18wCi0gcGNyZT04LjM5PTEKLSBwZXA4PTEuNy4wPXB5MjdfMAotIHBleHBlY3Q9NC4yLjE9cHkyN18wCi0gcGlja2xlc2hhcmU9MC43LjQ9cHkyN18wCi0gcGlsbG93PTQuMC4wPXB5MjdfMAotIHBpcD05LjAuMT1weTI3XzEKLSBwaXhtYW49MC4zNC4wPTAKLSBwcm9tcHRfdG9vbGtpdD0xLjAuOT1weTI3XzAKLSBwdHlwcm9jZXNzPTAuNS4xPXB5MjdfMAotIHB5Y2Fpcm89MS4xMC4wPXB5MjdfMAotIHB5Z21lbnRzPTIuMS4zPXB5MjdfMAotIHB5cGFyc2luZz0yLjEuND1weTI3XzAKLSBweXByb2o9MS45LjUuMT1weTI3XzAKLSBweXF0PTQuMTAuND1weTI3XzAKLSBweXNpZGU9MS4yLjE9cHkyN18xCi0gcHl0aG9uPTIuNy4xMz0wCi0gcHl0aG9uLWRhdGV1dGlsPTIuNi4wPXB5MjdfMAotIHB5dHo9MjAxNi4xMD1weTI3XzAKLSBweXptcT0xNi4wLjI9cHkyN18wCi0gcXQ9NC44LjU9MAotIHJlYWRsaW5lPTYuMj0yCi0gcmVxdWVzdHM9Mi4xMi40PXB5MjdfMAotIHMzdHJhbnNmZXI9MC4xLjEwPXB5MjdfMAotIHNjYW5kaXI9MS40PXB5MjdfMAotIHNjaXB5PTAuMTcuMT1ucDExMHB5MjdfMQotIHNjaXRvb2xzOjpiaWdndXM9MC4xNC4wPXB5MjdfMAotIHNjaXRvb2xzOjpjYXJ0b3B5PTAuMTUuMD1ucDExMHB5MjdfMAotIHNjaXRvb2xzOjpjZl91bml0cz0xLjEuMz1weTI3XzAKLSBzY2l0b29sczo6Z2Vvcz0zLjQuMj00Ci0gc2NpdG9vbHM6OmltYWdlaGFzaD0zLjE9cHkyN18wCi0gc2NpdG9vbHM6OmlyaXM9MS4xMS4wPXB5MjdfMgotIHNjaXRvb2xzOjpsaWJtb191bnBhY2s9My4wPTMKLSBzY2l0b29sczo6bW9fcGFjaz0wLjIuMD1ucDExMHB5MjdfMQotIHNjaXRvb2xzOjpuY190aW1lX2F4aXM9MS4wLjA9cHkyN18wCi0gc2NpdG9vbHM6Om93c2xpYj0wLjEwLjM9cHkyN18wCi0gc2NpdG9vbHM6OnByb2ouND00LjkuMT0xCi0gc2NpdG9vbHM6OnB5ZXBzZz0wLjIuMD1weTI3XzAKLSBzY2l0b29sczo6cHlrZT0xLjEuMT1weTI3XzEKLSBzY2l0b29sczo6cHlzaHA9MS4yLjM9cHkyN18wCi0gc2NpdG9vbHM6OnB5d2F2ZWxldHM9MC40LjA9bnAxMTBweTI3XzAKLSBzY2l0b29sczo6c2hhcGVseT0xLjUuMTM9bnAxMTBweTI3XzEKLSBzY2l0b29sczo6dWR1bml0czI9Mi4yLjIwPTAKLSBzZXR1cHRvb2xzPTI3LjIuMD1weTI3XzAKLSBzaGlib2tlbj0xLjIuMT1weTI3XzAKLSBzaW1wbGVnZW5lcmljPTAuOC4xPXB5MjdfMQotIHNpbmdsZWRpc3BhdGNoPTMuNC4wLjM9cHkyN18wCi0gc2lwPTQuMTUuNT1weTI3XzAKLSBzaXg9MS4xMC4wPXB5MjdfMAotIHNxbGl0ZT0zLjEzLjA9MAotIHNzbF9tYXRjaF9ob3N0bmFtZT0zLjQuMC4yPXB5MjdfMQotIHRrPTguNS4xOD0wCi0gdG9ybmFkbz00LjQuMj1weTI3XzAKLSB0cmFpdGxldHM9NC4zLjE9cHkyN18wCi0gd2N3aWR0aD0wLjEuNz1weTI3XzAKLSB3aGVlbD0wLjI5LjA9cHkyN18wCi0geHo9NS4yLjI9MQotIHplcm9tcT00LjEuNT0wCi0gemxpYj0xLjIuOD0zCi0gcGlwOgogIC0gYmFja3BvcnRzLWFiYz09MC41CiAgLSBiYWNrcG9ydHMuc2h1dGlsLWdldC10ZXJtaW5hbC1zaXplPT0xLjAuMAogIC0gYmFja3BvcnRzLnNzbC1tYXRjaC1ob3N0bmFtZT09My40LjAuMgogIC0gYm90bz09Mi40NS4wCiAgLSBjZi11bml0cz09MS4xLjMKICAtIGlweXRob24tZ2VudXRpbHM9PTAuMS4wCiAgLSBqYWRlLXV0aWxzPT0xLjAuMAogIC0ganVweXRlci1jbGllbnQ9PTQuNC4wCiAgLSBqdXB5dGVyLWNvcmU9PTQuMi4xCiAgLSBtby1wYWNrPT0wLjIuMAogIC0gbmMtdGltZS1heGlzPT0xLjAuMAogIC0gcHJvbXB0LXRvb2xraXQ9PTEuMC45CiAgLSBzaGFycHB5PT0xLjMuMApwcmVmaXg6IC9vcHQvY29uZGEvZW52cy9weTIKCg=="
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent