Comments (58)
/cc @windreamer
from ecosystem.
Hmm, it is a wild idea... maybe the most difficult part is how to map GPUs into different tasks?
from ecosystem.
/cc @yuefengz
from ecosystem.
Ping
from ecosystem.
It seems docker is not a must. it seems we can just using GPU allocator of Mesos to allocate resources. But experiments are still on-going.
from ecosystem.
Sorry, was just back from vacation last week and didn't see your issue. Will take a look at your issue shortly.
from ecosystem.
@yuefengz Thank you. @windreamer has already experimented the optional docker dependency in tfmesos.
from ecosystem.
Obviously I think that TPU resources are available only in managed google cloud and not in DC/OS on google compute engine right?
from ecosystem.
/cc @klueska
from ecosystem.
It seems that the question is about GPU support in DC/OS? Is that correct?
GPUs will be supported in the upcoming DC/OS 1.9 release with the limitations outlined in this pull request: dcos/dcos#766
The full documentation is not yet complete (it will be ready by 1.9 EA release coming out in early February though). Here is a preview: https://github.com/dcos/dcos-docs/blob/a20548c343a75258ea70799efb9d98c9c6aeeaf7/1.8/usage/gpu-support.md
from ecosystem.
@klueska Yes cause actually docker it is mandatory
from ecosystem.
What do you mean by "docker is mandatory". Can you give a little more detail on the context?
DC/OS actually supports two different container runtimes (The Docker Container Runtime and the Universal Container Runtime), both of which are able to run docker containers. The Docker Container Runtime does not yet support GPUs, but the Universal Container Runtime does.
from ecosystem.
Yes I know and tfmesos use both the solutions but see the flavour on how it is integrated here: https://github.com/tensorflow/ecosystem/blob/master/marathon/README.md
from ecosystem.
I see. Looking at https://github.com/tensorflow/ecosystem/blob/master/marathon/template.json.jinja
I see no reason that this can't be modified to run with the Universal Container Runtime instead of the Docker Container Runtime (a.k.a. the docker containerizer).
The only change would be to update the container type to "MESOS" and allocate it some "gpus: xxx" if desired.
In fact, that would be the recommended way of running it going forward.
from ecosystem.
Yes and this originated my original question. I'am also courious on how new hardware devices like TPU or other kind of accellerators could be exposed in the Universal Container Runtime.
from ecosystem.
@windreamer Regarding:
Hmm, it is a wild idea... maybe the most difficult part is how to map GPUs into different tasks?
You could consider launching all of the tasks as part of a task-group
(aka Pod
), in which case they would all share access to the same set of GPUs instead of having to allocate a different GPU to each of them.
from ecosystem.
@bhack Regarding:
Yes and this originated my original question. I'am also courious on how new hardware devices like TPU or other kind of accellerator could be exposed in the Universal Container Runtime.
That sounds like a great question for the mesos development list: http://mesos.apache.org/community/
There is nothing fundamental about how we do GPU allocation that couldn't be extended to TPUs.
from ecosystem.
@klueska Yes but I don't have a TPU :) Only google has this ASIC and I think will be only available in managed cloud so we cannot see it as Mesos resource also on Google Cloud.
from ecosystem.
Thanks @klueska ! I will try it when DC/OS 1.9 is available: The only change would be to update the container type to "MESOS" and allocate it some "gpus: xxx" if desired.
from ecosystem.
@bhack We will give more details about TPU but for now TPU is beyond the scope. We don't have to worry about supporting TPU, especially in Mesos, in the near future.
from ecosystem.
@bhack I am quite positive docker
is not necessary for tfmesos
cluster. mesos
uses cgroup
's device whitelist to isolate GPUs from each task. So when a GPU is allocated, only the task it assinged to can use this device.
I thought docker
was a must is based on a guess: Tensorflow
uses GPUs starting with id 0
, and when with nvidia-docker
GPUs are automatically re-numbered in docker
. But based on my experiments, it seemd Tensorflow
also do EnumerateDevices
to discover all available GPUs. And thus re-numbering GPU devices is not nessesary.
from ecosystem.
But please note most our node has only one GPU, so my experiment may have been wrongly done.
from ecosystem.
@yuefengz Thank you. This mean that TPU will be available only on managed Google Cloud for now. Right?
from ecosystem.
Note that we can't comment on any TPU plans :)
from ecosystem.
@jhseu Ok we wait you to be ready to release more info. Instead, will be there an impact of XLA in Mesos?
from ecosystem.
It's unlikely XLA will directly affect Mesos. The primary benefits of XLA are:
- Improved performance (from fusing ops, faster op kernels)
- Easier support for making new hardware work in TensorFlow
from ecosystem.
Mhh probably on task migration XLA could require to emit new native code or another ops fusing strategy. What do you think?
from ecosystem.
XLA shouldn't affect task migration because each worker will JIT compile native code when ops are executed for the first time.
from ecosystem.
Ok so JIT will be executed on each worker as required.
from ecosystem.
@klueska Is 1.9 ea released right? Could we update the container type in this repository example and related docs?
from ecosystem.
@bhack Yes, DC/OS 1.9 EA was released last night. Give it a shot and let me know if there are any problems.
from ecosystem.
from ecosystem.
The patch it is really trivial but I want to ask to maintainers if they have a cluster to test it.
from ecosystem.
Yeah, we can set up a cluster to test it, so feel free to send a pull request.
from ecosystem.
from ecosystem.
Ok please comment #38
from ecosystem.
@windreamer @klueska Any plan to push Tensorflow on DC/OS universe?
from ecosystem.
from ecosystem.
@klueska Take a look at TF distributed
from ecosystem.
It is also important tensorflow/tensorflow#2126
from ecosystem.
#38 is merged. I've opened mesosphere/universe#1026 for the universe topic.
from ecosystem.
I want to ask to all if makes sense to integrate more pymesos and tfmesos features here. What do you think?
from ecosystem.
Another probable alternative is https://github.com/daskos/mentos/ /cc @daskos
from ecosystem.
DC/OS 1.9 stable release is generally available
from ecosystem.
I see no activity here. Any further idea or action? Do we want to close it?
from ecosystem.
@klueska Is there any other feedback on how to improve DC/OS Mesos experience?
from ecosystem.
How this news hints can improve the mesos demo? Could be created another example on a "real" dataset like imagenet?
from ecosystem.
This thread it is quite dead. Try to see if there is any interest by @dcos team.
from ecosystem.
In the meantime https://dcos.io/blog/2017/tutorial-deep-learning-with-tensorflow-nvidia-and-apache-mesos-dc-os-part-1/index.html /cc @sascala
from ecosystem.
@jhseu After Google I/O I could suppose that TPU could not be related anymore to Ecosystem cause it is full managed. Is this conclusion correct? If you adopt ecosystem/mesos or DC/OS your are out of TPU device offering.
from ecosystem.
@bhack you should be able to run a TensorFlow job on a DC/OS cluster running on GCE nodes, and connect to a Cloud TPU using distributed TensorFlow.
from ecosystem.
Is it really an integration? Can I put TPU in https://github.com/tensorflow/ecosystem/blob/master/marathon/template.json.jinja#L9?
from ecosystem.
@bhack The initial release will work with vanilla VMs, so a cluster manager may still be useful.
from ecosystem.
Ok I will close.this in the next days.. seems to me that there is no input by other stakeholders to improve Mesos/Dcos resources in this repository.
from ecosystem.
I played around with tfmesos
quite a bit, but never quite got it working in a way that I liked on DC/OS.
We are now working on a distributed Tensorflow framework built using the DC/OS SDK https://github.com/mesosphere/dcos-commons
Once it's ready, that will likely be the suggested way to run Tensorflow on DC/OS.
I will update this repo with instructions on how to use it once it's ready.
from ecosystem.
Ok I leave this open so we can discuss a little bit what info we want to put here when the DC/OS work will be ready.
from ecosystem.
I know this thread is closed, but I wanted to point out the new release of distributed TensorFlow on DC/OS that we announced today. https://mesosphere.com/blog/tensorflow-gpu-support-deep-learning/
from ecosystem.
@klueska so what do you plan to do with the k8 official effort?
As DC/OS declared full k8 support just some weeks ago.
from ecosystem.
Related Issues (20)
- About spark-tensorflow-distributor---I have the same problem as you. When I use a spark cluster with two or more nodes, the algorithm needs the same time as if I have only one node. I don't understand why. Please, someone answer this doubt
- Does this connector work with TF 2.x? HOT 4
- Error when deserializing tfrecord's in TF 2.x: Only integers, slices (`:`), ellipsis (`...`), tf.newaxis (`None`) and scalar tf.int32/tf.int64 tensors are valid indices
- Failed to build spark-tensorflow-connector_2.12 HOT 3
- Functionality to save the trained model HOT 4
- Spark TensorFlow Distributor: Spark custom resource scheduling - when and how?
- Get error when I run spark-tensorflow-connector example... HOT 1
- Cannot convert field to unsupported data type StructType([StructField("user_flush_num", ArrayType(IntegerType(), True)),StructField("field2", ArrayType(IntegerType(), True))]) HOT 2
- Task failed while writing rows HOT 2
- spark-tensorflow-distributor: RAM overflow when running ResNet152 HOT 1
- ByteType not supported, while IntegerType is too space-consuming
- Is it possible to use the spark tensorflow distributor with Parameter server strategy?
- How to make the “YOLOv4 object detection” or “YOLOv5 object detection” or “U-Net segmentation” under the Spark cluster?
- Build new Spark-Tensorflow-Connector for scala 2.12 HOT 5
- Failing to save the model pickle. and how to check for the predictions? HOT 1
- java.lang.NullPointerException when save as TFRecords
- How to avoid aggregate(shuffle) in processing the tfrecord file?
- mirrored_strategy_runner.py for multiple executors / workers
- Limitation on Spark "task_gpu_amount" cannot be less than 1
- why spark-tensorflow-connector can't support double and integer data type.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ecosystem.