Revive the distributed solver efforts about caffe HOT 11 CLOSED

Decaf has already used MPI in a few places.

from caffe.

Just a precaution type note: I used mpi in my earlier projects that never
got open-sourced (parallel linear models over a reasonably sized cluster,
see e.g. my ICCV 2013 task adaptation paper). I don't recall completely
making mpi runnable under either decaf and caffe, though...

Yangqing

On Thu, Feb 6, 2014 at 9:30 PM, kloudkl [email protected] wrote:

Decaf has already used MPI in a few placeshttps://github.com/UCB-ICSI-Vision-Group/decaf-release/search?q=mpi&ref=cmdform
.

Reply to this email directly or view it on GitHubhttps://github.com//issues/65#issuecomment-34405438
.

from caffe.

The first open source large scale machine learning projects that I encountered were Vowpal Wabbit[1] and Edward Y. Chang's PSVM, PLDA, Parallel Spectral Clustering which all used MPI but none were based on CUDA. Neither did they train deep nonlinear models. But the achievements of the industry such as Baidu IDL should motivate the academy towards a comparable large scale distributed training framework. A progressive roadmap may be to implement a version on CPU at first and add GPU capability after the initial success.

[1] Alekh Agarwal, Olivier Chapelle, Miroslav Dudik, John Langford, A Reliable Effective Terascale Linear Learning System, 2011.

from caffe.

I am interested in working on this. Is there still work ongoing?

I am thinking something like [1], which uses MPI + cuda-convnet. There is also an interesting write-up by Netflix [2] where they use distributed computing for hyperparameter tuning.

[1] Paine, T., Jin, H., Yang, J., Lin, Z., & Huang, T. (2013). GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training. arXiv Preprint arXiv:1312.6186

[2] http://techblog.netflix.com/2014/02/distributed-neural-networks-with-gpus.html

from caffe.

@Yangqing, would you please recover the related commits?

for commit in 64e28ba 591c36b a3eb62a a48147c; do git cherry-pick $commit; done

from caffe.

Microsoft Project Adam sounds very promising [1].

[1] Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, Karthik Kalyanaraman, "Project Adam: Building an Efficient and Scalable Deep Learning Training System"
To appear in the 11th USENIX Symposium on Operating Systems Design and Implementation '14 (OSDI), Oct. 2014.

Unfortunately, the paper won't be public until the conference is held in October. Did anyone register the ODSI 2014 and have access to the paper?

from caffe.

Is it based on Hogwild ?

from caffe.

The answer is only in the paper.

from caffe.

@kloudkl Some other preliminar info on Project Adam.

from caffe.

The paper became public this Monday.

from caffe.

This is in-progress through #1148 so this place-holder issue is no longer needed.

from caffe.