This project is a platform for distributed optimization expriments using OpenMPI. This platform is used to implement the expeirments of our recent papers [1][2]. Please cite them if this code helps you. :)

[1] Zhouyuan Huo, Heng Huang, Asynchronous Stochastic Gradient Descent with Variance Reduction for Non-Convex Optimization. AAAI, 2017

[2]Zhouyuan Huo, Bin Gu, Heng Huang, Decoupled Asynchronous Proximal Stochastic Gradient Descent with Variance Reduction. arXiv

1. Clone this repository from github.
2. run ./install

Remark: ./install is only used for first time installation. After you modify the code and want to compile, just run make in build/ directory.

1. Generate distributed datasets. Go to build directory, and run ./splitdata ../data/covtype_binary 4 0. It will generate 4 data files in data/covtype_binary_split/ directory. ./splitdata will transform libsvm format data to armadillo format data, and distributes it to multiple files. run ./splitdata directly to see how to use it.

2. Run To run do

mpirun -np 5 ./mpiplatform -logistic_l2_l1 -num_workers=4 -data_file="absolutepathto/data/covtype_binary_split/" -print_loss_per_epoch -d1=54 -learning_rate=1e-1 -n_epochs=100 -mini_batch=100 -in_iters=1000 -svrg -max_delay=10

1. Generate distributed datasets. Go to build directory, and run ./splitdata ../data/covtype_multiclass 4 0. It will generate 4 data files in data/covtype_multiclass_split/ directory.

2. Run To run do

mpirun -np 5 ./mpiplatform -fcn -num_workers=4 -data_file="absolutepathto/data/covtype_multiclass_split/" -print_loss_per_epoch -d1=54 -d2=20 -d3=7 -learning_rate=1e-3 -n_epochs=100 -mini_batch=10 -in_iters=1000 -svrg -max_delay=10

1. Open an account on Amazon Web Services (AWS).

2. Launch instances on AWS EC2. Configure: (1) Choose AMI: Ubuntu Server, (3) Configure Instance: Number of instances 5, (6) Configure Security Group: Type: All TCP. Then click launch and download a key named FirstKey.pem.

3. Setup the public key. mv Firstkey.pem ~/.ssh/ ＆  chmod 400 ~/.ssh/Firstkey.pem

4. Generate compressed data. Go to data/ directory, and run ./compress_data.sh absolutepashto/data/covtype_binary_split/ 4.

5. Generate compressed code. Download our project from github again, copy Firstkey.pem to the repository and compress it. run tar cvf aws_code.tar MPIPlatform

6. Upload and install. Open upload_install_split.sh and fill in the hosts part with public ip of 5 machines we just launched.

hosts="
54.175.225.66
54.174.70.95
54.211.115.252
52.91.166.132
54.175.107.232
" 

First machine works as server. Then upload and install by running

./upload_install_split.sh absolutepathto/aws_code.tar  absolutepathto/covtype_binary_split/

7 . Login to server and setup. To login the server, do

ssh -i ~/.ssh/Firstkey.pem [email protected]

Go to bulid/ and create a hostfile like

54.175.225.66
54.174.70.95
54.211.115.252
52.91.166.132
54.175.107.232

8 . Run the program, do

mpirun -hostfile hostfile ./mpiplatform -logistic_l2_l1 -num_workers=4 -data_file="/home/ubuntu/" -distribute -print_loss_per_epoch -d1=54 -learning_rate=1e-1 -n_epochs=100 -mini_batch=100 -in_iters=1000 -svrg -max_delay=10

This repository uses code from Cyclades, we borrow the framework from this project. And we use code from LibSVM to read libsvm data and transform it to distributed armadillo format.