Define which formats Cirrus supports

S3.cpp needs to be refactored to a more object-oriented interface. As of now, this module has a couple of issues:

1. no way to create Aws::Client's with different options
2. no easy way to pass clients to S3 without passing these AWS::Client handles

S3.cpp should be a class (I suggest S3Client) so users can create multiple of these clients.

#52

#49

In one file:

SKLearn model objects provide a predict function that lets the user provide input for predictions/evaluations on the trained model. Cirrus should do that as well.

We should add a .clang-format to run in travis.

There are issues with the python frontend when the plotly version is 3.1.0. The plotly version has to be exactly 2.7.0 to work.

We should leverage flatbuffers for building messages used in the communication between the workers and the parameter server.

Periodic checkpointing of model to remote storage. #19

If I start a grid search run with UI on and kill the experiment mid way, it seems I can't rerun it without restarting jupyter.

The reason seems to be that the dash server is still bound on the port. What is the solution for this?

New softmax model for Cirrus

Consistent with old bug reported in multiple places, including here: https://sourceware.org/bugzilla/show_bug.cgi?id=10652

It seems that a few people (at least me, Andrew and Neel) have found that travis clang-format sometimes suggests different code formattings than the one installed locally.

I'm opening this issue to advance towards a solution to this problem.

Need to shard the models across multiple VMs/lambdas to provide more aggregate bandwidth.

If user's public key is already in the server's .ssh/authorized_keys file, there is no way to specify an empty .pem file.

These tests should be used to test the system in different environments: Windows, OSx.

https://docs.google.com/document/d/1G-2HCMcUWVcRAGICb5u0_b8HYcP66uTWEvGnWK1rU5A/edit?usp=sharing

Add support for Latent Dirichlet Allocation on Cirrus
#51

The 0th poll thread needs to start last. We need a barrier to ensure this, or else for large numbers of poll threads, parameter server is likely to fail.

pip install cirrus
#59

Is there any reason that there're two lambda definition exists in the code base?
One is

Right now we read the tail of the log file from the ErrorSparseTask.

We should probably create an UDP/TCP interface in the error task,

This bucket is not exist now,so I want to creat it again. But I'm confused what is in it？Looking forward to ur reply~~

As of now we don't count 1) cost of bandwidth from S3 to the Cirrus workers, 2) cost of S3 requests.

The cost of requests can be expensive for very high IOPS.

If the Error Task requests a model after the parameter server has started but before the PS has initialized the model, the error task will segfault.

Run multiple workloads at the same time and show progress in realtime with multiple plots.

Fix bug in min max scaling in python tests

Currently, our experiments only terminate when the user calls stop in the python interface. We should have an option to set a limit, so that the experiment automatically stops when it reaches a certain accuracy or some other limit.

It seems sometimes the travis tests fail because not enough time is given for the test to converge.

Should we increase the threshold? @andrewmzhang @yangryan0

Make a list here of different ML algorithms we could support on Cirrus.