databiosphere / toil Goto Github PK

Answer this question: On an 8 core machine, is it faster to parallelize 2 tasks by assigning them 4 cores each or is the same as if they were each assigned 8?

boto is failing to upload files via: set_contents_from_filename() that are greater than 5Gb.

try:
multipart_upload
except:
cancel
else:
complete

http://alex.vanboxel.be/2015/03/12/using-luigi-on-the-google-cloud-platform/
http://www.slideshare.net/EliasFreider/luigi-pydata-presentation

setup should start master and slave
teardown should kill master and slave

For this sprint and maybe the next we will assume there are no failures. Beyond that we should implement the protocol described below

http://mesos.apache.org/documentation/latest/reconciliation/

Thanks to @cket for finding this.

http://www.schedmd.com/slurmdocs /elastic_computing.html

Gets hung up on existince of jobTree directory. One of two things should happen: either an error message should be printed or the script should resume and skip previously finished tasks and then exit with 0 (success)

1. Write script to run pipeline on all pairs
2. Record evaluation (sens/spec/accuracy)

To fully allow all state of a jobTree to be encoded in the jobStore we need to remove the assumption that every jobTree batch has a jobTree directory on the file system. This is mostly done - but for a number of places where the directory is used for storing temporary files by the master.

Eliminate the auto detection of the BS

http://www.biomedcentral.com/1471-2105/13/77