Run builds on AWS Batch [issue] about cli HOT 7 CLOSED

I've just pushed the CLI changes to adding support for nextstrain build --aws-batch: https://github.com/nextstrain/cli/compare/aws-batch.

I would appreciate any review of the code and especially the user interaction/experience. Until I complete some of the ancillary items on the todo list above, it will only work if you have admin access to the lab's AWS account (this means @trvrb and maybe @jameshadfield for now). I will leave another comment when I've arranged wider access so more folks can test if they want, hopefully tomorrow or sometime soon this week.

Note that the remote jobs are currently using the nextstrain/base:branch-aws-batch image from the aws-branch of docker-base: https://github.com/nextstrain/docker-base/compare/aws-batch. Relevant entrypoint code is there.

from cli.

Folks in our lab should now be able to try this out more widely.

from cli.

Re: compute limits: In our current AWS Batch configuration, each job defaults to 2 vCPUs with 4GB of memory and will be terminated if it does not complete in 4 hours. These are adjustable on a per-job basis, but the cli itself does not change the defaults. (Though an authorized user could.) The Batch compute environment (i.e. managed pool of EC2 instances) is limited to no more than 256 combined vCPUs. Instances are automatically provisioned, including down to zero instances (no cost) running if there are no jobs in the queue.

We should keep an eye on Batch usage and Batch-driven costs to make sure this is functioning as we expect. Only @trvrb (or someone else with access to Billing details) can do this.

If we start submitting large jobs, we should consider increasing the default job resources.

from cli.

Re: cost tracking: @trvrb these may be of interest:

https://aws.amazon.com/aws-cost-management/aws-cost-explorer/
https://aws.amazon.com/aws-cost-management/aws-cost-and-usage-reporting/

from cli.

Documentation is now at https://github.com/nextstrain/cli/blob/aws-batch/doc/aws-batch.md.

That URL (well, the URL for master not aws-batch) is referenced from the output of nextstrain build --help-all under the AWS Batch section.

from cli.

I've bumped the default job resources to 8 vCPUs and (just under) 16GiB of memory, which should cost about 34¢/hour on a c5.2xlarge. Combined with my augur PR to auto-scale alignment and tree-building parallelism, this should make larger builds run much quicker.

from cli.

Merged and released as 1.7.0.

from cli.