Comments (4)
I will look into this. Realistically I will likely not have time until the new year.
I am posting an example for my reference:
https://github.com/dejonghe/aws-batch-example/blob/master/cloudformation/batch/batch-example.yml
The main custom components that Funnel will need is the JobDef and a custom AMI depending on your use case.
from funnel.
Current WIP:
AWSTemplateFormatVersion: '2010-09-09'
Description: 'AWS CloudFormation Sample Template Managed Single Batch Job Queue: This
template demonstrates the usage of simple Job Queue and EC2 style Compute Environment. **WARNING**
You will be billed for the AWS resources used if you create a stack from this template.'
Resources:
##-------------
## Network
##-------------
VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
InternetGateway:
Type: AWS::EC2::InternetGateway
RouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId:
Ref: VPC
VPCGatewayAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId:
Ref: VPC
InternetGatewayId:
Ref: InternetGateway
SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: EC2 Security Group for instances launched in the VPC by Batch
VpcId:
Ref: VPC
Subnet:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: 10.0.0.0/24
VpcId:
Ref: VPC
MapPublicIpOnLaunch: 'True'
Route:
Type: AWS::EC2::Route
Properties:
RouteTableId:
Ref: RouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId:
Ref: InternetGateway
SubnetRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId:
Ref: RouteTable
SubnetId:
Ref: Subnet
##-----------
## Roles
##-----------
BatchServiceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: batch.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole
EcsInstanceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2008-10-17'
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
EcsTaskRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2008-10-17'
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: FunnelS3
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- 's3:GetBucketLocation'
- 's3:GetObject'
- 's3:ListObjects'
- 's3:ListBucket'
- 's3:CreateBucket'
- 's3:PutObject'
Resource: '*'
- PolicyName: FunnelDynamodb
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- 'dynamodb:GetItem'
- 'dynamodb:PutItem'
- 'dynamodb:UpdateItem'
- 'dynamodb:Query'
Resource: '*'
##-----------------
## Batch Resources
##-----------------
JobDefinition:
Type: AWS::Batch::JobDefinition
Properties:
Type: container
ContainerProperties:
Image: docker.io/ohsu-comp-bio/funnel:latest
Vcpus: 1
Memory: 512
Command:
- 'worker'
- 'run'
- '--EventWriters'
- 'log'
- '--EventWriters'
- 'dynamodb'
- '--Database'
- 'dynamodb'
- '--DynamoDB.Region'
- Ref: AWS::Region
- '--DynamoDB.TableBasename'
- 'funnel'
- '--Worker.WorkDir'
- '/opt/funnel-work-dir'
- '--taskID'
- 'Ref::TASKID'
Volumes:
- Host:
SourcePath: /var/run/docker.sock
Name: docker-sock
- Host:
SourcePath: /opt/funnel-work-dir
Name: funnel-work-dir
MountPoints:
- ContainerPath: /var/run/docker.sock
ReadOnly: false
SourceVolume: docker-sock
- ContainerPath: /opt/funnel-work-dir
ReadOnly: false
SourceVolume: funnel-work-dir
Environment: []
Ulimits: []
ResourceRequirements: []
User: ""
ReadonlyRootFilesystem: true
Privileged: true
JobRoleArn:
Ref: EcsTaskRole
RetryStrategy:
Attempts: 1
JobQueue:
Type: AWS::Batch::JobQueue
Properties:
Priority: 1
ComputeEnvironmentOrder:
- Order: 1
ComputeEnvironment:
Ref: ComputeEnvironment
ComputeEnvironment:
Type: AWS::Batch::ComputeEnvironment
Properties:
Type: MANAGED
ComputeResources:
Type: EC2
MinvCpus: 0
DesiredvCpus: 0
MaxvCpus: 256
InstanceTypes:
- optimal
AllocationStrategy: BEST_FIT
Subnets:
- Ref: Subnet
SecurityGroupIds:
- Ref: SecurityGroup
InstanceRole:
Ref: EcsInstanceRole
ServiceRole:
Ref: BatchServiceRole
Outputs:
ComputeEnvironmentArn:
Value:
Ref: ComputeEnvironment
JobQueueArn:
Value:
Ref: JobQueue
JobDefinitionArn:
Value:
Ref: JobDefinition
from funnel.
@golharam the above seems to produce all of the necessary AWS Batch resources.
Note: the template does not handle setting up the Funnel server.
Let me know if this works for you.
from funnel.
This works. I only needed parts of it, but still good that all is there. I recommend you make a command to output the CloudFormation template and let the user create the resources, rather than having funnel create the resources.
from funnel.
Related Issues (20)
- Funnel responds with 'Closed explicitly' HOT 9
- Too many open files HOT 6
- Retry failed task HOT 3
- Time limits for each state for a task HOT 3
- SSE options for AWS s3 HOT 10
- Storage: add DRS support
- Compute: HashiCorp Nomad HOT 7
- Issues accessing presigned s3 URLs HOT 9
- Funnel in Docker Container HOT 1
- Webdash Task view fails on display of task with SystemError
- Resource limits of running task
- PermissionDenied error when Basic Auth security is enabled HOT 4
- Funnel returns TaskLog instead of ExecutorLog HOT 4
- funnel 0.10 truncates output on text view of a task
- S3 upload fails due to required encryption
- provide better error messages
- Using instances with GPUs HOT 3
- Provide user credentials for mongodb for the Worker nodes HOT 1
- Funnel has service-info under /tasks/service-info, should be /service-info HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from funnel.