Giter Club home page Giter Club logo

Comments (4)

adamstruck avatar adamstruck commented on June 21, 2024

I will look into this. Realistically I will likely not have time until the new year.

I am posting an example for my reference:
https://github.com/dejonghe/aws-batch-example/blob/master/cloudformation/batch/batch-example.yml

The main custom components that Funnel will need is the JobDef and a custom AMI depending on your use case.

from funnel.

adamstruck avatar adamstruck commented on June 21, 2024

Current WIP:

AWSTemplateFormatVersion: '2010-09-09'
Description: 'AWS CloudFormation Sample Template Managed Single Batch Job Queue: This
  template demonstrates the usage of simple Job Queue and EC2 style Compute Environment.  **WARNING**
  You will be billed for the AWS resources used if you create a stack from this template.'
Resources:
  ##-------------
  ## Network
  ##-------------
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
  InternetGateway:
    Type: AWS::EC2::InternetGateway
  RouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId:
        Ref: VPC
  VPCGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId:
        Ref: VPC
      InternetGatewayId:
        Ref: InternetGateway
  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: EC2 Security Group for instances launched in the VPC by Batch
      VpcId:
        Ref: VPC
  Subnet:
    Type: AWS::EC2::Subnet
    Properties:
      CidrBlock: 10.0.0.0/24
      VpcId:
        Ref: VPC
      MapPublicIpOnLaunch: 'True'
  Route:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId:
        Ref: RouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId:
        Ref: InternetGateway
  SubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId:
        Ref: RouteTable
      SubnetId:
        Ref: Subnet

  ##-----------
  ## Roles
  ##-----------
  BatchServiceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: batch.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole

  EcsInstanceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2008-10-17'
        Statement:
          - Sid: ''
            Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role

  EcsTaskRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2008-10-17'
        Statement:
          - Sid: ''
            Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: FunnelS3
          PolicyDocument:
            Version: '2012-10-17'
            Statement: 
              - Effect: Allow
                Action:
                  - 's3:GetBucketLocation'
                  - 's3:GetObject'
                  - 's3:ListObjects'
                  - 's3:ListBucket'
                  - 's3:CreateBucket'
                  - 's3:PutObject'
                Resource: '*'
        - PolicyName: FunnelDynamodb
          PolicyDocument:
            Version: '2012-10-17'
            Statement: 
              - Effect: Allow
                Action:
                  - 'dynamodb:GetItem'
                  - 'dynamodb:PutItem'
                  - 'dynamodb:UpdateItem'
                  - 'dynamodb:Query'
                Resource: '*'

  ##-----------------
  ## Batch Resources
  ##-----------------
  JobDefinition:
    Type: AWS::Batch::JobDefinition
    Properties:
      Type: container
      ContainerProperties:
        Image: docker.io/ohsu-comp-bio/funnel:latest
        Vcpus: 1
        Memory: 512
        Command:
          - 'worker'
          - 'run'
          - '--EventWriters'
          - 'log'
          - '--EventWriters'
          - 'dynamodb'
          - '--Database'
          - 'dynamodb'
          - '--DynamoDB.Region'
          - Ref: AWS::Region
          - '--DynamoDB.TableBasename'
          - 'funnel'
          - '--Worker.WorkDir'
          - '/opt/funnel-work-dir'
          - '--taskID'
          - 'Ref::TASKID'
        Volumes:
          - Host:
              SourcePath: /var/run/docker.sock
            Name: docker-sock
          - Host:
              SourcePath: /opt/funnel-work-dir
            Name: funnel-work-dir
        MountPoints:
          - ContainerPath: /var/run/docker.sock
            ReadOnly: false
            SourceVolume: docker-sock
          - ContainerPath: /opt/funnel-work-dir
            ReadOnly: false
            SourceVolume: funnel-work-dir
        Environment: []
        Ulimits: []
        ResourceRequirements: []
        User: ""
        ReadonlyRootFilesystem: true
        Privileged: true
        JobRoleArn:
          Ref: EcsTaskRole
      RetryStrategy:
        Attempts: 1

  JobQueue:
    Type: AWS::Batch::JobQueue
    Properties:
      Priority: 1
      ComputeEnvironmentOrder:
        - Order: 1
          ComputeEnvironment:
            Ref: ComputeEnvironment
 
  ComputeEnvironment:
    Type: AWS::Batch::ComputeEnvironment
    Properties:
      Type: MANAGED
      ComputeResources:
        Type: EC2
        MinvCpus: 0
        DesiredvCpus: 0
        MaxvCpus: 256
        InstanceTypes:
          - optimal
        AllocationStrategy: BEST_FIT
        Subnets:
          - Ref: Subnet
        SecurityGroupIds:
          - Ref: SecurityGroup
        InstanceRole:
          Ref: EcsInstanceRole
      ServiceRole:
        Ref: BatchServiceRole

Outputs:
  ComputeEnvironmentArn:
    Value:
      Ref: ComputeEnvironment
  JobQueueArn:
    Value:
      Ref: JobQueue
  JobDefinitionArn:
    Value:
      Ref: JobDefinition

from funnel.

adamstruck avatar adamstruck commented on June 21, 2024

@golharam the above seems to produce all of the necessary AWS Batch resources.

Note: the template does not handle setting up the Funnel server.

Let me know if this works for you.

from funnel.

golharam avatar golharam commented on June 21, 2024

This works. I only needed parts of it, but still good that all is there. I recommend you make a command to output the CloudFormation template and let the user create the resources, rather than having funnel create the resources.

from funnel.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.