awslabs / aws-batch-helpers Goto Github PK

View Code? Open in Web Editor NEW

132.0 15.0 44.0 14 KB

AWS Batch helpers is a collection of scripts and tools that can be used with AWS Batch.

License: Apache License 2.0

Shell 37.56% Python 62.44%

aws-batch-helpers's Introduction

aws-batch-helpers

This repo containers tools and scripts that can be used with AWS Batch - https://aws.amazon.com/batch/.

Issues

Any issues or feedback, should be posted in the AWS Batch forum - https://forums.aws.amazon.com/forum.jspa?forumID=240

aws-batch-helpers's People

Contributors

Stargazers

Watchers

aws-batch-helpers's Issues

Add LaunchTemplate Example

I hope I am in the right place for this comment:

I spent quite a bit of time working to get a aws batch launch template through a cloud formation template. I needed extra data volume on an EC2 instance I was hoping you could add this to as a second example instead of creating a new EC2 instance which is not required when you have LaunchTemplate that can be used in AWS Batch, thankfully!

What was really difficult to figure out was the version on ComputeEnvironment ref to the latest LaunchTemplate:

SpecialComputeEnvironment:
    Type: "AWS::Batch::ComputeEnvironment"
    Properties:
      Type: MANAGED
      ComputeEnvironmentName: !Sub "Special-ComputeEnvironment-${AWS::Region}-1" 
      ServiceRole: !Ref SpecialServiceRole
      ComputeResources:
        MaxvCpus: 256
        SecurityGroupIds:
          - !Ref SecurityGroup
        Subnets: !Ref Subnets
        Type: EC2
        MinvCpus: 0
        ImageId: ami-007571470797b8ffa # ami-0a6a36557ea3b9859
        InstanceRole: !Ref SpecialInstanceProfile
        InstanceTypes:
          - optimal
        LaunchTemplate:
          **LaunchTemplateId: !Ref SpecialComputeLaunchTemplate**
          **Version: !GetAtt SpecialComputeLaunchTemplate.LatestVersionNumber**
        DesiredvCpus: 0
      State: ENABLED

And the LaunchTemplates UserData:

  SpecialComputeLaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties:
      LaunchTemplateName: "Special-inc-volume-encrypt"
      LaunchTemplateData:
        BlockDeviceMappings:
          - DeviceName:  '/dev/sdb'  # Amazon ECS-Optimized Amazon Linux 2 ami accepts EDS vol /dev/sd[b-l]
            Ebs:
              DeleteOnTermination: true
              Encrypted: true
              VolumeSize: 250
              VolumeType: "gp2"
        UserData:
         "Fn::Base64": !Sub |
            MIME-Version: 1.0
            Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

            --==MYBOUNDARY==
            Content-Type: text/x-shellscript; charset="us-ascii"

            #!/bin/bash          
            mkfs.ext4  /dev/sdb
            (
            echo n # Add a new partition
            echo p # Primary partition
            echo 1 # Partition number
            echo   # First sector (Accept default: 1)
            echo   # Last sector (Accept default: varies)
            echo w # Write changes
            ) | fdisk  /dev/sdb
            mkfs.ext4  /dev/xvdb1
            mkdir /data
            echo " /dev/xvdb1       /data  ext4    defaults        0       2" >> /etc/fstab
            mount -a
            serSpeciale docker restart 

            --==MYBOUNDARY==

https://github.com/vfrank66/awsbatchlaunchtemplate

Private docker images

Launch Template :

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==

Content-Type: text/x-shellscript; charset="us-ascii"


#!/bin/bash
echo ECS_IMAGE_CLEANUP_INTERVAL=60m >> /etc/ecs/ecs.config
echo ECS_IMAGE_MINIMUM_CLEANUP_AGE=60m >> /etc/ecs/ecs.config
export DOCKER_USERNAME=boomtable
export DOCKER_PASSWORD=xyz
docker login  --username $DOCKER_USERNAME --password=$DOCKER_PASSWORD
--==MYBOUNDARY==--

Error while triggering the batch job !

Status reason

Task failed to start

Container message

CannotPullContainerError: Error response from daemon: pull access denied for myprivate/testbatch, repository does not exist or may require 'docker login'

Am i doing anything stupid !? is this a right way to do it ?

Where is the output?

Hey!

I am trying to use fetch & run script together with my custom script. I am trying to download a file using aws s3 cp S3_SOURCE . However, the file seems to get downloaded, but when doing an ls -al ., nothing is there... It seems fetch & run script is hiding stderr or something cause I have no idea what is going on..

The docker I am using is created like this:

Help appreciated, where is the file?

ParamValidationError for imageId create batch entities

When I run the create-batch-entities.py script to create batch entities, I get the following validation error. I have tried passing in the imageId with and without the 'ami-' prefix with the same outcome. I am using Python 2.7.12 and boto-2.48.0.

Traceback (most recent call last):
  File "create-batch-entities.py", line 167, in <module>
    main()
  File "create-batch-entities.py", line 158, in main
    keyPair=keyPair)
  File "create-batch-entities.py", line 56, in create_compute_environment
    'instanceRole': instanceRole
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 251, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 513, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 566, in _convert_to_request_dict
    api_params, operation_model)
  File "/Library/Python/2.7/site-packages/botocore/validate.py", line 270, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in computeResources: "imageId", must be one of: type, minvCpus, maxvCpus, desiredvCpus, instanceTypes, subnets, securityGroupIds, ec2KeyPair, instanceRole, tags, bidPercentage, spotIamFleetRole```

$ECS_INIT_PACKAGE not initialized

$ECS_INIT_PACKAGE not initialized in ami-checker.sh.

Error "Unable to find AWS CLI executable."

Summary

There are multiple resources available out there that reference the files in this project. I have painstakingly went through many of them attempting to find where the gap is on this issue and am hoping to save others that same pain.

The AWS CLI is indeed available in the image, so the error is actually misleading. After tracing the fetch-and-run/fetch_and_run.sh script the lines below throw false positives due to the which command no longer being part of the amazonlinux image.

# Check that necessary programs are available
which aws >/dev/null 2>&1 || error_exit "Unable to find AWS CLI executable."
which unzip >/dev/null 2>&1 || error_exit "Unable to find unzip executable."

You could simply remove those lines and operate under the assumption that these are installed (since you are explicitly installing them anyway) or you could keep the check by modifying them to something like the following:

# Check that necessary programs are available
aws --version >/dev/null 2>&1 || error_exit "Unable to find AWS CLI executable."
unzip -v >/dev/null 2>&1 || error_exit "Unable to find unzip executable."

awslabs / aws-batch-helpers Goto Github PK

aws-batch-helpers's Introduction

aws-batch-helpers

Issues

aws-batch-helpers's People

Contributors

Stargazers

Watchers

Forkers

aws-batch-helpers's Issues

Add LaunchTemplate Example

Private docker images

Where is the output?

ParamValidationError for imageId create batch entities

$ECS_INIT_PACKAGE not initialized

Error "Unable to find AWS CLI executable."

Summary

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent