nodejs 14

AWS HPC Cluster Deployment Guide 🚀

Step 1

To deploy, click:

Region	Launch
North Virginia (us-east-1)
Oregon (us-west-2)
Ireland (eu-west-1)
Frankfurt (eu-central-1)

More Regions (Click to expand)

** Warning! This QuickStart is untested in the following regions. Use at your own risk **

Region	Launch
Ohio (us-east-2)
California (us-west-1)
London (eu-west-2)
Paris (eu-north-1)
Stockholm (eu-north-1)
Middle East (me-south-1)
South America (sa-east-1)
Canada (ca-central-1)
Hong Kong (ap-east-1)
Tokyo (ap-northeast-1)
Seoul (ap-northeast-2)
Mumbai (ap-south-1)
Singapore (ap-southeast-1)
Sydney (ap-southeast-2)

If you have not authenticated with the AWS Management Console, you will be prompted to login with the AWS Account ID or alias, IAM user name, and password that was provided to you.

Step 2

After clicking Launch Stack you will be redirected to the CloudFormation Quick create stack screen.

Step 3

Scroll to the bottom of the page, and leave most of the Parameters as they are.

Enter an initial BudgetLimit for the project. This will be used to track spending and send an alert when you cross 80%.
Update NotificationEmail that will receive budget notifications.
Fill out the UserPasswordParameter with a temporary password. Keep it simple! You will be prompted to change it on first use.
Select I acknowledge that AWS Cloudformation might create IAM resources.
Now, click Create Stack to deploy the QuickStart Environment.

Deployment

Deployment takes about 15 minutes. This QuickStart provisions:

a Cloud9 Integrated Development Environment (IDE) in the selected region;
an AWS Parallel Cluster environment, named hpc-cluster, for batch scheduled jobs and interactive computing;
a non-root IAM User, with full Administrator access to the AWS Console to create custom architectures.
a Budget notification email at 80% of the BudgetLimit

Provisioning is complete when all Stacks show CREATE_COMPLETE.

Onboarding Users 👨‍💻👩‍💻

When all stacks show CREATE_COMPLETE, click on the Outputs tab of the AWS-HPC-Quickstart stack. Send the end-user the following information (3 of 5 are on the Outputs tab):

ResearchWorkspaceURL -- URL to directly access the Cloud9 Research Environment.
UserLoginUrl -- To authenticate into the AWS Console to create custom architectures.
UserName -- The AWS IAM username for the end user.
Password that you entered to launch the CloudFormation stack.
The USER_GUIDE included in this repository.

Embedding this Solution in Your Own CloudFormation Template

Click to expand

NoTears is provided as a standalone CloudFormation template with a large number of parameters. Use-cases where configuration options need to be limited, or NoTears embedded as one component of a broader architecture are also supported. To do this, nest the stack as follows:

AWSTemplateFormatVersion: '2010-09-09'
Resources:
    NoTearsHPC:
      Type: 'AWS::CloudFormation::Stack'
      Properties: 
          TemplateURL: 'https://notearshpc-quickstart.s3.amazonaws.com/0.2.3/cfn.yaml'
          Parameters: 
            OperatingSystem: alinux2
            EnableBudget: false
            CreateUserAndGroups: true
            EnableFSxLustre: false
            SpackVersion: v0.15.0

An illustrating example is provided in inherit.yaml.

FAQ 🧐

If you see the following: The security token included in the request is invalid

It's likely an issue with the account having been just created. Wait for some time and try again.
On first login to the Cloud9 terminal you may see something like:
```
Agent pid 3303
Identity added: /home/ec2-user/.ssh/AWS-HPC-Quickstart-NJCH9esX (/home/ec2-user/.ssh/AWS-HPC-Quickstart-NJCH9esX)
```
Ignore these messages. They indicate that your SSH key for Parallel Cluster was located.

Developer Setup

Click to expand

Note: This section is only for developing the solution, to create a cluster, see Launch the HPC Quickstart Envrionment

The first step is installing node.js, this can be done easily with Homebrew. After that completes, install aws-cdk:

$ brew install node
$ npm install -g aws-cdk

(Alternatively, use brew install aws-cdk)

Optionally create a python virtualenv or conda environment. We assume this is in ./.env.

Now you can activate the python virtualenv and install the python dependencies:

$ source .env/bin/activate               # Can be skipped if not using a virtualenv
$ pip install -r requirements.txt

Make sure your region and aws credentials are setup by running:

$ aws configure

Acquire a Github AuthToken:

sign into your account
Settings -> Developer Settings -> Personal Access Tokens
Create token without checking any boxes
Copy the provided token and add to your $HOME/.bashrc:

export GIT_AUTH_TOKEN=XXXXXXX

At this point, it's time to setup CDK, the following needs to be done once in each account:

$ cdk bootstrap

And finally, deploy the app:

$ cdk deploy --parameters UserPasswordParameter=******* --parameters NotificationEmail=*******@amazon.com

I've surely missed a bunch of python dependencies, the format for installing those is:

$ pip install aws-cdk.custom-resources

Once it finishes deploy, you'll get an ouput with a link to the Cloud9 URL. Click on that to quickly see the Cloud9 result:

Deploying to Public Bucket

Use the include upload.sh script to publish the template and assets (lambda zips, bootstrap scripts, etc.) in an S3 bucket.

$ sh upload.sh [target-bucket-name]

This will create a modified cfn.yaml in the app directory. The script also creates buckets:

s3://target-bucket-name, the central repository for the template and assets
s3://target-bucket-name-region, a clone of the central repository, where region expands to all AWS public regions.

Before publishing the template, the upload script synthesizes a yaml template and transforms the parameters and asset URLs to match the regional buckets. The final output, cfn.yaml, will reflect all changes.

Pro-Tips

Use cdk synth | less to see the generated template.

Provide parameters to the stack via cdk deploy --parameters pcluster:KEY=VALUE.

Useful commands

cdk ls list all stacks in the app
cdk synth emits the synthesized CloudFormation template
cdk deploy deploy this stack to your default AWS account/region (remember to update the version number on the quick launch buttons in the README!)
cdk diff compare deployed stack with current state
cdk docs open CDK documentation

Enjoy!

	# returns the InstanceId. Timeout of 20000 milliseconds.

	while True:
	# Filters SSM Managed Instances to find only the matching cloud9 env
	instance_info = ssm_client.describe_instance_information(Filters=[{
	'Key': 'tag:aws:cloud9:environment', 'Values': [ "{}".format(cloud9_environment)]
	}])
	# if instance not found/not registered to SSM, instance_info is empty.
	if instance_info.get('InstanceInformationList'):
	if instance_info.get('InstanceInformationList')[0].get('PingStatus') == 'Online':
	return instance_info.get('InstanceInformationList')[0].get('InstanceId')
	if context.get_remaining_time_in_millis() < 20000:

aws-samples / no-tears-cluster Goto Github PK

no-tears-cluster's Introduction

AWS HPC Cluster Deployment Guide 🚀

Step 1

Step 2

Step 3

Deployment

Onboarding Users 👨‍💻👩‍💻

Embedding this Solution in Your Own CloudFormation Template

FAQ 🧐

Developer Setup

Deploying to Public Bucket

Pro-Tips

Useful commands

no-tears-cluster's People

Contributors

Stargazers

Watchers

Forkers

no-tears-cluster's Issues

Feature Request

Recommend Projects

Recommend Topics

Recommend Org