aws-samples / aws-sagemaker-build Goto Github PK

Creates a CloudFormation template that uses AWS StepFunctions to automate the building and training of Sagemaker custom models based on S3 and GitHub events

License: Apache License 2.0

Python 5.45% Makefile 0.52% JavaScript 65.08% Jupyter Notebook 17.51% Shell 1.96% HTML 9.48%

aws-sagemaker-build's Introduction

AWS SageMaker Build

Automating the build and deployment of machine learning models is an important step in creating production machine learning services. Models need to be retrained and deployed when code and/or data are updated. This project provides a full implementation of a CI/CD workflow and includes jupyter notebooks showing how to create, launch, stop, and track the progress of builds using python and Amazon Alexa! The goal of aws-sagemaker-build is to provide a repository of common and useful SageMaker/Step Function pipelines, to be shared with the community and grown by the community.

Prerequisites

Run Linux. (tested on Amazon Linux)
Install npm >5 and node >8. (instructions)
Clone this repo.
Set up an AWS account. (instructions)
Configure AWS CLI and a local credentials file. (instructions)

setup

install all need packages

npm install

copy config.js.example to config.js
create an s3 bucket. instructions. open up config.js and set templateBucket and AssetBucket to the name of your s3 bucket.
launch stack

npm run up

Stack Management

npm run up #launches stack

npm run update #updates the launched stack

npm run down #shuts down stack

template is written to /cloudformation/build/template.json

Architecture

The following diagram describes the flow of the Step Function StateMachine. There are several points where the StateMachine has to poll and wait for a task to complete.

The following diagram shows how the services work together

Parameters

AWS Systems Manager Parameter Store provides a durable, centralized, and scalable data store. We will store the parameters of our training jobs and deployment here and the Step Function's Lambda functions will query the parameters from this store. To change the parameters you just change the JSON string in the store. The example notebooks included with aws-sagemaker-build show how to do this.

Common

hyperparameters: default=HyperParameters,
hostinstancecount: default=1,
hostinstancetype: default=ml.t2.medium,
traininstancecount: default=1,
traininstancetype: default=ml.m5.large,
trainvolumesize: default=10,
trainmaxrun: default=4,
inputmode: default=File,
modelhostingenvironment: default={}
hyperparameters: default={},

BYOD

dockerfile_path_Training:
dockerfile_path_Inference:
train: default=true
build: default={Inference:true, Training:true}
TrainingImage:

TensorFlow

tensorflowversion: default=1.8,
trainingsteps: default=1000,
evaluationsteps: default=100,
requirementsfile: default=none,
trainentrypoint: default=none,
trainsourcefile: default=none,
pyversion: default=py3,
hostentrypoint: default=none,
hostsourcefile: default=none,
enablecloudwatchmetrics: default=false,
containerloglevel: default=200,

MXNET

mxnetversion: default=1.1,
trainentrypoint: default=none,
trainsourcefile: default=none,
pyversion: default=py3,
hostentrypoint: default=none,
hostsourcefile: default=none,
enablecloudwatchmetrics: default=false,
containerloglevel: default=200,

HPO

maxtrainingjobs: default=1,
maxparalleltrainingjobs: default=1,

Amazon Algorithms

algorithm:

aws-sagemaker-build's People

Contributors

Stargazers

Watchers

aws-sagemaker-build's Issues

Deployment fails

getTrainingConfig - this Lambda is failing.

2018-10-18T00:55:46.691Z 7ef43ae0-37df-4897-8d46-2184666fbfb8
{
"errorMessage": "TypeError: Cannot read property '1' of null",
"errorType": "Error",
"stackTrace": [
"exports.handler (/var/task/index.js:57:18)"
]
}

Handle Throttling

lambdas that call aws api should retry on throttle exceptions

InvalidParameterException: An error occurred - issue with SNS

aws-sagemaker-build/cloudformation-template/SageMakerNotebook/notebook.ipynb

Line 247 in 2cb4184

"print(\"subscribed to topic\")"

Error seen:

InvalidParameterException: An error occurred (InvalidParameter) when calling the Subscribe operation: Invalid SMS endpoint: x-xxx-xxx-xxxx

nodejs8.10 is no longer a supported runtime

Should be replaced with nodejs10.x

Support Multiple Training Jobs at the same time

add support for additional zip to upload to notebook instance

Rename from SageBuild-test-*

Can we rename the stack to something that does not contain "test" word?

Just a suggestion, as this trickles down to a few more places.

Error: Cannot find module './stepfunctions'

When running npm run up I receive the following error:

> [email protected] up /home/ubuntu/environment/aws-sagemaker-build
> make && ./bin/upload.sh && ./templates/main/bin/launch.js

mkdir -p build; mkdir -p build/lambda; mkdir -p build/templates
for l in templates/activo templates/main; do    \
        make -C $l;                     \
done;
make[1]: Entering directory '/home/ubuntu/environment/aws-sagemaker-build/templates/activo'
./bin/check.js 
StepFunction:CrawlerStart 372/4096
StepFunction:CrawlerStatus 414/4096
{ ResponseMetadata: { RequestId: '0b457362-9215-11e9-ba71-b70688c9a33c' },
  Parameters: 
   [ { ParameterKey: 'UiTemplateS3Uri', NoEcho: false },
     { ParameterKey: 'AssetPrefix', NoEcho: false },
     { ParameterKey: 'WorkteamArn', NoEcho: false },
     { ParameterKey: 'MaxConcurrentTaskCount', NoEcho: false },
     { ParameterKey: 'NumberOfHumanWorkersPerDataObject',
       NoEcho: false },
     { ParameterKey: 'TaskAvailabilityLifetimeInSeconds',
       NoEcho: false },
     { ParameterKey: 'PreHumanTaskLambdaArn', NoEcho: false },
     { ParameterKey: 'MaxPercentageOfInputDatasetLabeled',
       NoEcho: false },
     { ParameterKey: 'TaskTimeLimitInSeconds', NoEcho: false },
     { ParameterKey: 'MaxHumanLabeledObjectCount', NoEcho: false },
     { ParameterKey: 'AssetBucket', NoEcho: false } ],
  Description: '',
  Capabilities: [ 'CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND' ],
  CapabilitiesReason: 'The following resource(s) require capabilities: [AWS::IAM::Role, AWS::CloudFormation::Stack]',
  DeclaredTransforms: [] }
Resources: 15
make[1]: Leaving directory '/home/ubuntu/environment/aws-sagemaker-build/templates/activo'
make[1]: Entering directory '/home/ubuntu/environment/aws-sagemaker-build/templates/main'
rm build/notebooks.zip ; cd SageMakerNotebook/notebooks && zip -r ../../../../build/notebooks.zip .
rm: cannot remove 'build/notebooks.zip': No such file or directory
  adding: mxnet/ (stored 0%)
  adding: mxnet/input.html (deflated 65%)
  adding: mxnet/mxnet_mnist.ipynb (deflated 67%)
  adding: mxnet/mnist.py (deflated 61%)
  adding: amazon/ (stored 0%)
  adding: amazon/linear_learner_mnist.ipynb (deflated 67%)
  adding: tensorflow/ (stored 0%)
  adding: tensorflow/utils.py (deflated 59%)
  adding: tensorflow/setup.sh (deflated 61%)
  adding: tensorflow/mnist.py (deflated 66%)
  adding: tensorflow/daemon.json (deflated 42%)
  adding: tensorflow/tensorflow_distributed_mnist.ipynb (deflated 69%)
  adding: HyperParameterOptimization/ (stored 0%)
  adding: HyperParameterOptimization/hpo_mxnet_mnist.ipynb (deflated 67%)
  adding: HyperParameterOptimization/input.html (deflated 65%)
  adding: HyperParameterOptimization/mnist.py (deflated 62%)
./bin/check.js 
(node:3990) UnhandledPromiseRejectionWarning: Error: Cannot find module './stepfunctions'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/home/ubuntu/environment/aws-sagemaker-build/templates/main/step_function/stateMachines/DockerTrainDeploy/index.js:6:10)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at next (/home/ubuntu/environment/aws-sagemaker-build/templates/main/step_function/stateMachines/index.js:30:39)
    at Object.<anonymous> (/home/ubuntu/environment/aws-sagemaker-build/templates/main/step_function/stateMachines/index.js:18:32)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
(node:3990) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:3990) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
make[1]: Leaving directory '/home/ubuntu/environment/aws-sagemaker-build/templates/main'
make -C ./lambda
make[1]: Entering directory '/home/ubuntu/environment/aws-sagemaker-build/lambda'
for l in ./util; do \
        make -C $l;             \
done;
make[2]: Entering directory '/home/ubuntu/environment/aws-sagemaker-build/lambda/util'
echo "Building util"; cd nodejs && npm install && cp -r lib/* node_modules && cd .. && zip -r -q ../../build/lambda/util.zip .
Building util
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN [email protected] No description
npm WARN [email protected] No repository field.

added 20 packages from 75 contributors and audited 23 packages in 1.761s
found 0 vulnerabilities

make[2]: Leaving directory '/home/ubuntu/environment/aws-sagemaker-build/lambda/util'
make[1]: Leaving directory '/home/ubuntu/environment/aws-sagemaker-build/lambda'
bootstrap bucket is jasbarto-virginia-ml-bucket/artifacts/blog
upload: build/notebooks.zip to s3://jasbarto-virginia-ml-bucket/artifacts/blog/notebooks.zip
upload: build/lambda/util.zip to s3://jasbarto-virginia-ml-bucket/artifacts/blog/lambda/util.zip
console launch url:
https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?stackName=SageBuild&templateURL=http://s3.amazonaws.com/jasbarto-virginia-ml-bucket/artifacts/blog/templates/main.json&param_AssetBucket=jasbarto-virginia-ml-bucket&param_AssetPrefix=artifacts/blog&param_Username=Admin
(node:4070) UnhandledPromiseRejectionWarning: Error: Cannot find module './stepfunctions'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/home/ubuntu/environment/aws-sagemaker-build/templates/main/step_function/stateMachines/DockerTrainDeploy/index.js:6:10)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at next (/home/ubuntu/environment/aws-sagemaker-build/templates/main/step_function/stateMachines/index.js:30:39)
    at Object.<anonymous> (/home/ubuntu/environment/aws-sagemaker-build/templates/main/step_function/stateMachines/index.js:18:32)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
(node:4070) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 3)
(node:4070) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

add support for lambda deployments

Unable to pull TensorFlow Container

When deploying a TensorFlow training job with SageMaker Build, we get the error:

Failure reason
ClientError: Cannot pull algorithm container. Either the image does not exist or its permissions are incorrect.

The container ECR ARN is: 520713654638.dkr.ecr.eu-west-1.amazonaws.com/sagemaker-tensorflow:1.0.0-cpu-py2.

We tried pulling the container via docker CLI after successful ECR login with AWS CLI from shell, we get the error: Error response from daemon: manifest for 520713654638.dkr.ecr.eu-west-1.amazonaws.com/sagemaker-tensorflow:1.0.0-cpu-py2 not found.

Going through the documentation on pre-built containers, https://docs.aws.amazon.com/sagemaker/latest/dg/pre-built-containers-frameworks-deep-learning.html, it turns out that the account 520713654638 does not have container images in the region eu-west-1. However SageMaker TensorFlow container images are available in the account 763104351884 and we were able to pull the containers successfully via docker pull:

docker pull 763104351884.dkr.ecr.eu-west-1.amazonaws.com/tensorflow-training:1.14-gpu-py2

The would probably be here:

aws-sagemaker-build/lambda/util/nodejs/lib/CreateImageURI.js

Line 37 in 5a16995

var account='520713654638'

Thanks a lot for your help

Support for Spot Instances

@JohnCalhoun, with support training jobs on spot instances released yesterday (https://aws.amazon.com/blogs/aws/managed-spot-training-save-up-to-90-on-your-amazon-sagemaker-training-jobs/. Will this feature be added to SageMaker Build?

Thanks in advance

Add VPC support

npm run down fails

chzar:~/efs/C/John/aws-sagemaker-build (master) $ npm run down

[email protected] down /home/ec2-user/efs/C/John/aws-sagemaker-build
node ./sagebuild/bin/delete.js

SageBuild-test-1
Waiting on stack:SageBuild-test-1
✖ SageBuild-test-1:Stack with id SageBuild-test-1 does not exist
(node:9505) UnhandledPromiseRejectionWarning: ValidationError: Stack with id SageBuild-test-1 does not exist
at Request.extractError (/home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/protocol/query.js:47:29)
at Request.callListeners (/home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at Request.emit (/home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/home/ec2-user/efs/C/John/aws-sagemaker-build/node_modules/aws-sdk/lib/request.js:38:9)
(node:9505) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:9505) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Any simple sample for AWS Sagemaker

Hello,

I am a beginner with CloudFormation, I am trying to generate an instance of AWS Sagemaker via CloudFormation, do you have any simple template (to start) ?

Thank you,
Andre

add support for private link

Deploy Endpoint Multiple ProductionVariants Variants

AWS SageMaker Endpoint config supports multiple ProductionVariants. Is this a supported feature or something to expect in the future?

Thanks in advance

Add Tags Support

Use SageMaker SparkML Training and Serving

@JohnCalhoun, thanks for the great support.

Is there a possibility to train and deploy SageMaker SparkML models with SageMaker Build? And do you have any working examples I can use?

Thanks in advance

Unable to find config directory

When running npm run up the following results:

> [email protected] up /home/ubuntu/environment/aws-sagemaker-build
> make && ./bin/upload.sh && ./templates/main/bin/launch.js

mkdir -p build; mkdir -p build/lambda; mkdir -p build/templates
for l in templates/activo templates/main; do    \
        make -C $l;                     \
done;
make[1]: Entering directory '/home/ubuntu/environment/aws-sagemaker-build/templates/activo'
./bin/check.js 
module.js:550
    throw err;
    ^

Error: Cannot find module '../config'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/home/ubuntu/environment/aws-sagemaker-build/templates/activo/bin/aws.js:2:12)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
Makefile:3: recipe for target 'all' failed
make[1]: *** [all] Error 1
make[1]: Leaving directory '/home/ubuntu/environment/aws-sagemaker-build/templates/activo'
make[1]: Entering directory '/home/ubuntu/environment/aws-sagemaker-build/templates/main'
rm build/notebooks.zip ; cd SageMakerNotebook/notebooks && zip -r ../../../../build/notebooks.zip .
rm: cannot remove 'build/notebooks.zip': No such file or directory
updating: mxnet/ (stored 0%)
updating: mxnet/input.html (deflated 65%)
updating: mxnet/mxnet_mnist.ipynb (deflated 67%)
updating: mxnet/mnist.py (deflated 61%)
updating: amazon/ (stored 0%)
updating: amazon/linear_learner_mnist.ipynb (deflated 67%)
updating: tensorflow/ (stored 0%)
updating: tensorflow/utils.py (deflated 59%)
updating: tensorflow/setup.sh (deflated 61%)
updating: tensorflow/mnist.py (deflated 66%)
updating: tensorflow/daemon.json (deflated 42%)
updating: tensorflow/tensorflow_distributed_mnist.ipynb (deflated 69%)
updating: HyperParameterOptimization/ (stored 0%)
updating: HyperParameterOptimization/hpo_mxnet_mnist.ipynb (deflated 67%)
updating: HyperParameterOptimization/input.html (deflated 65%)
updating: HyperParameterOptimization/mnist.py (deflated 62%)
./bin/check.js 
module.js:550
    throw err;
    ^

Error: Cannot find module '../config'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/home/ubuntu/environment/aws-sagemaker-build/templates/main/bin/aws.js:2:12)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
Makefile:3: recipe for target 'all' failed
make[1]: *** [all] Error 1
make[1]: Leaving directory '/home/ubuntu/environment/aws-sagemaker-build/templates/main'
Makefile:21: recipe for target 'templates' failed
make: *** [templates] Error 2
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! [email protected] up: `make && ./bin/upload.sh && ./templates/main/bin/launch.js`
npm ERR! Exit status 2
npm ERR! 
npm ERR! Failed at the [email protected] up script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/ubuntu/.npm/_logs/2019-06-17T13_49_20_803Z-debug.log```

Race condition between inference image and training

All things being equal, after a few clean runs of the excellent utility, it seems that there is a race condition. Between inference image build and training. As the first invocation of build inference image, fails always. Second invocation after completing the training of the model, it succeeds. Possible race issue, but don’t have proof right now. But still worth looking at.

Script fails due to Node.js deprecated version

The CFN script receives an CREATE_FAILED du to Nodejs 6..10 deprectation for lambda:

The runtime parameter of nodejs6.10 is no longer supported for creating or updating AWS Lambda functions. We recommend you use the new runtime (nodejs8.10) while creating or updating functions. (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 219305e6-762d-11e9-876c-156d067a48cc)
  | 11:46:21 UTC+0200 | CREATE_FAILED | AWS::Lambda::Function | FailLambda

Therefore npm run up fails with the following:

SageBuild-test-1
Waiting on stack:SageBuild-test-1
✖ SageBuild-test-1:CREATE_FAILED
(node:50395) UnhandledPromiseRejectionWarning: CREATE_FAILED
(node:50395) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:50395) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

New SageMaker Step Functions Integration

With the news about native integration with Step Functions, https://aws.amazon.com/about-aws/whats-new/2019/10/aws-step-functions-expands-amazon-sagemaker-service-integration/, Will there be changes coming to SageMaker Build based on this news?

Unable to Pull SciKit Learn Image

When training a Scikit Learn model, I get this error: ClientError: Cannot pull algorithm container. Either the image does not exist or its permissions are incorrect..

On investigating I discovered that the image: 683313688378.dkr.ecr.eu-west-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3 does not exist in EU_WEST_1 region.

Further checks reveal that there are two ways to collect scikit learn images:

aws-sagemaker-build/lambda/util/nodejs/lib/CreateImageURI.js

Lines 9 to 28 in 5a16995

 exports.scikit=function(params){ 

 var account={ 

 "us-east-1":"746614075791", 

 "us-west-2":"246618743249", 

 "eu-west-1":"683313688378", 

 "us-east-2":"257758044811", 

 "us-gov-west-1":"414596584902", 

 "ap-northeast-1":"354813040037", 

 "ap-northeast-2":"366743142698", 

 "ap-south-1":"720646828776", 

 "ap-southeast-1":"121021644041", 

 "ap-southeast-2":"783357654285", 

 "ca-central-1":"341280168497", 

 "eu-central-1":"492215442770", 

 "eu-west-1":"141502667606", 

 "eu-west-2":"764974769150" 

 }[process.env.AWS_REGION] 

 var instance=params.traininstancetype.split('.')[1][0]==="p" ? "gpu" : "cpu" 

 return `${account}.dkr.ecr.${process.env.AWS_REGION}.amazonaws.com/sagemaker-chainer:${params.frameworkversion}-${instance}-${params.pyversion}` 

 }

aws-sagemaker-build/lambda/util/nodejs/lib/CreateImageURI.js

Lines 30 to 34 in 5a16995

 exports.scikit=function(params){ 

 var account='683313688378' 

 var instance=params.traininstancetype.split('.')[1][0]==="p" ? "gpu" : "cpu" 

 return `${account}.dkr.ecr.${process.env.AWS_REGION}.amazonaws.com/sagemaker-scikit-learn:${params.frameworkversion}-${instance}-${params.pyversion}` 

 }

However, #L9-L28 actually refers to chainer. Maybe some error there.

Would be great if you could help fix this.

Impressive work!!! 👍

add skip

add ability to skip steps

	exports.scikit=function(params){
	var account={
	"us-east-1":"746614075791",
	"us-west-2":"246618743249",
	"eu-west-1":"683313688378",
	"us-east-2":"257758044811",
	"us-gov-west-1":"414596584902",
	"ap-northeast-1":"354813040037",
	"ap-northeast-2":"366743142698",
	"ap-south-1":"720646828776",
	"ap-southeast-1":"121021644041",
	"ap-southeast-2":"783357654285",
	"ca-central-1":"341280168497",
	"eu-central-1":"492215442770",
	"eu-west-1":"141502667606",
	"eu-west-2":"764974769150"
	}[process.env.AWS_REGION]
	var instance=params.traininstancetype.split('.')[1][0]==="p" ? "gpu" : "cpu"
	return `${account}.dkr.ecr.${process.env.AWS_REGION}.amazonaws.com/sagemaker-chainer:${params.frameworkversion}-${instance}-${params.pyversion}`
	}

	exports.scikit=function(params){
	var account='683313688378'
	var instance=params.traininstancetype.split('.')[1][0]==="p" ? "gpu" : "cpu"
	return `${account}.dkr.ecr.${process.env.AWS_REGION}.amazonaws.com/sagemaker-scikit-learn:${params.frameworkversion}-${instance}-${params.pyversion}`
	}

aws-samples / aws-sagemaker-build Goto Github PK

aws-sagemaker-build's Introduction

AWS SageMaker Build

Prerequisites

setup

Stack Management

Architecture

The following diagram describes the flow of the Step Function StateMachine. There are several points where the StateMachine has to poll and wait for a task to complete.

The following diagram shows how the services work together

Parameters

Common

BYOD

TensorFlow

MXNET

HPO

Amazon Algorithms

aws-sagemaker-build's People

Contributors

Stargazers

Watchers

Forkers

aws-sagemaker-build's Issues

Recommend Projects

Recommend Topics

Recommend Org