This is a sample solution for bringing your own ML models and inference code, and running them at scale using AWS serverless services.

License: MIT No Attribution

JavaScript 2.96% Shell 12.58% Dockerfile 6.49% Python 25.67% TypeScript 52.30%

aws-serverless-for-machine-learning-inference's Introduction

Machine learning inference at scale using AWS serverless

This sample solution shows you how to run and scale ML inference using AWS serverless services: AWS Lambda and AWS Fargate. This is demonstrated using an image classification use case.

Architecture

The following diagram illustrates the solutions architecture for both batch and real-time inference options.

Deploying the solution

To deploy and run the solution, you need access to:

An AWS account
A terminal with AWS Command Line Interface (CLI), CDK, Docker, git, and Python installed.
- You may use the terminal on your local X86_64 machine or use an AWS Cloud9 environment (This code is validated on a m5.large AWS Cloud9 instance).

To deploy the solution, open your terminal window and complete the following steps.

Clone the GitHub repo
git clone https://github.com/aws-samples/aws-serverless-for-machine-learning-inference.git
Navigate to the /install directory and deploy the CDK application.
./install.sh
or
If using Cloud9:
./cloud9_install.sh
Enter Y to proceed with the deployment on the confirmation screen.

Running inference

The solution lets you get predictions for either a set of images using batch inference or for a single image at a time using real-time API end-point.

Batch inference

Get batch predictions by uploading image files to Amazon S3.

Upload one or more image files to the S3 bucket path, ml-serverless-bucket--/input, from Amazon S3 console or using AWS CLI.
aws s3 cp <path to jpeg files> s3://ml-serverless-bucket-<acct-id>-<aws-region>/input/ --recursive
This will trigger the batch job, which will spin-off Fargate tasks to run the inference. You can monitor the job status in AWS Batch console.
Once the job is complete (this may take a few minutes), inference results can be accessed from the ml-serverless-bucket--/output path

Real-time inference

Get real-time predictions by invoking the API endpoint with an image payload.

Navigate to the CloudFormation console and find the API endpoint URL (httpAPIUrl) from the stack output.
Use a REST client, like Postman or curl command, to send a POST request to the /predict api endpoint with image file payload.
curl -v -H "Content-Type: application/jpeg" --data-binary @<your jpg file name> <your-api-endpoint-url>/predict
Inference results are returned in the API response.

Cleaning up

Navigate to the /app directory from the terminal window and run the following command to destroy all resources and avoid incurring future charges.
cdk destroy -f

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.

aws-serverless-for-machine-learning-inference's People

Contributors

Stargazers

Watchers

Forkers

comeddy tkaden wq9 inquestgeronimo alangunning virginiasu sandy4321 nauw jmarqs27

aws-serverless-for-machine-learning-inference's Issues

Cloud9 Deployment running into errors

Hi again, I'm deploying the Cloud9 installation flow and I'm getting the following error:

`Deploying CDK Application
Try to get prebuilt lambda
Try to get prebuilt lambda
[Error at /MLServerlessStack/MLServerlessStack-base] You are not authorized to perform this operation.

Found errors`

This looks like permission errors (I could be wrong), is there a way to debug this to find out exactly where the error is? Because it seems I have full permissions for all AWS services mentioned in this repo, so this is a bit surprising. Thanks again.

can't get started

Hi recent changes were made to the directory structure and it seems that running installation via the current instructions in the readme doesn't work.

In addition, after I run ./install.sh, CLI states "Deploying CDK Application" and nothing else happens afterwards. The only thing that was created was a local image/container on my local machine's Docker. What are the next steps for getting this onto my AWS account for real-time inference?

In the root readme it states, Navigate to the CloudFormation console and find the API endpoint URL (httpAPIUrl) from the stack output. however I'm not able to find the API endpoint on my account's cloudformation. Thank you.

Type 'ConstructNode' is missing the following properties from type 'Node': _locked, _children, _context, _metadata, and 6 more.

Hello

I'm running ./cloud9_install.sh on Cloud9 and I get the following error:

Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
253 packages excluded due to repository priority protections
Package jq-1.5-1.amzn2.0.2.x86_64 already installed and latest version
Nothing to do
export ACCOUNT_ID=937224085622
export AWS_REGION=us-east-1
us-east-1
Checking CDK Status
Updating CDK

up to date, audited 487 packages in 2s

39 packages are looking for funding
run npm fund for details

found 0 vulnerabilities
Installing Typescript

up to date, audited 487 packages in 2s

39 packages are looking for funding
run npm fund for details

found 0 vulnerabilities
Installing Node Modules

up to date, audited 487 packages in 2s

39 packages are looking for funding
run npm fund for details

found 0 vulnerabilities
Creating SciPy and MXNet Lambda layer artifact
Creating layer compatible with python version 3.6
Requirement already satisfied: pip in /var/lang/lib/python3.6/site-packages (21.0)
Collecting pip
Downloading pip-21.3.1-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 21.0
Uninstalling pip-21.0:
Successfully uninstalled pip-21.0
Successfully installed pip-21.3.1
Collecting scipy==1.5.2
Downloading scipy-1.5.2-cp36-cp36m-manylinux1_x86_64.whl (25.9 MB)
Collecting mxnet==1.4.0
Downloading mxnet-1.4.0-py2.py3-none-manylinux1_x86_64.whl (29.6 MB)
Collecting numpy>=1.14.5
Downloading numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl (14.8 MB)
Collecting requests>=2.20.0
Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
Collecting graphviz<0.9.0,>=0.8.1
Downloading graphviz-0.8.4-py2.py3-none-any.whl (16 kB)
Collecting numpy>=1.14.5
Downloading numpy-1.14.6-cp36-cp36m-manylinux1_x86_64.whl (13.8 MB)
Collecting certifi>=2017.4.17
Downloading certifi-2022.6.15.1-py3-none-any.whl (160 kB)
Collecting idna<4,>=2.5
Downloading idna-3.3-py3-none-any.whl (61 kB)
Collecting charset-normalizer~=2.0.0
Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
Downloading urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests, numpy, graphviz, scipy, mxnet
Successfully installed certifi-2022.6.15.1 charset-normalizer-2.0.12 graphviz-0.8.4 idna-3.3 mxnet-1.4.0 numpy-1.14.6 requests-2.27.1 scipy-1.5.2 urllib3-1.26.12
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Done creating layer!
-rw-r--r-- 1 root root 74M Sep 13 00:06 layer.zip
/home/ec2-user/environment/aws-serverless-for-machine-learning-inference
Deploying CDK Application
/home/ec2-user/environment/aws-serverless-for-machine-learning-inference/node_modules/ts-node/src/index.ts:859
return new TSError(diagnosticText, diagnosticCodes, diagnostics);
^
TSError: ⨯ Unable to compile TypeScript:
lib/batch-resources.ts:87:7 - error TS2345: Argument of type 'this' is not assignable to parameter of type 'Construct'.
Type 'BatchResources' is not assignable to type 'Construct'.
Types of property 'node' are incompatible.
Type 'ConstructNode' is missing the following properties from type 'Node': _locked, _children, _context, _metadata, and 6 more.

87 this,
~~~~

at createTSError (/home/ec2-user/environment/aws-serverless-for-machine-learning-inference/node_modules/ts-node/src/index.ts:859:12)
at reportTSError (/home/ec2-user/environment/aws-serverless-for-machine-learning-inference/node_modules/ts-node/src/index.ts:863:19)
at getOutput (/home/ec2-user/environment/aws-serverless-for-machine-learning-inference/node_modules/ts-node/src/index.ts:1077:36)
at Object.compile (/home/ec2-user/environment/aws-serverless-for-machine-learning-inference/node_modules/ts-node/src/index.ts:1433:41)
at Module.m._compile (/home/ec2-user/environment/aws-serverless-for-machine-learning-inference/node_modules/ts-node/src/index.ts:1617:30)
at Module._extensions..js (node:internal/modules/cjs/loader:1180:10)
at Object.require.extensions.<computed> [as .ts] (/home/ec2-user/environment/aws-serverless-for-machine-learning-inference/node_modules/ts-node/src/index.ts:1621:12)
at Module.load (node:internal/modules/cjs/loader:1004:32)
at Function.Module._load (node:internal/modules/cjs/loader:839:12)
at Module.require (node:internal/modules/cjs/loader:1028:19) {

diagnosticCodes: [ 2345 ]
}

Subprocess exited with error 1

Editing lambda handler

Hi I'm trying to customize the lambda handler for real-time inference. However, If I alter the requirements.txt, deployment in cloudformation fails. For example, if I add boto3>=1.15.0 to the requirements.txt file deployment fails even though the local image boto3 installs just fine. Is the realtime inference stack sensitive to dependency changes?

Here is my requirements.txt (added boto3 at the end):

mxnet==1.6.0 numpy==1.19.5 urllib3==1.26.5 certifi==2022.12.7 idna==2.10 graphviz==0.8.4 chardet==3.0.4 boto3>=1.15.0

Is it possible for you to reproduce by editing the requirements.txt to see if it fails on your end as well? I'm using cdk version 2.59.0 (build b24095d) and using the ./install.sh flow.

FYI, if I try to deploy the default code/requirements it works fine. But if I alter the requirements file, deployment fails.

Thank you

aws-samples / aws-serverless-for-machine-learning-inference Goto Github PK

aws-serverless-for-machine-learning-inference's Introduction

Machine learning inference at scale using AWS serverless

Architecture

Deploying the solution

To deploy and run the solution, you need access to:

To deploy the solution, open your terminal window and complete the following steps.

Running inference

Batch inference

Real-time inference

Cleaning up

Security

License

aws-serverless-for-machine-learning-inference's People

Contributors

Stargazers

Watchers

Forkers

aws-serverless-for-machine-learning-inference's Issues

Recommend Projects

Recommend Topics

Recommend Org