smwshop

Build the Apache MXNet deployment package for AWS Lambda

Launch an Amazon EC2 instance with the proper AMI

In order to compile MXNet for the Lambda function, you would need an environment running the AMI that the Lambda service is using. The current version is:

amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2

double-checking that the current version is still the one above by visiting https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html.

You will have to launch an Amazon EC2 instance with that AMI and then access it by using an SSH client.

Install the required libraries

First, install development tools and the libraries required to compile MXNet. Apache MXNet requires several libraries as prerequisites. In this tutorial we will be using:

• Atlas (Linear Algebra Library) http://math-atlas.sourceforge.net/
• OpenBlas (Basic Linear Algebra Subprograms) https://www.openblas.net/
• Lapack (Linear Algebra Package) http://www.netlib.org/lapack/

We are NOT going to use OpenCV, required by MXNet for image processing functions, and NVIDIA CUDA / cuDNN since we do not need to run Apache MXNet with GPU support.

If you want to know more on the dependencies and options to build Apache MXNet from sources, please visit https://mxnet.incubator.apache.org/install/build_from_source.html.

Execute the following commands in your home folder:

sudo yum groupinstall -y "Development Tools" && sudo yum install -y git

sudo yum install atlas-devel

sudo yum install openblas openblas-devel.x86_64 lapack-devel.x86_64

Build Apache MXNet

We are now ready to build Apache MXNet from sources. We are going to get the sources from the GitHub repository where MXNet is maintained and then build with the proper settings.

git clone --recursive https://github.com/apache/incubator-mxnet mxnet
cd mxnet

make -j $(nproc) USE_OPENCV=0 USE_CUDNN=0 USE_CUDA=0 USE_BLAS=openblas USE_LAPACK=1

Install language bindings

When the build process is completed, we will need to install the Python bindings as we want to use Python as programming language. All language bindings supported by MXNet are configured after building the framework, except C++ that needs to be included during the compilation.

To install language bindings, execute the following commands:

cd python

sudo python setup.py install

Build the package

Once the bindings are installed, we are ready to build a package that will contain all the relevant libraries to run MXNet in Lambda.

Please note that these instructions assume using the latest version of Apache MXNet and Numpoy, which, at the time being, are 1.3.1 and 1.15.2. This means that, if a new version is released, you might have to update the paths below.

cd

mkdir mxnetpackage

cp -r /usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet mxnetpackage/

cd mxnetpackage

sudo pip install numpy

cp -r /usr/local/lib/python2.7/site-packages/numpy-1.15.2-py2.7-linux-x86_64.egg/numpy/ .

mkdir lib

cp /usr/lib64/atlas/libatlas.so.3 lib/

cp /usr/lib64/atlas/libcblas.so.3 lib/

cp /usr/lib64/atlas/libclapack.so.3 lib/

cp /usr/lib64/atlas/libf77blas.so.3 lib/

cp /usr/lib64/libgfortran.so.3 lib/

cp /usr/lib64/libgfortran.so.3.0.0 lib/

cp /usr/lib64/libgomp.so.1 lib/

cp /usr/lib64/libgomp.so.1.0.0 lib/

cp /usr/lib64/atlas/liblapack.so.3 lib/

cp /usr/lib64/libopenblas.so.0 lib/

cp /usr/lib64/atlas/libptcblas.so.3 lib/

cp /usr/lib64/atlas/libptf77blas.so.3 lib/

cp /usr/lib64/libquadmath.so.0 lib/

Save the package

We are now going to compress and save the package to Amazon S3 as the final step.

tar cfz mxnet.tar.gz *

aws s3 cp mxnet.tar.gz s3://{your_bucket}/

Now you can run a test by creating an AWS Lambda function starting from the package content and write a simple Python lambda handler that executes some basic MXNet code.

Finally, after ensuring the package works as expected, you can terminate the EC2 instance to avoid incurring in unexpected charges.