This repository provides a template for running reinforcement learning algorithms that trains a ml-agents for a Unity environment on Amazon AWS Sagemaker service.

In addition, it offers a configurable script to generate a custom Docker container on which training algorithms will run as training jobs on AWS Sagemaker.

Here we collected some link to useful resources that can help you understand AWS and how to use the services.

• What is Amazon Web Services https://aws.amazon.com/what-is-aws
• AWS Account creation https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account
• AWS User creation https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html
• AWS Educational program https://aws.amazon.com/education/awseducate/
• AWS Sagemaker documentation https://docs.aws.amazon.com/sagemaker/latest/dg/whatis.html
• AWS Sagemaker Python SDK https://github.com/aws/sagemaker-python-sdk
• AWS S3 create bucket https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html
• AWS Identity and Access Management (IAM) guide https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html
• AWS Sagemaker notebook instance creation https://docs.aws.amazon.com/sagemaker/latest/dg/howitworks-create-ws.html

In addition, we provided further description of the different components in each sub-folder. Please check also them to get a complete overview.

• Create an AWS Account and carry out the initial security measures, such as creating a non-root user and granting them proper permissions (in our case it means full access to Sagemaker, S3 and ECR services).
• Open Sagemaker console and create a new notebook instance. During the process it might be useful to create a new role, which will be required to execute the training jobs. In addition, a S3 bucket is automatically created and associated to this notebook instance. In case of different needs, it is still possible to create another S3 bucket as container for Sagemaker input/output data, but it might require to configure the proper permissions.
  In case you already have created a repository for your project, during the creation of the notebook there is the possibility to associate it with the notebook environment. (More details are provided in the template folder description).
• If required, build and upload the custom Docker image tailored to allow TensorFlow interact with Unity environments (check the detailed description here).
• Exploit the template repository to implement your own training algorithm.
• Synch your repository project with the notebook environment and execute the job launcher.
• Once the training job is finished, you can collect the model and potential output data from S3 associated bucket.

Note: after executing the training job, if you do not require to run further job, please stop from Sagemaker console the notebook instance. This allows yoy to save computation hours (and costs) of the AWS instance.

In order to use your trained model for performing inference, that means choosing which action the agent should carry out, there are two possibilities:

• write a custom Python inferencing script, similar to the training one, that interface with Open AI Gym to send commands to the Unity environment.
• convert your trained model into a format that can be employed by Unity ML-Agents library.

In both cases the adopted Unity environment should be build for inferencing and not for training (unmark control checkbox in the Academy object).

Unity ML-Agents library adopted a particular model format: Barracuda. Therefore, in order to use your trained model it is necessary to convert it into the proper format. In the documentation of Unity ML-Agents is explained this procedure. The idea is to first convert your neural network model into either the TensorFlow checkpoint format (.pb) or in the Open Neural Network eXchange format (.onnx). Then it would be possible to use the converter script to easily obtain the proper model format.

Note: current Barracuda version shipped with Unity ML-Agents still does not support many features and the conversion of .onnx models. Therefore it is necessary to adopt Barracuda development version, which can be found here.