Giter Club home page Giter Club logo

artifactory-sagemaker's Introduction

Artifactory-Sagemaker

Demo to implement the integration between JFrog Artifactory and AWS Sagemaker

Directory Contents

dev

Includes files related to Notebooks created within SageMaker or launched from SageMaker console


Notebook subdirectory - for use when launching a notebook instance from the SageMaker console

  • lifecycle_configuration.sh

    Code of the lifecycle configuration script to launch a sagemaker Notebook instance


Studio subdirectory - for use when working within SageMaker Studio Classic

  • example.ipynb

    Jupyter notebook with commands to demonstrate the following Artifactory and Sagemaker integrations after creating the notebook in an environment started with the lifecycle configuration script:

    1. JFrog CLI
    2. Artifactory PyPI virtual repository
    3. Artifactory HuggingFace ML remote repository
  • lifecycle_configuration.sh

    Code of the lifecycle configuration script to launch a sagemaker studio lab IDE


inference

Includes files related to deploying a model for real-time inference including retrieving a custom model from an Artifactory HuggingFace ML repository and creating a custom Docker image to launch with the SageMaker inference service


sagemaker subdirectory

  • Dockerfile

    Dockerfile to build a custom image for inference

    Example docker commands to log in to the Artifactory Docker registry and then to build and push the inference image to Artifactory:

    docker login myartifactory.jfrog.io -u <USER> -p <TOKEN>
    
    docker build -t \
      myartifactory.jfrog.io/sagemaker-docker-virtual/sagemaker/inference-service:1.0_huggingface \
      --build-arg "ARTIFACTORY_DOCKER_REGISTRY=myartifactory.jfrog.io" \
      --build-arg "ARTIFACTORY_DOCKER_REPO=sagemaker-docker-virtual" .
    
    docker push myartifactory.jfrog.io/sagemaker-docker-virtual/sagemaker/inference-service:1.0_huggingface
    
  • entrypoint.py

    Entrypoint script used in the custom Docker image to set up the inference handler service

  • inference.py

    Code included in the custom Docker image to handle inference using a custom Hugging Face ML model


  • deploy-model.py

    Code to deploy the trained model as a SageMaker real-time inference

  • test-inference.py

    Code to test the deployed SageMaker real-time inference


infrastructure

Includes files related to setting up the AWS secret and Lambda required for the use of Artifactory within a SageMaker VPC environment


  • deploy-lambda.sh
  • deploy-secret.sh
  • lambda_function.py

train

Includes files related to creating and running a custom Docker image to train and store a model in Artifactory


  • Dockerfile

    Dockerfile to build a model train image

    Example docker commands to log in to the Artifactory Docker registry and then to build and push the training image to Artifactory:

    docker login myartifactory.jfrog.io -u <USER> -p <TOKEN>
    
    docker build -t \
      myartifactory.jfrog.io/sagemaker-docker-virtual/sagemaker/train:1.0_huggingface \
      --secret id=pipconfig,src=pip.conf \
      --build-arg "ARTIFACTORY_DOCKER_REGISTRY=myartifactory.jfrog.io" \
      --build-arg "ARTIFACTORY_DOCKER_REPO=sagemaker-docker-virtual" .
    
      docker push myartifactory.jfrog.io/sagemaker-docker-virtual/sagemaker/train:1.0_huggingface
    
  • pip.conf

    File used to configure the Artifactory PyPI repository for use during the Python package installation section of the Docker image build (used as a Docker build secret)

  • requirements.txt

    Python package requirements for use during the custom Docker image build

    These requirements satisfy everything needed for both the train.py and inference.py scripts (both are combined to simplify the demo)

  • run-train-job.py

    Code to create and execute SageMaker train job using a custom Docker image

  • train.py

    Model train code that the SageMaker train job will execute

artifactory-sagemaker's People

Contributors

haizafrog avatar mjmckay avatar tomerm12 avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

Forkers

pavanvikas

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.