Giter Club home page Giter Club logo

keda-aci's Introduction

Bursting Jobs with KEDA

This project enables KEDA to spawn jobs into Azure Containter Instances (ACI).

Known Issues

There is a known issue with ACI containers not having access to MSI, more can be found here.

Requirements

Running

To run the following environment variables should be set first:

export RESOURCE_GROUP="<resource-group>"
export VNET_NAME="<vnet-name>"
export SP_APP_ID="<service-principal-app/client-id>"
export SP_SECRET="<service-pricipal-secret"
export STORAGE_ACCOUNT="<storage-account-name>"
export ACR="<azure-container-registry-name>"

1. Deploying the infrastrucuture

In the /deploy/ directory run:

$ ./deployinfra.sh

This will create the following resources:

  1. A Resource Group
  2. A VNET
  3. A Subnet on said VNET
  4. Azure Kubernetes Cluster
  5. Enables Virtual Nodes on AKS
  6. A Managed Service Identitiy (MSI)
  7. A Storage Account
  8. A Queue
  9. An Azure Container Registry
  10. Assigns roles to Service Principals and Managed Service Identity

NOTE: Role assignment may take a few minutes to propogate, you may need to run this script a couple of times

2. Push the Queue Consumer to the ACR

This builds and pushes the Queue Consumer to the Azure Container Registry

$ ./dockerpush.sh

3. Deploy AAD, KEDA, and Queue consumer to AKS

This will generate kubernetes yamls using helm template. This removes the requirement of Tiller. To run this:

$ ./deploytokubernetes

4. Push item to the Azure Queue

In the Azure portal navigate to your Storage Account with the queue you created. Click + to add a message to the queue.

--or--

$ az storage message get -q "queue" --auth-mode "login" --account-name $STORAGE_ACCOUNT

5. Destroying Kube templates

To destroy all running items on Kubernetes, run:

$ ./prepareforredeploy.sh

Note this may need to be run multiple times as there are some issues with KEDA crds and deleting. Run this script twice to ensure crds can be patched and deleted.

Queue Consumer

This is a simple python container that processes 1 message from a Queue at a time. KEDA will spin up 1 instance of the Queue Consumer per queue message. This should be replaced with a more robust message processor and is only used as a PoC.

This uses Managed Service Identities to retrieve PaaS resources without any secrets.

Two environment variables are required for this to run:

  1. STORAGE_ACCOUNT, the name of your Azure Storage Account
  2. QUEUE_NAME, the name of your Azure Queue.

Code can be found here

TODO

  1. Refactor deployinfra.sh
  2. Improve documentation

Contributing

Pull requests and issues are appreciated.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

keda-aci's People

Contributors

mapasco-msft avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.