aws-samples / kda-flink-app-autoscaling Goto Github PK

🚨 August 30, 2023: Amazon Kinesis Data Analytics has been renamed to Amazon Managed Service for Apache Flink.

This sample is meant to help users auto-scale their Kinesis Data Analytics for Java (KDA) applications using AWS Application Autoscaling. KDA currently only supports CPU based autoscaling, and readers can use the guidance in this repo to scale their KDA applications based on other signals - such as operator throughput, for instance.

We've included guidance for both step scaling and target tracking scaling. For official documentation on AWS Application Autoscaling, please visit:

• Step scaling
• Target tracking scaling

You may be wondering: "Why use Application Autoscaling; why not just trigger a Lambda function via a CloudWatch alarm and SNS?". The main reason is that Application Autoscaling has a well defined API for specifying scaling policies and associated attributes such as cooldown periods. In addition, we can take advantage of all three scaling types included with Application Autoscaling: step scaling, target tracking scaling, and schedule-based scaling (not covered in this doc).

AWS Application autoscaling allows users to scale in/out custom resources by specifying a custom endpoint that can be invoked by Application Autoscaling. In this example, this custom endpoint is implemented using API Gateway and an AWS Lambda function. Here's a high level flow depicting this approach:

CW Alarm => Application Autoscaling => Custom Endpoint (API GW + Lambda) => Scale KDA App

The accompanying CloudFormation template takes care of provisioning all of the above components.

When invoked by Application Autoscaling, the Lambda function (written in Python) will call UpdateApplication with the desired capacity specified. In addition, it will also re-configure the alarm thresholds to take into account the current parallelism of the KDA application.

You can review the Python code for the Lambda function associated with step scaling here and for target tracking autoscaling here.

1. When scaling out/in, in this sample we only update the overall parallelism; we don't adjust parallelism/KPU.
2. When scaling occurs, the KDA app experiences downtime. Please take this into consideration when configuring the step scaling increments.
3. Please keep in mind that the throughput of a Flink application is dependent on many factors (complexity of processing, destination throughput, etc...). The step-scaling example assumes a simple relationship between incoming record throughput and scaling. And similarly millisBehindLatest for target-tracking autoscaling.