This demo demonstrates how cloud functions can produce and consume from Confluent Cloud. The current examples are with AWS Lambda and are baked by Terraform.
- Check if you have a Confluent Cloud Account
- Create Confluent Cloud API keys
https://registry.terraform.io/providers/confluentinc/confluent/latest/docs/guides/sample-project#create-a-cloud-api-key - Check if you have Terraform installed
terraform -version
https://learn.hashicorp.com/tutorials/terraform/install-cli#install-terraform - Check if you have the Confluent CLI installed
confluent version
https://docs.confluent.io/confluent-cli/current/install.html - Check if you have the AWS CLI installed
aws --version
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html - Optional: Create a Openweathermap Key for the OneCall API: https://openweathermap.org/api
- Optional: Deploy standalone self-managed REST Proxy that talks to Confluent Cloud on Elastic Container Service: ./aws-ecs-rest-proxy-to-confluent-cloud/deploy-rest-proxy-on-ecs.sh This script deploys ECR, ECS and few other AWS resources to use REST calls by Cloud Functions or curl to interact with the Confluent Cloud Cluster.
- Clone the repo and enter into the directory
git clone https://github.com/senadjukic/serverless-cloud-functions-with-confluent-cloud.git \
&& cd serverless-cloud-functions-with-confluent-cloud
- Create the terraform.tfvars with your credentials
cat > $PWD/confluent-cloud-cluster/terraform.tfvars <<EOF
confluent_cloud_api_key = "(see Confluent Cloud settings)"
confluent_cloud_api_secret = "(see Confluent Cloud settings)"
environment_name = "(your-name)-cloud-functions"
cluster_name = "aws-basic"
topic_name = "temperature"
aws_access_key_id = "(see AWS IAM)"
aws_secret_access_key = "(see AWS IAM)"
lambda_sink_function_name = "Connector_Sink_Lambda_Function"
EOF
cat > $PWD/aws-lambda-producer-to-confluent-cloud/terraform.tfvars <<EOF
lambda_sink_function_name = "(your-name)_Producer_to_Confluent_Cloud_Lambda_Function"
owner_email = "([email protected])"
EOF
cat > $PWD/aws-lambda-sink-connector-invocation/terraform.tfvars <<EOF
lambda_sink_function_name = "(your-name)_Connector_Sink_Lambda_Function"
owner_email = "([email protected])"
EOF
terraform -chdir=confluent-cloud-cluster/ initterraform -chdir=confluent-cloud-cluster/ apply -auto-approve- Create the input file for the Python variables
cat > $PWD/aws-lambda-producer-to-confluent-cloud/python/env.py <<EOF
env_topic_name = "$(terraform output -raw topic_name)"
env_cluster_bootstrap_endpoint = "$(terraform output -raw cluster_bootstrap_endpoint | cut -c 12-)"
env_producer_kafka_api_key = "$(terraform output -raw producer_kafka_api_key)"
env_producer_kafka_api_secret = "$(terraform output -raw producer_kafka_api_secret)"
env_topic_name = "$(terraform output -raw topic_name)"
EOF
- Package Python packages for AWS Lambda deployment
pip3 install --target $PWD/aws-lambda-producer-to-confluent-cloud/python/ confluent_kafka
pip3 install --target $PWD/aws-lambda-sink-connector-invocation/python/ requests
-
Optional: Add your Openweather API key
echo 'env_openweather_key = "(your openweather key)"' > $PWD/aws-lambda-sink-connector-invocation/python/env.py -
terraform -chdir=aws-lambda-producer-to-confluent-cloud/ init -
terraform -chdir=aws-lambda-producer-to-confluent-cloud/ apply -auto-approve -
terraform -chdir=aws-lambda-sink-connector-invocation/ init -
terraform -chdir=aws-lambda-sink-connector-invocation/ apply -auto-approve
How to see the records in the CLI?
confluent kafka topic consume -b temperature
How to invoke the functions from AWS CLI?
export RANDOM_TEMPERATURE=$(( RANDOM % 10 ))
echo $RANDOM_TEMPERATURE
aws lambda invoke \
--function-name Producer_to_Confluent_Cloud_Lambda_Function \
--payload '{"temperature_guess":"'"${RANDOM_TEMPERATURE}"'"}' \
/dev/stdout | cat
Example value:
aws lambda invoke \
--function-name Producer_to_Confluent_Cloud_Lambda_Function \
--payload '{"temperature_guess":"25.5"}' \
/dev/stdout | cat
Via AWS CLI:
aws lambda delete-function --function-name Producer_to_Confluent_Cloud_Lambda_Function
aws lambda delete-function --function-name Connector_Sink_Lambda_Function
Via Terraform:
terraform -chdir=aws-lambda-producer-to-confluent-cloud/ apply -auto-approve -destroy
terraform -chdir=aws-lambda-sink-connector-invocation/ apply -auto-approve -destroy
terraform -chdir=confluent-cloud-cluster/ apply -auto-approve -destroy
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent