• Introduction
• Architecture
• Prerequisites
  • Tools
  • Versions
• Deployment
  • Authenticate gcloud
  • Configure gcloud settings
  • Setup this project
  • Provisioning the compute instances
• Tear Down
• Troubleshooting
• Relevant Material

This guide demonstrates how to create a kafka cluster with default size 3X3 also this will guide us on setting security for the cluster using kerberos and ssl.

1. Kafka nodes - 3
2. zookeeper nodes - 3
3. schema registry - 1
4. administrator node - 1
5. kerberos node - 1

1. Terraform >= 0.11.10
2. Google Cloud SDK version >= 229.0.0
3. bash or bash compatible shell
4. GNU Make 3.x or later
5. A Google Cloud Platform project where you have permission to create networks

The Google Cloud SDK is used to interact with your GCP resources. Installation instructions for multiple platforms are available online.

Terraform is used to automate the manipulation of cloud infrastructure. Its installation instructions are also available online.

The steps below will walk you through using terraform to deploy a Compute Engine cluster that you will then use for working with Kafka.

Prior to running this demo, ensure you have authenticated your gcloud client by running the following command:

gcloud auth application-default login

Run gcloud config list and make sure that compute/zone, compute/region and core/project are populated with values that work for you. You can set their values with the following commands:

# Where the region is us-east1
gcloud config set compute/region us-east1

Updated property [compute/region].

# Where the zone inside the region is us-east1-c
gcloud config set compute/zone us-east1-c

Updated property [compute/zone].

# Where the project name is my-project-name
gcloud config set project my-project-name

Updated property [core/project].

This project requires the following Google Cloud Service APIs to be enabled:

• compute.googleapis.com

In addition, the terraform configuration takes three parameters to determine where the compute instances should be created:

• project
• region
• zone

For simplicity, these parameters are to be specified in a file named terraform.tfvars, in the terraform directory. To ensure the appropriate APIs are enabled and to generate the terraform/terraform.tfvars file based on your gcloud defaults, run:

cat terraform/terraform.tfvars

project="YOUR_PROJECT"
region="YOUR_REGION"
zone="YOUR_ZONE"

If you need to override any of the defaults, simply replace the desired value(s) to the right of the equals sign(s). Be sure your replacement values are still double-quoted.

Next, apply the terraform configuration with:

# From within the project root, use make to apply the terraform plan
cd terraform \
   && terraform init && \
   && terraform apply --auto-approve

# From within the project root, use make to destroy the terraform plan.
cd terraform \
   && terraform destory --auto-approve

The credentials that Terraform is using do not provide the necessary permissions to create resources in the selected projects. Ensure that the account listed in gcloud config list has necessary permissions to create resources. If it does, regenerate the application default credentials using gcloud auth application-default login.

• Terraform Google Provider
• Apache Kafka
• Apache Zookeeper
• Prometheus
• Grafana
• Kerberos