This is a tiny project to try out OCI Streams, using IaC with Terraform and consuming the streams with a python client, also using a kafka-connect.

Stacks:

• Kafka
  
• Python Kafka Library
  
• OCI Streams
  
• Terraform
  
• Docker
  
• Docker Compose
  
• Kafka Connect
  

• Infrastructure

  • Copy your private key (.pem format) from your OCI account into a file called private_key.pem inside etc/deployment;

  • Create the file terraform.tfvars inside etc/deployment with the required variables, for example;

  tenancy_ocid         = ""
  user_ocid            = ""
  user_key_fingerprint = ""
  tags                 = { "ENVIRONMENT" = "DEV", "MANAGED_BY" = "Terraform" }

  • Use the Makefile to create the infrastructure:

  make deploy MYSQL_PASS=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 16 | head -n 1)

• Kafka Connect

  • Use the Makefile to run the kafka connect:

  make run-kafka-connect

• Insert Random Data

  • Use the Makefile to run to insert random data:

  make run-mysql-insert-data

• Consumer Client

  • Use the Makefile to run the consumer client:

  make run-consumer-client

• Clean up

  • Use the Makefile to up the clean the resources from OCI and local containers:

  make destroy