kafka-docker-playground

Playground for Kafka/Confluent Docker experimentations...

How to run

You just need to have docker and docker-compose

If you want to run it on EC2 Instance, you can use the AWS CloudFormation template provided here.

ℹ️ By default Confluent Platform version 5.5.0 is used, but you can test with another version (greater or equal to 5.0.0) simply by exporting TAG environment variable:

Example:

export TAG=5.3.2

Table of Contents

• How to run
• 🔗 Kafka Connectors
  • ↘️ Source
  • ↗️ Sink
• ☁️ Confluent Cloud
  • Confluent Cloud Demo
  • 🔗 Kafka Connectors connected to Confluent Cloud
  • Other
• 🔄 Confluent Replicator and Mirror Maker 2
• 🔐 Environments
• Confluent Commercial
• CP-Ansible Playground
• 👾 Other Playgrounds
• 📚 Useful Resources

🔗 Kafka Connectors

Quick start examples from Confluent docs but in Docker version for ease of use.

↘️ Source

• Hadoop
  • HDFS 2
  • HDFS 3
• AWS
  • S3
  • Kinesis
  • SQS
    • using SASL_SSL
    • using SSL
  • CloudWatch Logs
  • AWS Redshift (using JDBC)
• Debezium
  • MySQL
  • PostgreSQL
  • MongoDB
  • SQL Server
• IBM MQ
• Solace
• ActiveMQ
• TIBCO EMS
• Syslog
• JDBC
  • MySQL
  • Oracle 11
  • Oracle 12
  • PostGreSQL
  • SQL Server
  • AWS Redshift
• MQTT
• JMS TIBCO EMS
• InfluxDB
• Splunk
• RabbitMQ
• Spool Dir
• GCP
  • Pub/Sub
  • GCS
  • Firebase
• Couchbase
• SFTP
• MongoDB
• Kudu
• SNMP
• ServiceNow
• Data Diode
• Azure
  • Blob Storage
  • Event Hubs
  • Service Bus

↗️ Sink

• Hadoop
  • HDFS 2
  • HDFS 3
• AWS
  • S3
  • Redshift
  • DynamoDB
  • Lambda
  • CloudWatch Metrics
• Elasticsearch
• HTTP
• GCP
  • BigQuery
  • Cloud Functions
  • GCS
    • using SASL_SSL
    • using SSL
    • using Kerberos GSSAPI
    • using LDAP Authorizer SASL/PLAIN
    • using RBAC environment SASL/PLAIN
  • Firebase
  • Spanner
• Solace
• Splunk
• TIBCO EMS
• IBM MQ
• MQTT
• InfluxDB
• Cassandra
• JDBC
  • MySQL
  • Oracle 11
  • Oracle 12
  • PostGreSQL
  • SQL Server
  • Vertica
  • Hive
• ActiveMQ
• OmniSci
• JMS
  • ActiveMQ
  • Solace
  • TIBCO EMS
• Azure
  • Blob Storage
  • Data Lake Gen1
  • Data Lake Gen2
  • SQL Data Warehouse
  • Search
• Neo4j
• Couchbase
• SFTP
• MongoDB
• HBase
• Redis
• Kudu
• Vertica
• ServiceNow
• Minio
• Prometheus
• Snowflake

☁️ Confluent Cloud

Confluent Cloud Demo

• How to connect your components to Confluent Cloud
• How to monitor your Confluent Cloud cluster
• How to restrict access
• etc...

🔗 Kafka Connectors connected to Confluent Cloud

• ServiceNow source
• ServiceNow sink
• MongoDB source
• Firebase

Other

• Using cp-ansible with Confluent Cloud
• Demo using dabz/ccloudexporter in order to pull Metrics API data from Confluent Cloud cluster and export it to Prometheus (Grafana dashboard is also available)
• .NET client (producer/consumer)
• Go client (producer/consumer)
• kafka-admin Managing topics and ACLs using matt-mangia/kafka-admin
• Confluent Replicator OnPrem to cloud and Cloud to Cloud examples
• Multi-Cluster Schema Registry with hybrid configuration (onprem/confluent cloud)
• Confluent REST Proxy Security Plugin with Principal Propagation
• Migrate Schemas to Confluent Cloud using Confluent Replicator

🔄 Confluent Replicator and Mirror Maker 2

Using Multi-Data-Center setup with US 🇺🇸 and EUROPE 🇪🇺 clusters.

• Using Confluent Replicator as connector
  • Using PLAINTEXT
  • Using SASL_PLAIN
  • Using Kerberos
• 👾 Using Confluent Replicator as executable
  • Using PLAINTEXT
  • Using SASL_PLAIN
  • Using Kerberos
• Using Mirror Maker 2
  • Using PLAINTEXT

🔐 Environments

Single cluster:

• PLAINTEXT: no security
• SASL_PLAIN: no SSL encryption, SASL/PLAIN authentication
• SASL/SCRAM no SSL encryption, SASL/SCRAM-SHA-256 authentication
• SASL_SSL: SSL encryption, SASL/PLAIN authentication
• 2WAY_SSL: SSL encryption, SSL authentication
• Kerberos: no SSL encryption, Kerberos GSSAPI authentication
• SSL_Kerberos SSL encryption, Kerberos GSSAPI authentication
• LDAP Authorizer with SASL/PLAIN no SSL encryption, SASL/PLAIN authentication, LDAP Authorizer for ACL authorization
• RBAC with SASL/PLAIN RBAC with no SSL encryption, SASL/PLAIN authentication

Multi-Data-Center setup:

• PLAINTEXT: no security
• SASL_PLAIN: no SSL encryption, SASL/PLAIN authentication
• Kerberos: no SSL encryption, Kerberos GSSAPI authentication

Confluent Commercial

• Control Center
  • Control Center in "Read-Only" mode
  • Configuring Control Center with LDAP authentication
• Tiered Storage
  • Tiered storage with AWS S3
  • Tiered storage with Minio (unsupported)
• Confluent Rebalancer
• JMS Client
• RBAC with SASL/PLAIN RBAC with no SSL encryption, SASL/PLAIN authentication
• Audit Logs
• Confluent REST Proxy Security Plugin with SASL_SSL and 2WAY_SSL Principal Propagation

CP-Ansible Playground

Easily play with Confluent Platform Ansible playbooks by using Ubuntu based Docker images generated daily from this cp-ansible-playground repository

There is also a Confluent Cloud version available here

👾 Other Playgrounds

• Confluent Replicator [also with SASL_SSL and 2WAY_SSL]
• Testing Separate principals (connector.client.config.override.policy) for Source connector (SFTP source)
• Testing Separate principals (connector.client.config.override.policy) for Source connector (SFTP sink)
• How to write logs to files when using docker-compose
• Publish logs to kafka with Elastic Filebeat
• .NET basic producer
• Monitor Confluent Platform with Datadog

📚 Useful Resources

• A Kafka Story 📖: A step by step guide to use Kafka ecosystem (Kafka Connect, KSQL, Java Consumers/Producers, etc..) with Docker
• Kafka Boom Boom 💥: An attempt to break kafka
• Kafka Security playbook 🔒: demonstrates various security configurations with Docker
• MDC and single views 🌍: Multi-Data-Center setup using Confluent Replicator
• Kafka Platform Prometheus 📊: Simple demo of how to monitor Kafka Platform using Prometheus and Grafana.