kafka-docker-playground ![Build Status](https://camo.githubusercontent.com/35b108dd6e2e2d86a7e158ab84c0aa426aa5c25fec72f13457d5e732399b1ef9/68747470733a2f2f7472617669732d63692e636f6d2f7664657361626f752f6b61666b612d646f636b65722d706c617967726f756e642e7376673f6272616e63683d6d6173746572)
Playground for Kafka/Confluent Docker experimentations...
How to run
You just need to have docker and docker-compose
If you want to run it on EC2 Instance, you can use the AWS CloudFormation template provided here.
βΉοΈ By default Confluent Platform version 5.5.0 is used, but you can test with another version (greater or equal to 5.0.0) simply by exporting TAG
environment variable:
Example:
export TAG=5.3.2
Table of Contents
- How to run
- π Kafka Connectors
- βοΈ Confluent Cloud
- π Confluent Replicator and Mirror Maker 2
- π Environments
- Confluent Commercial
- CP-Ansible Playground
- πΎ Other Playgrounds
- π Useful Resources
π Kafka Connectors
Quick start examples from Confluent docs but in Docker version for ease of use.
βοΈ Source
Hadoop
AWS
S3
Kinesis
SQS
CloudWatch Logs
AWS Redshift (using JDBC)
Debezium
IBM MQ
Solace
ActiveMQ
TIBCO EMS
Syslog
JDBC
MQTT
JMS TIBCO EMS
InfluxDB
Splunk
RabbitMQ
Spool Dir
GCP
Couchbase
SFTP
MongoDB
Kudu
SNMP
ServiceNow
Data Diode
Azure
βοΈ Sink
Hadoop
AWS
Elasticsearch
HTTP
GCP
BigQuery
Cloud Functions
GCS
- using SASL_SSL
- using SSL
- using Kerberos GSSAPI
- using LDAP Authorizer SASL/PLAIN
- using RBAC environment SASL/PLAIN
Firebase
Spanner
Solace
Splunk
TIBCO EMS
IBM MQ
MQTT
InfluxDB
Cassandra
JDBC
ActiveMQ
OmniSci
JMS
Azure
Neo4j
Couchbase
SFTP
MongoDB
HBase
Redis
Kudu
Vertica
ServiceNow
Minio
Prometheus
Snowflake
βοΈ Confluent Cloud
Confluent Cloud Demo
- How to connect your components to Confluent Cloud
- How to monitor your Confluent Cloud cluster
- How to restrict access
- etc...
π Kafka Connectors connected to Confluent Cloud
ServiceNow source
ServiceNow sink
MongoDB source
Firebase
Other
- Using cp-ansible with Confluent Cloud
- Demo using dabz/ccloudexporter in order to pull Metrics API data from Confluent Cloud cluster and export it to Prometheus (Grafana dashboard is also available)
.NET client (producer/consumer)
Go client (producer/consumer)
kafka-admin Managing topics and ACLs using matt-mangia/kafka-admin
Confluent Replicator OnPrem to cloud and Cloud to Cloud examples
Multi-Cluster Schema Registry with hybrid configuration (onprem/confluent cloud)
- Confluent REST Proxy Security Plugin with Principal Propagation
- Migrate Schemas to Confluent Cloud using Confluent Replicator
π Confluent Replicator and Mirror Maker 2
Using Multi-Data-Center setup with US
πΊπΈ and EUROPE
πͺπΊ clusters.
Using Confluent Replicator as connector
- Using PLAINTEXT
- Using SASL_PLAIN
- Using Kerberos
- πΎ Using Confluent Replicator as executable
- Using PLAINTEXT
- Using SASL_PLAIN
- Using Kerberos
Using Mirror Maker 2
- Using PLAINTEXT
π Environments
Single cluster:
- PLAINTEXT: no security
- SASL_PLAIN: no SSL encryption, SASL/PLAIN authentication
- SASL/SCRAM no SSL encryption, SASL/SCRAM-SHA-256 authentication
- SASL_SSL: SSL encryption, SASL/PLAIN authentication
- 2WAY_SSL: SSL encryption, SSL authentication
- Kerberos: no SSL encryption, Kerberos GSSAPI authentication
- SSL_Kerberos SSL encryption, Kerberos GSSAPI authentication
- LDAP Authorizer with SASL/PLAIN no SSL encryption, SASL/PLAIN authentication, LDAP Authorizer for ACL authorization
- RBAC with SASL/PLAIN RBAC with no SSL encryption, SASL/PLAIN authentication
Multi-Data-Center setup:
- PLAINTEXT: no security
- SASL_PLAIN: no SSL encryption, SASL/PLAIN authentication
- Kerberos: no SSL encryption, Kerberos GSSAPI authentication
Confluent Commercial
- Control Center
- Tiered Storage
- Tiered storage with AWS S3
- Tiered storage with Minio (unsupported)
- Confluent Rebalancer
- JMS Client
- RBAC with SASL/PLAIN RBAC with no SSL encryption, SASL/PLAIN authentication
- Audit Logs
- Confluent REST Proxy Security Plugin with SASL_SSL and 2WAY_SSL Principal Propagation
CP-Ansible Playground
Easily play with Confluent Platform Ansible playbooks by using Ubuntu based Docker images generated daily from this cp-ansible-playground repository
There is also a Confluent Cloud version available here
πΎ Other Playgrounds
- Confluent Replicator [also with SASL_SSL and 2WAY_SSL]
- Testing Separate principals (
connector.client.config.override.policy
) for Source connector (SFTP source) - Testing Separate principals (
connector.client.config.override.policy
) for Source connector (SFTP sink) - How to write logs to files when using docker-compose
- Publish logs to kafka with Elastic Filebeat
.NET basic producer
Monitor Confluent Platform with Datadog
π Useful Resources
- A Kafka Story π: A step by step guide to use Kafka ecosystem (Kafka Connect, KSQL, Java Consumers/Producers, etc..) with Docker
- Kafka Boom Boom π₯: An attempt to break kafka
- Kafka Security playbook π: demonstrates various security configurations with Docker
- MDC and single views π: Multi-Data-Center setup using Confluent Replicator
- Kafka Platform Prometheus π: Simple demo of how to monitor Kafka Platform using Prometheus and Grafana.