As you might know, the Kafka protocol is such that clients have to be able to address individual kafka nodes after connecting to the bootstrap servers and getting the metadata which contain the nodes list and the partition assignments. Also, since consumers typically read from the leader (exclusively before Apache Kafka 2.3), that is another reason that load balancing Kafka is not possible. However, we can setup a reverse proxy to front a Kafka cluster as long as we can allow individual nodes to be addressable. The following setup gives one such example using nginx and using a separate advertised port per broker to route the requests to the right broker

> docker-compose up

This should install a 3 node cluster and an nginx proxy advertising kafka brokers at localhost:9092, localhost:9093, localhost:9094

> kafkacat -L -b localhost:9092
> kafkacat -L -b localhost:9093
> kafkacat -L -b localhost:9094

> kafkacat -P -b localhost:9092 -t new_topic
1
2
3
4
5

> kafkacat -C -b localhost:9092 -t new_topic