Comments (4)
Hi,
you can configure such topology configuring proxy01, proxy02 and proxy03 using bootstrap-server-mapping
and external-server-mapping
e.g.
proxy01:
kafka-proxy server --bootstrap-server-mapping "b1-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32500,proxy01:32500" \
--bootstrap-server-mapping "b2-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32501,proxy01:32501" \
--bootstrap-server-mapping "b3-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32502,proxy01:32502" \
--bootstrap-server-mapping "b4-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32503,proxy01:32503" \
--external-server-mapping "b5-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy02:32500" \
--external-server-mapping "b6-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy02:32501" \
--external-server-mapping "b7-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy02:32502" \
--external-server-mapping "b8-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy02:32503" \
--external-server-mapping "b9-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32500" \
--external-server-mapping "b10-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32501" \
--external-server-mapping "b11-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32502" \
--external-server-mapping "b12-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32503" \
--dynamic-listeners-disable
proxy02:
kafka-proxy server --external-server-mapping "b1-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy01:32500" \
--external-server-mapping "b2-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy01:32501" \
--external-server-mapping "b3-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy01:32502" \
--external-server-mapping "b4-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy01:32503" \
--bootstrap-server-mapping "b5-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32500,proxy02:32500" \
--bootstrap-server-mapping "b6-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32501,proxy02:32501" \
--bootstrap-server-mapping "b7-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32502,proxy02:32502" \
--bootstrap-server-mapping "b8-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32503,proxy02:32503" \
--external-server-mapping "b9-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32500" \
--external-server-mapping "b10-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32501" \
--external-server-mapping "b11-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32502" \
--external-server-mapping "b12-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32503" \
--dynamic-listeners-disable
proxy03:
....
from kafka-proxy.
You can also put multiple proxies with the same configuration behind a load balancer
e.g.
- LB DNS proxy01 load balancing to multpile instances with proxy01 config
- LB DNS proxy02 load balancing to multpile instances with proxy02 config
- ....
from kafka-proxy.
Hi @everesio , thank you for your reply! Why do we use external-server-mapping here --external-server-mapping "b5-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy02:32500"
. Will it work if I use --dial-address-mapping "b5-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy02:32500"
?
from kafka-proxy.
bootstrap-server-mapping
starts a TLS listener and changes/remaps the kafka broker addresses in the response. external-server-mapping
only changes the kafka broker addresses
from kafka-proxy.
Related Issues (20)
- [Question] Can I attach 3 bootstrap server endpoints to a single port? HOT 1
- [Question] If my Kafka brokers are running version 2.8.1, should I be using kafka-proxy version 0.2.9? HOT 1
- "Metadata" request (ApiKey=3 and ApiVersion=5) in the Kafka Proxy is not following the protocol structure defined by Kafka protocol guide HOT 1
- [Question] is there a plan to release a Java implementation of Kafka Proxy ? HOT 1
- [Need Help] Sending Custom METADATA response through Kafka Proxy
- [Question] is there a plan to support HTTPS proxy ?
- will there be an update to resovle 7 vulnerabilitys
- tls: failed to parse private key AWS MSK HOT 6
- bad performance when executing kafka-producer-perf-test.sh HOT 3
- Can not use grepplabs/kafka-proxy ARM image as base image HOT 1
- one port mapping to 6 broker HOT 2
- AWS Invalid API Key. What did I miss ?
- tls: failed to parse private key HOT 2
- Can't get proxy-listener tls to work HOT 2
- Kafka 3.7.0 and producer error "produce version 10 is not supported" HOT 2
- Is Kafka-proxy support Kafka Cluster in KRaft mode?
- Experiencing issue with AWS MSK IAM between Proxy and Brokers, and SASL Plain between Client and Kafka Proxy
- Running kafka-proxy in k8s with more then 1 replica HOT 1
- MSK Serverless Net Address Not Found HOT 1
- Exposing proxy through istio virtual service HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-proxy.