Comments (5)
@everesio My client is outside of kubernetes cluster.
I was finally able to figure out the issue. I was using nlb with the k8s service. In AWS, when nlb is created, it creates a target group which comes with tcp health check for each port exposed in service. kafka proxy didn;t understand this health check and failed with the message above. Solution was to use elb instead of nlb and expose port 9080(default healthcheck port) Doing this, there is no backend target group creation and healthcheck is only for first port listed in your service. I put port 9080 as first port(which maps to http healthcheck). This way there is no tcp check on broker listener port(in my case port 443)
Here is deployment and service yaml. Couple of things I also in deployment below is external dns integration and cert for proxy listener. This has nothing to do with the issue though.
---
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-proxy-1
name: kafka-proxy-1
annotations:
external-dns.alpha.kubernetes.io/hostname: broker1.mydomain.net
spec:
ports:
- name: health
port: 9080
protocol: TCP
targetPort: 9080
- name: kafka-proxy-1
port: 443
protocol: TCP
targetPort: 443
selector:
app: kafka-proxy-1
sessionAffinity: None
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-proxy-2
name: kafka-proxy-2
annotations:
external-dns.alpha.kubernetes.io/hostname: broker2.mydomain.net
spec:
ports:
- name: health
port: 9080
protocol: TCP
targetPort: 9080
- name: kafka-proxy-2
port: 443
protocol: TCP
targetPort: 443
selector:
app: kafka-proxy-2
sessionAffinity: None
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-proxy-3
name: kafka-proxy-3
annotations:
external-dns.alpha.kubernetes.io/hostname: broker3.mydomain.net
spec:
ports:
- name: health
port: 9080
protocol: TCP
targetPort: 9080
- name: kafka-proxy-3
port: 443
protocol: TCP
targetPort: 443
selector:
app: kafka-proxy-3
sessionAffinity: None
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kafka-proxy-1
spec:
replicas: 1
selector:
matchLabels:
app: kafka-proxy-1
template:
metadata:
labels:
app: kafka-proxy-1
spec:
hostNetwork: true
containers:
- name: kafka-proxy
securityContext:
capabilities:
add: ["NET_ADMIN", "SYS_TIME","NET_BIND_SERVICE"]
image: grepplabs/kafka-proxy:latest
args:
- 'server'
- '--log-format=json'
- '--bootstrap-server-mapping=b-1.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:443,broker1.mydomain.net:443'
- '--external-server-mapping=b-1.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker1.mydomain.net:443'
- '--external-server-mapping=b-2.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker2.mydomain.net:443'
- '--external-server-mapping=b-3.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker3.mydomain.net:443'
- '--tls-enable'
- '--log-level=debug'
- '--dynamic-listeners-disable'
- '--tls-insecure-skip-verify'
- '--proxy-request-buffer-size=32768'
- '--proxy-response-buffer-size=32768'
- '--proxy-listener-read-buffer-size=32768'
- '--proxy-listener-write-buffer-size=131072'
- '--kafka-connection-read-buffer-size=131072'
- '--kafka-connection-write-buffer-size=32768'
- '--proxy-listener-key-file=/opt/tls/tls.key'
- '--proxy-listener-cert-file=/opt/tls/tls.crt'
- '--proxy-listener-tls-enable'
ports:
- name: kafka-port1
containerPort: 443
- name: health
containerPort: 9080
volumeMounts:
- mountPath: /opt/tls
name: kafka-wildcard-cert
readOnly: true
volumes:
- name: kafka-wildcard-cert
secret:
defaultMode: 420
secretName: kafka-wildcard-cert
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kafka-proxy-2
spec:
replicas: 1
selector:
matchLabels:
app: kafka-proxy-2
template:
metadata:
labels:
app: kafka-proxy-2
spec:
hostNetwork: true
containers:
- name: kafka-proxy
securityContext:
capabilities:
add: ["NET_ADMIN", "SYS_TIME","NET_BIND_SERVICE"]
image: grepplabs/kafka-proxy:latest
args:
- 'server'
- '--log-format=json'
- '--bootstrap-server-mapping=b-2.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:443,broker2.mydomain.net:443'
- '--external-server-mapping=b-1.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker1.mydomain.net:443'
- '--external-server-mapping=b-2.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker2.mydomain.net:443'
- '--external-server-mapping=b-3.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker3.mydomain.net:443'
- '--tls-enable'
- '--log-level=debug'
- '--dynamic-listeners-disable'
- '--tls-insecure-skip-verify'
- '--proxy-request-buffer-size=32768'
- '--proxy-response-buffer-size=32768'
- '--proxy-listener-read-buffer-size=32768'
- '--proxy-listener-write-buffer-size=131072'
- '--kafka-connection-read-buffer-size=131072'
- '--kafka-connection-write-buffer-size=32768'
- '--proxy-listener-key-file=/opt/tls/tls.key'
- '--proxy-listener-cert-file=/opt/tls/tls.crt'
- '--proxy-listener-tls-enable'
ports:
- name: kafka-port2
containerPort: 443
- name: health
containerPort: 9080
volumeMounts:
- mountPath: /opt/tls
name: kafka-wildcard-cert
readOnly: true
volumes:
- name: kafka-wildcard-cert
secret:
defaultMode: 420
secretName: kafka-wildcard-cert
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kafka-proxy-3
spec:
replicas: 1
selector:
matchLabels:
app: kafka-proxy-3
template:
metadata:
labels:
app: kafka-proxy-3
spec:
hostNetwork: true
containers:
- name: kafka-proxy
securityContext:
capabilities:
add: ["NET_ADMIN", "SYS_TIME","NET_BIND_SERVICE"]
image: grepplabs/kafka-proxy:latest
args:
- 'server'
- '--log-format=json'
- '--bootstrap-server-mapping=b-3.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:443,broker3.mydomain.net:443'
- '--external-server-mapping=b-1.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker1.mydomain.net:443'
- '--external-server-mapping=b-2.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker2.mydomain.net:443'
- '--external-server-mapping=b-3.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker3.mydomain.net:443'
#- '--dial-address-mapping=b-3.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:443'
- '--tls-enable'
- '--log-level=debug'
- '--dynamic-listeners-disable'
- '--tls-insecure-skip-verify'
- '--proxy-request-buffer-size=32768'
- '--proxy-response-buffer-size=32768'
- '--proxy-listener-read-buffer-size=32768'
- '--proxy-listener-write-buffer-size=131072'
- '--kafka-connection-read-buffer-size=131072'
- '--kafka-connection-write-buffer-size=32768'
- '--proxy-listener-key-file=/opt/tls/tls.key'
- '--proxy-listener-cert-file=/opt/tls/tls.crt'
- '--proxy-listener-tls-enable'
ports:
- name: kafka-port3
containerPort: 443
- name: health
containerPort: 9080
volumeMounts:
- mountPath: /opt/tls
name: kafka-wildcard-cert
readOnly: true
volumes:
- name: kafka-wildcard-cert
secret:
defaultMode: 420
secretName: kafka-wildcard-cert
from kafka-proxy.
Here are my pods and nodes
➜ kafka-proxy kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kafka-proxy-1-7845cb57cc-hgf46 1/1 Running 0 2m30s 20.10.3.191 ip-20-10-3-252.ec2.internal <none> <none>
kafka-proxy-2-f9c556868-4kvl7 1/1 Running 0 2m30s 20.10.2.84 ip-20-10-2-185.ec2.internal <none> <none>
kafka-proxy-3-cc5cb5c6b-g4r6v 1/1 Running 0 2m30s 20.10.1.210 ip-20-10-1-197.ec2.internal <none> <none>
➜ kafka-proxy kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kafka-proxy-1-7845cb57cc-hgf46 1/1 Running 0 2m47s 20.10.3.191 ip-20-10-3-252.ec2.internal <none> <none>
kafka-proxy-2-f9c556868-4kvl7 1/1 Running 0 2m47s 20.10.2.84 ip-20-10-2-185.ec2.internal <none> <none>
kafka-proxy-3-cc5cb5c6b-g4r6v 1/1 Running 0 2m47s 20.10.1.210 ip-20-10-1-197.ec2.internal <none> <none>
from kafka-proxy.
@everesio I tried your example here. But that did not work .
from kafka-proxy.
Could you provide your client configuration ?
How do you want to access the proxy , from a client running in or outside kubernetes ?
from kafka-proxy.
Hi @rajk0007
Had a question. Why did you disable the dynamic listeners on this ? Also don't you see Client closed local connection on 172.17.0.2:30002 from 10.223.<<>>.76:60268 (b-2.<<>>.8af4q1.c14.kafka.us-east-1.amazonaws.com:9092)" in the logs of containers ?
Regards
from kafka-proxy.
Related Issues (20)
- [Question] Can I attach 3 bootstrap server endpoints to a single port? HOT 1
- [Question] If my Kafka brokers are running version 2.8.1, should I be using kafka-proxy version 0.2.9? HOT 1
- "Metadata" request (ApiKey=3 and ApiVersion=5) in the Kafka Proxy is not following the protocol structure defined by Kafka protocol guide HOT 1
- [Question] is there a plan to release a Java implementation of Kafka Proxy ? HOT 1
- [Need Help] Sending Custom METADATA response through Kafka Proxy
- [Question] is there a plan to support HTTPS proxy ?
- will there be an update to resovle 7 vulnerabilitys
- tls: failed to parse private key AWS MSK HOT 6
- bad performance when executing kafka-producer-perf-test.sh HOT 3
- Can not use grepplabs/kafka-proxy ARM image as base image HOT 1
- one port mapping to 6 broker HOT 2
- AWS Invalid API Key. What did I miss ?
- tls: failed to parse private key HOT 2
- Can't get proxy-listener tls to work HOT 2
- Kafka 3.7.0 and producer error "produce version 10 is not supported" HOT 2
- Is Kafka-proxy support Kafka Cluster in KRaft mode?
- Experiencing issue with AWS MSK IAM between Proxy and Brokers, and SASL Plain between Client and Kafka Proxy
- Running kafka-proxy in k8s with more then 1 replica HOT 1
- MSK Serverless Net Address Not Found HOT 1
- Exposing proxy through istio virtual service HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-proxy.