Comments (7)
everesio
commented on June 26, 2024
Please provide your kafka config, the way you start the proxy and logs (--log-level debug)
from kafka-proxy.
brunodomenici
commented on June 26, 2024
Hi,
I'm proxying a Confluent Cloud broker instance. I don't have much information on broker side. I know that I have 6 nodes.
On proxy side, I have a Terraform to deploy Kafka-proxy on GKE.
Here my proxy config:
Name: kafka-proxy-bootstrap
Namespace: kafka-proxy
CreationTimestamp: Tue, 05 Nov 2019 16:33:04 +0100
Labels: app=kafka-proxy-bootstrap
Annotations: deployment.kubernetes.io/revision: 5
Selector: app=kafka-proxy-bootstrap
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 25% max surge
Pod Template:
Labels: app=kafka-proxy-bootstrap
node=kafka-proxy-bootstrap
Annotations: prometheus.io/scrape: true
Containers:
kafka-proxy:
Image: grepplabs/kafka-proxy:latest
Ports: 9080/TCP, 9092/TCP
Host Ports: 0/TCP, 0/TCP
Args:
server
--log-level=debug
--debug-enable
--bootstrap-server-mapping=xxx.gcp.confluent.cloud
[kafka-proxy.log](https://github.com/grepplabs/kafka-proxy/files/3813772/kafka-proxy.log)
:9092,0.0.0.0:9092,bootastrap.mydomain.com:9092
--external-server-mapping=b0-xxx.gcp.confluent.cloud:9092,node-0.mydomain.com:9092
--external-server-mapping=b1-xxx.gcp.confluent.cloud:9092,node-1.mydomain.com:9092
--external-server-mapping=b2-xxx.gcp.confluent.cloud:9092,node-2.mydomain.com:9092
--external-server-mapping=b3-xxx.gcp.confluent.cloud:9092,node-3.mydomain.com:9092
--external-server-mapping=b4-xxx.gcp.confluent.cloud:9092,node-4.mydomain.com:9092
--external-server-mapping=b5-xxx.gcp.confluent.cloud:9092,node-5.mydomain.com:9092
--tls-enable
--proxy-listener-tls-enable
--proxy-listener-cert-file=/var/run/secret/kafka-client-certificate/tls.crt
--proxy-listener-key-file=/var/run/secret/kafka-client-key/tls.key
Liveness: http-get http://:9080/health delay=5s timeout=1s period=3s #success=1 #failure=3
Readiness: http-get http://:9080/health delay=5s timeout=5s period=10s #success=2 #failure=5
Environment: <none>
Mounts:
/var/run/secret/kafka-client-certificate from tls-client-cert-file (rw)
/var/run/secret/kafka-client-key from tls-client-key-file (rw)
Volumes:
tls-client-cert-file:
Type: Secret (a volume populated by a Secret)
SecretName: tls-secret
Optional: false
tls-client-key-file:
Type: Secret (a volume populated by a Secret)
SecretName: tls-secret
Optional: false
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: kafka-proxy-bootstrap-67b69ddd9 (1/1 replicas created)
I have deployed on Kubernetes 1 proxy for bootstrap and 1 proxy for each node with the same proxy config.
Beside some TLS related messages, I'm not able to find anything on log, but here goes: kafka-proxy.log
Thank you!
from kafka-proxy.
brunodomenici
commented on June 26, 2024
I isolated one consumer group, enabled TRACE on kafka-consumer-group --describe --group svc-dst-event-test and I compare logs:
When I run through proxy I got [{error_code=16,group_id=svc-dst-event-test,group_state=,protocol_type=,protocol_data=,members=[],authorized_operations=0}]}:
[2019-11-07 11:41:28,645] TRACE [AdminClient clientId=adminclient-1] Sending DESCRIBE_GROUPS {groups=[svc-dst-event-test],include_authorized_operations=false} with correlation id 5 to node 4 (org.apache.kafka.clients.NetworkClient)
[2019-11-07 11:41:28,646] TRACE [AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=4162) (org.apache.kafka.clients.admin.KafkaAdminClient)
[2019-11-07 11:41:28,646] TRACE [AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 0 response(s) (org.apache.kafka.clients.admin.KafkaAdminClient)
[2019-11-07 11:41:28,646] TRACE [AdminClient clientId=adminclient-1] Trying to choose nodes for [] at 1573123288646 (org.apache.kafka.clients.admin.KafkaAdminClient)
[2019-11-07 11:41:28,646] TRACE [AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=4161) (org.apache.kafka.clients.admin.KafkaAdminClient)
[2019-11-07 11:41:28,668] TRACE [AdminClient clientId=adminclient-1] Completed receive from node 4 for DESCRIBE_GROUPS with correlation id 5, received {throttle_time_ms=0,groups=[{error_code=16,group_id=svc-dst-event-test,group_state=,protocol_type=,protocol_data=,members=[],authorized_operations=0}]} (org.apache.kafka.clients.NetworkClient)
[2019-11-07 11:41:28,669] TRACE [AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 1 response(s) (org.apache.kafka.clients.admin.KafkaAdminClient)
[2019-11-07 11:41:28,671] TRACE [AdminClient clientId=adminclient-1] Call(callName=describeConsumerGroups, deadlineMs=1573123292807) got response {throttle_time_ms=0,groups=[{error_code=16,group_id=svc-dst-event-test,group_state=,protocol_type=,protocol_data=,members=[],authorized_operations=0}]} (org.apache.kafka.clients.admin.KafkaAdminClient)
[2019-11-07 11:41:28,671] TRACE [AdminClient clientId=adminclient-1] Trying to choose nodes for [] at 1573123288669 (org.apache.kafka.clients.admin.KafkaAdminClient)
[2019-11-07 11:41:28,671] TRACE [AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=299532) (org.apache.kafka.clients.admin.KafkaAdminClient)
Error: Executing consumer group command failed due to org.apache.kafka.common.errors.NotCoordinatorException: This is not the correct coordinator.
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NotCoordinatorException: This is not the correct coordinator.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.$anonfun$describeConsumerGroups$1(ConsumerGroupCommand.scala:402)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.map(TraversableLike.scala:237)
at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.describeConsumerGroups(ConsumerGroupCommand.scala:401)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.collectGroupsOffsets(ConsumerGroupCommand.scala:417)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.describeGroups(ConsumerGroupCommand.scala:312)
at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:63)
at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
Caused by: org.apache.kafka.common.errors.NotCoordinatorException: This is not the correct coordinator.
When I run directly to Confluent Cloud I got {error_code=0,group_id=svc-dst-event-test,group_state=Empty,protocol_type=consumer,protocol_data=,members=[],authorized_operations=0}:
[2019-11-07 11:37:50,018] TRACE [AdminClient clientId=adminclient-1] Sending DESCRIBE_GROUPS {groups=[svc-dst-event-test],include_authorized_operations=false} with correlation id 5 to node 4 (org.apache.kafka.clients.NetworkClient)
[2019-11-07 11:37:50,018] TRACE [AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=4214) (org.apache.kafka.clients.admin.KafkaAdminClient)
[2019-11-07 11:37:50,019] TRACE [AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 0 response(s) (org.apache.kafka.clients.admin.KafkaAdminClient)
[2019-11-07 11:37:50,019] TRACE [AdminClient clientId=adminclient-1] Trying to choose nodes for [] at 1573123070019 (org.apache.kafka.clients.admin.KafkaAdminClient)
[2019-11-07 11:37:50,019] TRACE [AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=4213) (org.apache.kafka.clients.admin.KafkaAdminClient)
[2019-11-07 11:37:50,048] TRACE [AdminClient clientId=adminclient-1] Completed receive from node 4 for DESCRIBE_GROUPS with correlation id 5, received {throttle_time_ms=0,groups=[{error_code=0,group_id=svc-dst-event-test,group_state=Empty,protocol_type=consumer,protocol_data=,members=[],authorized_operations=0}]} (org.apache.kafka.clients.NetworkClient)
[2019-11-07 11:37:50,049] TRACE [AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 1 response(s) (org.apache.kafka.clients.admin.KafkaAdminClient)
I have no idea why, any idea?
Thank you!
from kafka-proxy.
everesio
commented on June 26, 2024
Seems like a configuration problem:
"time=""2019-11-06T09:52:42Z"" level=info msg=""Reading data from local connection on 10.44.1.40:9092 from 10.44.1.16:60056 (xxx.gcp.confluent.cloud:9092) had error: tls: first record does not look like a TLS handshake""
Either the server nodes xxx.gcp.confluent.cloud:9092 are not configured with TLS (--tls-enable) or your client (bin/kafka-consumer-groups) is not configured to use TLS configured on proxy (--proxy-listener-tls-enable)
from kafka-proxy.
brunodomenici
commented on June 26, 2024
Okay, I've solved the message on TLS, but I still have problem.
All other operations works fine. I can consumer and produce messages. It's just consumer groups related operations that I have problem using Kafka-proxy.
Is there any change that I could problem when a consumer group is created without proxy and then try to get information of it using proxy?
Any other recommendation?
Thank you!
from kafka-proxy.
everesio
commented on June 26, 2024
Could you provide the new way how you start the proxy and debug logs ?
I've tested with local kafka and kafka-consumer-groups.sh from kafka_2.12-2.3.1 and had no problems. I guess this is a config problem
kafka-proxy server --log-level debug --bootstrap-server-mapping "localhost:19092,0.0.0.0:30001" \ --bootstrap-server-mapping "localhost:29092,0.0.0.0:30002" \ --bootstrap-server-mapping "localhost:39092,0.0.0.0:30003"
`
kafka_2.12-2.3.1 bin/kafka-consumer-groups.sh --bootstrap-server localhost:30002 --all-groups --all-topics --describe --verbose
Consumer group 'KafkaExampleConsumer2' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
KafkaExampleConsumer2 topic-start-old-2 0 0 0 0 - - -
`
from kafka-proxy.
brunodomenici
commented on June 26, 2024
I've solved my problem here! :0]
The problem was my config. Actually in my Terraform I mapped just --bootstrap-server for my bootstrap node, it's necessary to map it in each node --bootstrap-server=mynode-0, not only on --external-server-.
The strange is that all APIs worked fine (consumer/producer/topics/etc), but when we call some API related to consumer groups we had problem. I think that is related do reverse DNS/SSL problem.
Thanks a lot for your help and for your project! 🥇
from kafka-proxy.
Related Issues (20)
- cannot connect to aws kafka from on-prem server with SASL_SSL enabled HOT 2
- Simple use case to connect docker compose container to remote vpn kafka cluster over socks5a proxy via ssh HOT 1
- Pod startup issue after version 0.3.3-all HOT 3
- CVE-2023-37788 - github.com/elazarl/goproxy HOT 3
- AWS MSK Serverless - had error: api key -13567 is invalid HOT 4
- updating to tag 0.3.7-all from 0.3.3-all getting error auth-local-command HOT 1
- [Question] Can I attach 3 bootstrap server endpoints to a single port? HOT 1
- [Question] If my Kafka brokers are running version 2.8.1, should I be using kafka-proxy version 0.2.9? HOT 1
- "Metadata" request (ApiKey=3 and ApiVersion=5) in the Kafka Proxy is not following the protocol structure defined by Kafka protocol guide HOT 1
- [Question] is there a plan to release a Java implementation of Kafka Proxy ? HOT 1
- [Need Help] Sending Custom METADATA response through Kafka Proxy
- [Question] is there a plan to support HTTPS proxy ?
- will there be an update to resovle 7 vulnerabilitys
- tls: failed to parse private key AWS MSK HOT 6
- bad performance when executing kafka-producer-perf-test.sh HOT 3
- Can not use grepplabs/kafka-proxy ARM image as base image HOT 1
- one port mapping to 6 broker HOT 2
- AWS Invalid API Key. What did I miss ?
- tls: failed to parse private key HOT 2
- Can't get proxy-listener tls to work HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-proxy.