Comments (5)
In terms of consumer groups, I believe most of the points you raised are already available in Kafdrop. The consumer groups show up in the topic overview, and you can click through the consumer to see more information, which includes things like lag. See screenshot below.
The only thing it's not showing at the moment is which consumer is currently assigned to which partition, which is not something you can easily get out of the Kafka Admin API (there's probably a hacky way to get that out of ZK).
from kafdrop.
Cool! I missed that. My bad. Thank you for pointing me to it :)
which consumer is currently assigned to which partition
I've never had the opportunity to use the Kafka Admin API so I'm not able to help you right now. Currently to get this info from my cluster I'm using the following command
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
obtaining the following output
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
topic3 0 241019 395308 154289 consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2
topic2 1 520678 803288 282610 consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2
topic3 1 241018 398817 157799 consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2
topic1 0 854144 855809 1665 consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1 consumer1
topic2 0 460537 803290 342753 consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1 consumer1
topic3 2 243655 398812 155157 consumer4-117fe4d3-c6c1-4178-8ee9-eb4
It's strange that there isn't a counterpart in the Kafka Admin API to extract this information.
from kafdrop.
Then it likely does :) I must've missed it.
from kafdrop.
Thanks for your feedback; much valued. I think the auto-refresh is sensible and can be done with the current API.
Things like stats, rates and other metrics are probably something that should be taken out of JMX, and would need another integration into Kafka from Kafdrop. (Do-able but considerable amount of work.) It would also encroach into the sort of functionality that Prometheus/Grafana tend to offer. So I'm not sure if it's worth doing in Kafdrop, given that tools like that already do it and we'll only be playing catch up.
For your last point, would you be able to clarify what additional information could be presented?
from kafdrop.
Sorry for the late response.
About stats etc, you are right you must get this information from JMX (other product like Kafka Manager by Yahoo requires that to extract stats) and it requires time.
About the additional information about consumers, I meant something like:
- the list of available consumer groups;
- topics they consumes from;
- which consumer is connected to one (or more) partition/s;
- the log size, the consumer offset, lag for each consumer;
- % of Partitions assigned to a consumer instance.
Those are the stats I used to check manually to assess the Kafka cluster/server is/are ok :)
from kafdrop.
Related Issues (20)
- Kafdrop failing to connect with Kafka broker running on docker HOT 2
- kafdrop with sasl plein text auth timeout
- in KRaft mode, WARN: Broker may not be available HOT 2
- Unable to disable vulnerable TLS protocols TLSv1, TLSv1.0 HOT 2
- Please help create new release & image HOT 3
- Arm64 container images HOT 1
- Kafdrop 4 complains about missing key store password though keystore was not set HOT 7
- Kafdrop 4.0.0 Docker image doesn't start HOT 15
- Invalid value SASL_PLAINTEXT HOT 3
- StackOverflow exception HOT 3
- I can't connect Broker when i run by Docker on Mac M1 HOT 2
- Only one node in the cluster is available, but kafdrop shows all nodes are online HOT 2
- When running kafdrop in a separate container, I can't configure the kafka broker address. HOT 2
- IllegalFormatConversionException on Long Search Queries in Message Search HOT 2
- Is there any way to hide the messages from kafdrop as in PROD it is not suitable to show the messages of eventhub or kafka HOT 5
- I cannot view messages within the topics. (No messages found in partition 0 at offset 0)
- arm64 architecture HOT 3
- Does it support deployment with spring native ? HOT 2
- NPE HOT 2
- Consumer Group Id is not html escaped in the UI HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafdrop.