auto set offset of view to tail to avoid burrow alert about kafka-webview HOT 6 CLOSED

The issue with setting an explicit group.id for a view is if multiple users simultaneously browse the same "view", they will only be able see a portion of the data.

Example, lets assume you have a topic with 4 partitions.
User 1 - Browses within WebView the topic using group.id="MyGroup". Kafka will assign them partitions 1-4.
User 2 - Browses within WebView the topic using the same group.id="MyGroup". The kafka client will trigger a rebalance as now the group has 2 consumers. User 2 may get assigned only partitions 3-4. User 1 will be reassigned only partitions 1-2.

Unless I'm misunderstanding, At this point User 1 and User 2 is no longer able to view all of the data, only a subset for the partitions they got assigned within their consumer group.

Reference: https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example

That said after reviewing burrow's documentation it looks like you can define a group Id blacklist to be ignored. Currently KafkaWebView sets a unique consumer Id for every view + user session. Perhaps we should also set a unique group.id with a standardized prefix? This would allow you to blanket blacklist the group Ids from KafkaWebview, while having them still remain unique and avoid the issue I've described above. Something like group.id = "KafkaWebView::"

Do you think that would work?

from kafka-webview.

Hey Jiming!

I'm not sure I fully understand your use case here and have a few questions to understand better.

1 - Why would you want the consumer to automatically consume a topic fully? WebView allows you to manually page thru data in Kafka so you can inspect it. I'm not sure in what case you'd want WebView to automatically move your position through a topic in a background process.

2 - My understanding is that a consumer group is used to split consuming between multiple clients to prevent consuming the same piece of data more than once. In the context of WebView I'm not sure this makes sense? Is there a use case for WebView where you'd want to split what data is displayed in the browser across multiple clients/browsers?

from kafka-webview.

@Crim

I use burrow to monitor the lag of consumer. If a consumer lag happened, it will alert me by email or web hook.

Webview create a consumer group with group.id equals "" , and if I do not manually browse the topic, its offset will stay still. Burrow will consider it as a lag problem and send out alert.

Unfortunately, the group.id is "" makes me cannot operate it at all. If there had a group.id, I could write my own script to set the offset of webview consumer to tail to avoid burrow alert.

Maybe allow user to set a group.id when create view is an solution, what do you say?

Thanks!
Jiming

from kafka-webview.

That would definitely work! Thanks!

That said after reviewing burrow's documentation it looks like you can define a group Id blacklist to be ignored. Currently KafkaWebView sets a unique consumer Id for every view + user session. Perhaps we should also set a unique group.id with a standardized prefix? This would allow you to blanket blacklist the group Ids from KafkaWebview, while having them still remain unique and avoid the issue I've described above. Something like group.id = "KafkaWebView::"

from kafka-webview.

Hey @jiming I've cut release 1.0.3 which should set a standard prefix on all consumerIds and groupIds to "KafkaWebView"

Please let me know if this allows you to work around your issues.

from kafka-webview.

@Crim great, thank you very much!

Jiming

from kafka-webview.