Logstash-Kafka Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. about helk HOT 5 CLOSED

Updated pipeline.batch.size to 325 and the max_poll_records to 300. better performance in the lab environment in a few computers. testing it more. so far I have not had any error messages.

from helk.

Updated max_poll_records to 325 and performance improved. Setting the kafka input property max_poll_records to the same value as the pipeline.batch.size value improves performance.

from helk.

Updated max_poll_records and pipeline.batch.size to 500. Performance has been steady with HELK running on a local VM. (2 virtual cores). 150 event/s 👍 Not tested at scale yet. Error is not showing anymore. Updates to request_timeout_ms and session_timeout_ms helped to fix the error message. Closing this issue with commit.

from helk.

More references:
https://discuss.elastic.co/t/kafka-input-performance-config-tuning/70923
https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-max_poll_interval_ms
logstash-plugins/logstash-input-kafka#114

from helk.

THis is a compatibility issue , try updating the kafka version

from helk.