about batch message about cppkafka HOT 10 CLOSED

librdkafka doesn't have a batch consume interface, so you'll have to stick to the single message one. It's still really fast as it uses internal queues so you shouldn't have to worry about performance.

from cppkafka.

rd_kafka_produce_batch
rd_kafka_consume_batch_queue

i find delay between produce message to consume message one by one is not satisfy

from cppkafka.

I just pushed Consumer::poll_batch that consumes a batch of messages. I don't think it's worth adding a wrapper for rd_kafka_produce_batch as it's a very different API from rd_kafka_producev (it uses rd_kafka_message_t, rd_kafka_topic_t, etc). How many messages per second are you producing that producing one by one is not performant? Keep in mind kafka will batch them automatically already.

from cppkafka.

producer is not important , the key is comsumer needs poll_batch
i create a topic for 1 rep and 4 partitions for test
producer code like this
while(true) {
std::string comment = "stephen";
int partition = (count++) % 4;
comment += std::to_string(count);
producer.produce(MessageBuilder("deal").partition(partition).payload(comment));
if(count % 1000 == 0)
sleep(1);
}
and i create 4 consumers ,but i have only 1 cpu core...... it really too slow for one consumer，even other consumer is done

from cppkafka.

Are you committing the offsets manually or doing that automatically via rdkafka on the consumer side?

from cppkafka.

{ "enable.auto.commit", false }

from cppkafka.

That's not very helpful, it only means you're not committing offsets automatically. Are you committing messages syncrhonously? Asynchronously? One commit per message? One commit per multiple messages? How many messages?

from cppkafka.

while(running) {
Message msg = consumer.poll();
if(msg) {
if(msg.get_error()) {
if(!msg.is_eof()) {
cout << "[+] Received error notification: " << msg.get_error() << endl;
}
} else {
if(msg.get_key()) {
cout << msg.get_key() << " -> ";
}
cout << msg.get_payload() << endl;
consumer.commit(msg);
}
}
}

from cppkafka.

So, you're doing a synchronous commit, once every message, which is literally the worst commit strategy you could use. Each commit can take well over 100ms, so you're doing at most 10 messages per second. What's taking time on your consumer is not the consumption but the message commits.

You should batch commits rather than messages and only commit once every X messages.

from cppkafka.

ok thanks :)

from cppkafka.