Clarify kafka commit strategy "latest" in documentation about quarkus HOT 5 CLOSED

Indeed that phrase can use some clarification. Async processing in that case refers to processing where order is not preserved like Blocking with ordered=false, or RunOnVirtualThread.

The message loss scenario is not that complicated really:
Imagine receiving 3 records from a topic-partition with offsets 3,4,5 and they are concurrently processed. If the processing of 5 gets completed and it gets committed back to Kafka, you may have an error during processing 4 and you'd never receive it again, because you've already committed a later offset.

Throttled strategy prevents exactly this case by tracking processed offsets and committing offsets later.

Hope this clarifies.

from quarkus.

/cc @alesj (kafka), @cescoffier (kafka), @ozangunalp (kafka)

from quarkus.

You added a link to a Zulip discussion, please make sure the description of the issue is comprehensive and doesn't require accessing Zulip

_{This message is automatically generated by a bot.}

from quarkus.

thanks for the clarification. I can submit a doc PR if you want.
an alternative enhancement would be to provide a strategy "latest-consecutive-message" to avoid message loss. but I am not sure this would add a lot of value compared to throttled.

from quarkus.

Hi, I believe I wrote the original documentation. At that time with latest if the user's code returned a completed future before acking the record then data could be lost (the record's offset could be committed before processing actually takes place). With throttled that is not the case since the offsets are updates only after acking. This presupposes an async method signature:

@Incoming("my-data")
@Acknowledgment(Acknowledgment.Strategy.MANUAL)
public CompletionStage<Void> process(Message<MyData> message) {

 // ack message out of order on some other vertx context, executor thread, etc.

  return CompletableFuture.completedFuture(null);
}

EDIT: latest will commit on ack, so if the user processes out of order then that's when data can be lost. User requests 3 records and receives them with offsets 1,2.3. User only acks 3, Latest will commit to 3+1 even though the previous 2 records haven't been acked yet. Throttled on the other hand will wait until 1 and 2 are acked before committing to 3+1.

from quarkus.