Controlling Offsets about pentaho-kafka-consumer HOT 4 CLOSED

Hi,

No, for now it's not possible.

On Tue, Jul 5, 2016 at 3:42 PM, daviesgj2 [email protected] wrote:

Is it possible to control when an offset is written by PDI, i.e., I would
like to update the offset once the PDI transform has completed successfully
rather than when it read the message.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#13,
or mute the thread
https://github.com/notifications/unsubscribe/AAJpOwjjxL1Wg4pYGOfIJXKetPCbjswjks5qSlEwgaJpZM4JFE6H
.

from pentaho-kafka-consumer.

I wanted to introduce an Idea that could help to SOLVE this.

In general, the current architecture of PDI does not allow an INPUT step (In this case KAFKA Consumer) to check if the ROW/MESSAGE was completely processed before flagging it as successfully read. but there is an alternative...

The Idea is based on a new TYPE of Kafka Consumer Step for PDI that instead of a INPUT STEP, it extends the Single Threaded Step or the Execute Transformation Step and for EACH message it calls a Sub-transformation to process the message. Based on the transformation result SUCCESS/FAIL, it can make different options with the message, one of them flagging as READ or not.

This method would open a new world of possibilities

from pentaho-kafka-consumer.

Wouldn't Kafka consumer step represent a bottle-neck in such case?

from pentaho-kafka-consumer.

@pentaho-aschurman Could you explain your thought a little more?
is there a way to see the result of that child transform? (I have need for a similar functionality, and may be able to spend some time building out that functionality).

from pentaho-kafka-consumer.