Comments (3)
Kafka Consumer plug-in implements so called "high level" consumer. That means that latest offsets will be committed periodically according to the setting "auto.commit.interval.ms". What happens if a process fails? After a process will be restarted it will start consuming Kafka messages from the last committed offset, which means the same messages might be re-processed again. In short, no messages will be lost given there's enough retention in Kafka, but there might be some duplication. I suggest you to read this document for better understanding Kafka delivery semantics: https://kafka.apache.org/08/documentation.html#semantics
In order to provide "exactly once" semantics, the plug-in must be based on Kafka simple consumer, so we could better manage what offsets to start consuming from.
from pentaho-kafka-consumer.
Thank you @spektom for this explanation. Let me explore some details for my specific scenario: kafka_consumer -> transformation_executor(called every 1000 rows)
What I see is that kafka_consumer outputs up to 10000 rows in the intra-step buffer ("Nr of rows in rowset" in transformation miscellaneous properties) and then waits. The called-transformation processes 1000 record and when it finishes another 1000 are fetched from kafka_consumer and so on. The speed -as always- depends on the slower step. Let's say that the inner transformation performs 200rows/second. So every execution lasts 5 seconds (consuming 1000 rows we said). On average.
My point is: if the transformation crashes I need to replay the 10k messages in buffer. So their offset should not be committed. How can I achieve that? May I set the "auto.commit.interval.ms" to 20.000? I mean, after reading some data wait 20 seconds before committing THAT offset, so that, if you are still alive after 20s, I can assume that the inner trasformation (that should take 5seconds) is over.
But, and this is my question, if this property means that every 20s the client commits the newest offset it has in memory (aka in the buffer) there is no way to replay messages upon crash because those in the buffer are committed but never get sent to the rest fo the pipeline.
Have I made myself clear?
Thank you for your work and support
Virgilio
from pentaho-kafka-consumer.
Hi Virgilio,
I believe I answered you in #34 already :)
Yes, you can play with auto.commit.interval.ms
parameter, and set it even to 3000.
This is the downside of Kafka high-level consumer.
from pentaho-kafka-consumer.
Related Issues (20)
- Some questions HOT 6
- does it work with pentaho 6.1? HOT 6
- Controlling Offsets HOT 4
- Kafka 0.10 support is missing HOT 3
- Apache Kafka Consumer - using variables HOT 2
- Sub-transforms HOT 2
- Using consumer with pdi 6.1.0.3 enterprise - java.lang.linkageError: loader constraint violation HOT 1
- Kafka Consumer Step Question HOT 3
- SSL Encryption HOT 3
- kafka 2.11.010 err HOT 3
- Kafka Consumer doesn't allow parameters in the properties section... HOT 3
- New client access for new versions of Kafka HOT 3
- after kafka connect JavaScript have question HOT 2
- does the zookeerper server parameter allow to have multiple nodes ? HOT 1
- process parition specific data HOT 1
- Testing in JDK 9 is not supported
- consumer can't work?
- Consumer reads messages more than once HOT 2
- I want to connect bootstrap.servers , but there are no parameters HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pentaho-kafka-consumer.