Comments (4)
Hi,
No, for now it's not possible.
On Tue, Jul 5, 2016 at 3:42 PM, daviesgj2 [email protected] wrote:
Is it possible to control when an offset is written by PDI, i.e., I would
like to update the offset once the PDI transform has completed successfully
rather than when it read the message.—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#13,
or mute the thread
https://github.com/notifications/unsubscribe/AAJpOwjjxL1Wg4pYGOfIJXKetPCbjswjks5qSlEwgaJpZM4JFE6H
.
from pentaho-kafka-consumer.
I wanted to introduce an Idea that could help to SOLVE this.
In general, the current architecture of PDI does not allow an INPUT step (In this case KAFKA Consumer) to check if the ROW/MESSAGE was completely processed before flagging it as successfully read. but there is an alternative...
The Idea is based on a new TYPE of Kafka Consumer Step for PDI that instead of a INPUT STEP, it extends the Single Threaded Step or the Execute Transformation Step and for EACH message it calls a Sub-transformation to process the message. Based on the transformation result SUCCESS/FAIL, it can make different options with the message, one of them flagging as READ or not.
This method would open a new world of possibilities
from pentaho-kafka-consumer.
Wouldn't Kafka consumer step represent a bottle-neck in such case?
from pentaho-kafka-consumer.
@pentaho-aschurman Could you explain your thought a little more?
is there a way to see the result of that child transform? (I have need for a similar functionality, and may be able to spend some time building out that functionality).
from pentaho-kafka-consumer.
Related Issues (20)
- Some questions HOT 6
- does it work with pentaho 6.1? HOT 6
- Kafka 0.10 support is missing HOT 3
- Apache Kafka Consumer - using variables HOT 2
- Sub-transforms HOT 2
- Using consumer with pdi 6.1.0.3 enterprise - java.lang.linkageError: loader constraint violation HOT 1
- Kafka Consumer Step Question HOT 3
- SSL Encryption HOT 3
- Kafka reprocess messages HOT 3
- kafka 2.11.010 err HOT 3
- Kafka Consumer doesn't allow parameters in the properties section... HOT 3
- New client access for new versions of Kafka HOT 3
- after kafka connect JavaScript have question HOT 2
- does the zookeerper server parameter allow to have multiple nodes ? HOT 1
- process parition specific data HOT 1
- Testing in JDK 9 is not supported
- consumer can't work?
- Consumer reads messages more than once HOT 2
- I want to connect bootstrap.servers , but there are no parameters HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pentaho-kafka-consumer.