Comments (3)
This is more of a general akka/akka-cluster question, around how to achieve at-least-once delivery rather than the default of akka, which is at-most-once.
There is no free lunch here, so you will have to do some bookkeeping to achieve your goals.
Assuming you are using the batch consumer here, but if not the same principles apply, the implementation may be slightly different.
The first thing you might try is co-locating a 'bookkeeping' actor with the consumer. The bookkeeper would then be the receiver of messages from the consumer. When the bookkeeper receives a message, it would send the batch into your cluster and wait for a response. If the response was not delivered within some timeout, the bookkeeper would re-send, and continue to do so after each timeout encountered.
This is relatively straightforward using the scheduler, something like
val clusterActorRef: ActorRef = ... ///entrypoint to your cluster processing
val consumer:ActorRef = ... ///the akka-kafka consumer
var cancelTimer:Option[Cancellable] = None
//Bookkeeper Recieve
case batch: Batch =>
cancelTimer = Some(context.system.scheduler.schedule(Duration.zero, redeliveryTimeout, clusterActorRef, batch))
case BatchAck =>
cancelTimer.foreach(_.cancel)
consumer ! BatchConnectorFSM.BatchProcessed
If this is not 'reliable enough' for you, you should probably investigate using akka-persistence AtLeastOnceDelivery http://doc.akka.io/docs/akka/2.3.9/scala/persistence.html#At-Least-Once_Delivery
you could use that with a local leveldb journal. But in this case I would think that this is 'good enough' since if the consumer/bookkeeper node crashes, on restart you will get any in-flight messages replayed since they were not committed.
Make sense?
from akka-kafka.
Thanks a lot for your detailed explanation.
Appreciate your help
from akka-kafka.
Cheers!
from akka-kafka.
Related Issues (20)
- Any reason for losing the key HOT 2
- StateTimeout after 1 message is processed HOT 7
- Consumer with a Router HOT 5
- Question : How can I consume my kafka topic as fast as possible ? HOT 22
- No license explicitly stated HOT 2
- Race / off-by-one issue in BatchFSM? (Stuck temporarily in WaitingToSendBatch) HOT 3
- Empty Queue HOT 1
- Unable to cleanly stop AkkaConsumer HOT 1
- Batch consumer priority HOT 2
- ConsumerTimeoutException HOT 5
- Guidance Requested: Error Handling HOT 2
- Can AkkaConsumber inherit from a common base trait? HOT 2
- New Kafka .9 consumer HOT 3
- Multiple workers actors HOT 1
- `unhandled event StartProcessing` Warning
- Specifying multiple topic HOT 1
- Version in sonatype.org HOT 3
- Question, how to set SupervisorStrategy for StreamFSM?
- NPE in BatchConnectorFSM
- What exactly is outstanding? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from akka-kafka.