Comments (3)
Hi @tarmath
If your processing actor(s)are parallelizable, then the actorRef you use with akka-kafka could certainly be a Router backed by several underlying processor actors.
Akka benchmarks show that a single actor can process 10s of millions of messages per second (at least), so you have to be going pretty fast to botteneck there.
In many cases you will want to have a single actor that manages some state. Imagine the simplest case where your processing actor simply counts messages recieved var count:Int = 0
. Using a single actor here makes the problem trivial. (Correctly) aggreagting state across a set of processing actors to calculate the total count is less so.
Make sense?
That said, I have considered having a way to allow for specifying a processing actor per StreamFSM to allow for ordered processing (where each processing actor will see the messages in order from the stream/kafka partition(s) it is associated with)
from akka-kafka.
Using a router would certainly solve my problem here, though it also sounds a little less "parallel" to me, as there is only one router taking in the messages from multiple StreamFSM. At any rate, and as you mentioned, it would take a lot of messages for the router to be overwhelmed while I can create more processors as needed, which is really where the slow processing happens (not due to akka but to the nature of my problem).
As for the issue where you would need to count the messages received, wouldn't that work trivially if you declare streams = 1 since you don't want parallelism in your receiver anyway? I guess I don't see the advantage of using multiple StreamFSM if it all ends up in the same Receiver? Doesn't it break the idea that having multiple Kafka streams is typically needed when the processing is parallelizable?
Finally, if you do end up making it possible to have a processing Actor per Stream, I would definitively use that feature!
Thanks again for your quick feedback and your suggestion!
from akka-kafka.
@tarmath one thing to note about number of streams...
If your consumer has less streams (in aggregate across the consumer group) than the number of partitions in the topic, then the underlying kafka consumer won't be receiving messages from all topics, but will instead switch the set of partitions it is consuming from every 10 minutes (by default)
There is more doc on that behavior (if you look closely enough ;). At the kafka website
from akka-kafka.
Related Issues (20)
- Any reason for losing the key HOT 2
- StateTimeout after 1 message is processed HOT 7
- Consumer with a Router HOT 5
- Question : How can I consume my kafka topic as fast as possible ? HOT 22
- No license explicitly stated HOT 2
- Race / off-by-one issue in BatchFSM? (Stuck temporarily in WaitingToSendBatch) HOT 3
- Empty Queue HOT 1
- Unable to cleanly stop AkkaConsumer HOT 1
- Batch consumer priority HOT 2
- ConsumerTimeoutException HOT 5
- Guidance Requested: Error Handling HOT 2
- Can AkkaConsumber inherit from a common base trait? HOT 2
- New Kafka .9 consumer HOT 3
- Multiple workers actors HOT 1
- `unhandled event StartProcessing` Warning
- Specifying multiple topic HOT 1
- Version in sonatype.org HOT 3
- Question, how to set SupervisorStrategy for StreamFSM?
- NPE in BatchConnectorFSM
- What exactly is outstanding? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from akka-kafka.