Comments (3)
Hi Andre
Thanks for the report.
It would be great if you could write a PR with a test case that reproduces this behavior. (The tests are run with it:test in sbt, see the bottom of the readme for how to get kafka/zk running locally)
Also what config are you using, if any, that is non default?
I think I have an intuition for why this is happening and the solution may be similar to what you describe, but instead of the scheduler, we would use the akka FSM StateTimeout mechanism inside the unused state, and when we timeout, we send a continue. (It may or may not be more complex than that)
If we use the scheduler, the Continue could happen in any state, which we don't want.
The issue you see is only a problem when you are not committing messages, because you get to try again after a commit when you are.
Regarding your second question, it could be that there are GC or other pauses that cause the underlying kafka consumer to timeout, a test case that reproduces this would let us reason better about it.
Thanks!
Sent from my iPhone
On Dec 11, 2014, at 4:33 AM, Andre Kuhnen [email protected] wrote:
For some reasons the first time the StreamFSM tries to get messages from kafka it does not find any message and goes to state Unused forever (If I use CommitConfig(None, none).
My idea is to have a scheduler that should send a message (Continue) after some time, for exemple:When(Unused)
system.scheduler.schedule(10 seconds, 1 second, self, Continue) //Just onceIf you agree with this , I can make a pull request
By the way, Any idea why if I set a high maxMessagesInFlight (5000 for example), the KafkaConsumer stops receiving messages and then it goes to Unsued state ?. It seems some problem with the KafkaConsumer when I try to read a lot of messages really fast
—
Reply to this email directly or view it on GitHub.
from akka-kafka.
Hi Sclasen,
I will try to reproduce the behavior about the unused state and use the FSMstate timeout, then I will make a pull request.
About the second question I was overriding some actors names and for some strange reason akka would not throw an exception. This problem is fixed now.
Thanks
from akka-kafka.
#25 in process to resolve this.
from akka-kafka.
Related Issues (20)
- Any reason for losing the key HOT 2
- StateTimeout after 1 message is processed HOT 7
- Consumer with a Router HOT 5
- Question : How can I consume my kafka topic as fast as possible ? HOT 22
- No license explicitly stated HOT 2
- Race / off-by-one issue in BatchFSM? (Stuck temporarily in WaitingToSendBatch) HOT 3
- Empty Queue HOT 1
- Unable to cleanly stop AkkaConsumer HOT 1
- Batch consumer priority HOT 2
- ConsumerTimeoutException HOT 5
- Guidance Requested: Error Handling HOT 2
- Can AkkaConsumber inherit from a common base trait? HOT 2
- New Kafka .9 consumer HOT 3
- Multiple workers actors HOT 1
- `unhandled event StartProcessing` Warning
- Specifying multiple topic HOT 1
- Version in sonatype.org HOT 3
- Question, how to set SupervisorStrategy for StreamFSM?
- NPE in BatchConnectorFSM
- What exactly is outstanding? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from akka-kafka.