Comments (5)
Hi @AdrianoRamalho21 to be honest i've never had this use case, but from what i know about kafka, you typically starts either from earliest or the latest offset. If the consumer crashes, then all partitions of the crashed consumer will be reset to the last commited offset, and all messages arrived since the last commit will be read again. Kafka ensures at least once
delivery, which means that no messages willbe missed, but duplicated can occur, so you should handle it in your app.
from laravel-kafka.
@mateusjunges thank you for the fast response.
I changed my offset_reset
from "latest" to "earliest", based on this stackoverflow thread https://stackoverflow.com/questions/48320672/what-is-the-difference-between-kafka-earliest-and-latest-offset-values#comment83661137_48320871
just in case.
Can you answer me one more question?
Is it possible with your package to get a message from a kafka topic with a specific offset ?
The thing is, Im listening to 2 topics, lets say topic-1 and topic-2, where the handle of the topic-2 has some logic that depends on the data saved from topic-1, and I am pretty sure the producer sends the message from topic-1 first, than, a few minutes later, sends the message from topic-2, and there is the problem.
If the handling of the topic-2 is missing the data from the topic-1, it throws a custom Exception, and that is happening a lot lately :/
from laravel-kafka.
@AdrianoRamalho21 sorry for the delay, i was very busy at work this past days. Currently, consuming from specific offset is no possible, since it is not implemented yet. I will take a look at this and possibly implement as a new feature, but don't expect it for the next few days.
from laravel-kafka.
@mateusjunges no problem ;) Thank you, and I think it will be a useful new feature, in some particularly/specific cases.
No need to rush it tho :D
Thanks one more time, and you may close this issue.
Good work and have a nice day
from laravel-kafka.
Thank you. Feel free to open a new issue as a feature request for that. Closing this for now.
from laravel-kafka.
Related Issues (20)
- Wrong return type in Junges\Kafka\Config\Config->getCommit HOT 2
- Cannot Consume published messages in laravel 11 and php 8.3.6 HOT 15
- Issue consuming records from Avro topic HOT 1
- Cannot consume produced messages in v2.0 HOT 2
- What do we need to set for the `--consumer` flag. HOT 1
- Message consuming is still an issue in v2.1 HOT 10
- Encoded payloads: Is it possible with Avro Serializer to Save encoded messages in Kafka when sending it using AvroSerialize and doing the opposite when reading with Deserializer HOT 1
- Is there any support to Retry policies ? HOT 2
- [QUESTION] Possibility to describe topic / calculate read backlog HOT 1
- [Question/bug] Consuming Tombstones and send it to dlq throws $payload must be of type string, null given HOT 1
- cant send message HOT 8
- Error "Specified group generation id is not valid" during consumption HOT 3
- How to commit each message in withHandler HOT 1
- Missing messages HOT 2
- Customizable flush retried and timeout
- stopAfterLastMessage not working as expected. HOT 1
- Undefined constant "Junges\Kafka\RD_KAFKA_PARTITION_UA", HOT 2
- Consumer middlewares cannot be implemented at described HOT 4
- Support for php 7.4 and laravel 8 HOT 1
- Key cannot be encoded with AVRO when producing HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from laravel-kafka.