Comments (5)
Hi @nickpoulos
I was curious how the published config file was actually used within the library.
It's used to define default configurations for kafka consumers/producers.
I see there is a Kafka Consume command that is not mentioned in the documentation. The way it is written now, this command does not know how to handle the messages it is consuming.
How do we pass a message handler into this command?
You have a good point here. The way it's written now, it's impossible to pass the message handler to the artisan consumer. I think we can have a default handler
within config/kafka.php
but also allow to pass the class via command options, which would allow using the built-in command via supervisor specifying a different handler for each message type.
Feel free to submit a PR if you want to, or i'll work on that after work.
Thanks!
from laravel-kafka.
Hi @mateusjunges thanks for the quick reply. I started creating the PR, but had some questions about the --consumer
option in the command. How was this designed to be used originally? It seems if we can pass in a Consumer class, the handler could be set there first.
What is the purpose of the consumer
property within the Config class?
from laravel-kafka.
@nickpoulos actually, the consumer
option was created to be the message handler, and this consumer must extend the Junges\Kafka\Contracts\Consumer
class. Within ConsumerBuilder.php
, the config class receives a CallableConsumer
instance for the consumer option, which is created using the handler passed and the middlewares, if any. For the KafkaConsumerCommand
, it isn't working as expected. I'm pretty sure i forgot this class completely over the last updates i made to this library. Also, the command option should be called handler
, not consumer, since the consumer is created using the message handler. It's not clear and one point in it's fabor is that even i got confused about it. So, it's not impossible to pass a message handler to the command, but the command itself isn't working and needs some fixes. The Consumer
class now users two parameters: the config and the message deserializer, but the command only pass the configuration. I think it would be nice to allow the user to choose which deserializer to be used while consuming messages using this command (using command arguments/options). Sorry about the answer i gave you in my previous comment, now looking at the code carefully i understand whats happening here.
from laravel-kafka.
Ok that all makes sense to me for the most part.
But there also seems to be some confusion around Options/Config?
It seems like we want to be able to pass in options via the CLI command, but then if those options are not provided we want to fall back to whatever is in config. And you started making the Options class to handle this merging? Is that correct? It is hard to tell because it appears unfinished, but I can continue down this route and fix the discrepancies if you agree.
Another idea I had was, rather than provide more and more options via the CLI, what if we setup our /config/kafka.php
more similar to how Laravel's mysql options are setup? Ex: you would create a named connection
the config file, and then for the CLI command, we pass in a connection
param.
Let me know what you think, I can continue down either path here.
from laravel-kafka.
It seems like we want to be able to pass in options via the CLI command, but then if those options are not provided we want to fall back to whatever is in config. And you started making the Options class to handle this merging? Is that correct? It is hard to tell because it appears unfinished, but I can continue down this route and fix the discrepancies if you agree.
Yes, the Options
class receives the options passed to the artisan command, and if it's not present, we should fall back to config defaults.
Another idea I had was, rather than provide more and more options via the CLI, what if we setup our
/config/kafka.php
more similar to how Laravel's mysql options are setup? Ex: you would create a namedconnection
the config file, and then for the CLI command, we pass in aconnection
param.
I think the connection option is a really good idea, so we can create connections in kafka.php
and it decrease the number of options we need to pass via cli.
from laravel-kafka.
Related Issues (20)
- Have to specify topic name in each Message HOT 3
- Failed to resolve `broker:9092` HOT 4
- Getting a timeout issue, while trying to publish a message. HOT 7
- Undefined constant \"Junges\\Kafka\\RD_KAFKA_PARTITION_UA\ HOT 3
- Upstash Support - Producer not sending messages. HOT 3
- Wrong return type in Junges\Kafka\Config\Config->getCommit HOT 2
- Cannot Consume published messages in laravel 11 and php 8.3.6 HOT 15
- Issue consuming records from Avro topic HOT 1
- Cannot consume produced messages in v2.0 HOT 2
- What do we need to set for the `--consumer` flag. HOT 1
- Message consuming is still an issue in v2.1 HOT 10
- Encoded payloads: Is it possible with Avro Serializer to Save encoded messages in Kafka when sending it using AvroSerialize and doing the opposite when reading with Deserializer HOT 1
- Is there any support to Retry policies ? HOT 2
- [QUESTION] Possibility to describe topic / calculate read backlog HOT 1
- [Question/bug] Consuming Tombstones and send it to dlq throws $payload must be of type string, null given HOT 1
- cant send message HOT 8
- Error "Specified group generation id is not valid" during consumption HOT 3
- How to commit each message in withHandler HOT 1
- Missing messages HOT 2
- Customizable flush retried and timeout
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from laravel-kafka.