Comments (4)
Absolutely happy to get feedback!
(1) and (2), an examples folder has long been on my (undocumented) todo list. I'll get to these.
(3), right I'll add documentation on NewClient
. The client itself is basically just an initialized struct on return; nothing has actually happened yet. But you're right, the update metadata loop goroutine has been spun up, so soon enough, brokers will be discovered.
(4), this sounds like maybe request and response hooks would be beneficial here? Optional functions that are called before requests are issued, and before responses are parsed?
from franz-go.
Thanks for your replies.
Regarding (4): I don't think I need hooks, but as of now I can only get broker handles that do not expose any metadata (not even the broker id is public). I think if you store the metadata after connecting to the broker in the Broker
struct and make this accessible that's good enough. Otherwise/additionally maybe an OnBrokerConnected()
hook which contains the metadata?
I'm looking forward to what's coming 🚀
Edit: BrokerMetadata seems to contain the following data, which is what I'm looking for:
type MetadataResponseBroker struct {
// NodeID is the node ID of a Kafka broker.
NodeID int32
// Host is the hostname of a Kafka broker.
Host string
// Port is the port of a Kafka broker.
Port int32
// Rack is the rack this Kafka broker is in.
Rack *string // v1+
}
from franz-go.
re(4) still, it may be better to talk in the discord about this, but I'm curious: do you want just the information in the metadata response, or do you want information about connected brokers and whatnot? If it's the former, would issuing your own MetadataRequest be sufficient?
If the latter, I think it's worth talking more about what the goal is so that I can write an API that may be more generally useful / expandable in the future. As it currently is, brokers aren't connected to until a request is just about to be written, and a Broker
struct itself is a very thin abstraction to direct requests to a specific broker.
I was thinking about the hooks a bit more, one complicating factor is that the sink and source files use their own custom internal produceRequest
and fetchRequest
types. I've thought that hooks would be useful for having metrics around latencies for requests & responses, but those hooks would just need to know the request key that was issued (not the full request, and this is actually the reason I avoided having a ConnTimeoutFn(func(kmsg.Request) time.Duration)
option). So, a general hook is out.
Another option is to change or supplement the Dialer
option,
either 1) change it to func Dialer(fn func(context.Context, MetadataResponseBroker) (net.Conn, error)) Opt
or 2) supplement it with func MetadataDialer(..the same..) Opt
This would allow a direct hook into which broker is actually dialed when, and allow you to hook into the net.Conn to know when the broker connection is lost.
But anyway, most importantly is, what is the goal with this information?
from franz-go.
re(4) still, it may be better to talk in the discord about this, but I'm curious: do you want just the information in the metadata response, or do you want information about connected brokers and whatnot?
What I usually want to achieve is to connect to a Kafka cluster and afterwards print something that indicates "everything worked, I'm connected to at least one broker and discovered these 6 other brokers". Often users have troubles connecting to their cluster because of missing certificates, connectivity, wrong credentails etc. so that's really important for me.
I also want to await this point, so that I can then proceed with the rest of my application (e. g. starting a HTTP server afterwards / switching my probe endpoints to "ok" etc).
from franz-go.
Related Issues (20)
- Transaction is not properly aborted HOT 1
- Need to "re-assume the IAM role when re-authenticating" to avoid "Session too short" errors on MSK HOT 6
- PollFetches take long time for the first fetch HOT 2
- question about sync=>async=>sync messaging HOT 1
- add ability to provide own member id HOT 3
- Produce Latency Spikes Due To Race Condition When Brokers Are Scaled Down HOT 8
- TXN: error beginning transaction: invalid attempt to begin a transaction while already in a transaction HOT 3
- Clarify producer batch config, once again HOT 3
- [Question] Checking client state before using it HOT 1
- Confluent: Unable to create a topic due to policy violation HOT 3
- How to fetch zero records without waiting HOT 1
- GroupTransactSession Close hangs, preventing restart HOT 4
- Use EOS sessions with BlockRebalanceOnPoll HOT 3
- ProduceSync returning "context canceled" error when parent context hasn't been cancelled HOT 5
- Is there a way to limit consumer PollRecords buffered fetches by size? HOT 1
- Question: Why does kgo.StopProducerOnDataLossDetected() exist? HOT 2
- Incorrect documentation regarding sarama partitioner compatibility SaramaCompatHasher HOT 7
- Feature request: expose metadata to the user HOT 2
- Documentation request / question HOT 1
- RequestTimeoutOverhead appears to override Context cancellation HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from franz-go.