User feedback about franz-go HOT 4 CLOSED

Absolutely happy to get feedback!

(1) and (2), an examples folder has long been on my (undocumented) todo list. I'll get to these.

(3), right I'll add documentation on NewClient. The client itself is basically just an initialized struct on return; nothing has actually happened yet. But you're right, the update metadata loop goroutine has been spun up, so soon enough, brokers will be discovered.

(4), this sounds like maybe request and response hooks would be beneficial here? Optional functions that are called before requests are issued, and before responses are parsed?

from franz-go.

Thanks for your replies.

Regarding (4): I don't think I need hooks, but as of now I can only get broker handles that do not expose any metadata (not even the broker id is public). I think if you store the metadata after connecting to the broker in the Broker struct and make this accessible that's good enough. Otherwise/additionally maybe an OnBrokerConnected() hook which contains the metadata?

I'm looking forward to what's coming 🚀

Edit: BrokerMetadata seems to contain the following data, which is what I'm looking for:

type MetadataResponseBroker struct {
	// NodeID is the node ID of a Kafka broker.
	NodeID int32

	// Host is the hostname of a Kafka broker.
	Host string

	// Port is the port of a Kafka broker.
	Port int32

	// Rack is the rack this Kafka broker is in.
	Rack *string // v1+
}

from franz-go.

re(4) still, it may be better to talk in the discord about this, but I'm curious: do you want just the information in the metadata response, or do you want information about connected brokers and whatnot? If it's the former, would issuing your own MetadataRequest be sufficient?

If the latter, I think it's worth talking more about what the goal is so that I can write an API that may be more generally useful / expandable in the future. As it currently is, brokers aren't connected to until a request is just about to be written, and a Broker struct itself is a very thin abstraction to direct requests to a specific broker.

I was thinking about the hooks a bit more, one complicating factor is that the sink and source files use their own custom internal produceRequest and fetchRequest types. I've thought that hooks would be useful for having metrics around latencies for requests & responses, but those hooks would just need to know the request key that was issued (not the full request, and this is actually the reason I avoided having a ConnTimeoutFn(func(kmsg.Request) time.Duration) option). So, a general hook is out.

Another option is to change or supplement the Dialer option,

either 1) change it to func Dialer(fn func(context.Context, MetadataResponseBroker) (net.Conn, error)) Opt
or 2) supplement it with func MetadataDialer(..the same..) Opt

This would allow a direct hook into which broker is actually dialed when, and allow you to hook into the net.Conn to know when the broker connection is lost.

But anyway, most importantly is, what is the goal with this information?

from franz-go.

re(4) still, it may be better to talk in the discord about this, but I'm curious: do you want just the information in the metadata response, or do you want information about connected brokers and whatnot?

What I usually want to achieve is to connect to a Kafka cluster and afterwards print something that indicates "everything worked, I'm connected to at least one broker and discovered these 6 other brokers". Often users have troubles connecting to their cluster because of missing certificates, connectivity, wrong credentails etc. so that's really important for me.

I also want to await this point, so that I can then proceed with the rest of my application (e. g. starting a HTTP server afterwards / switching my probe endpoints to "ok" etc).

from franz-go.