AssignGroup fails with an invalid coordinator ID about franz-go HOT 11 CLOSED

I've noticed this as well, this happens for me immediately when I start a cluster. What I know so far is that Kafka actually does not create a group coordinator (nor transaction coordinator) until a group (or txn id) is used at least once. During the period of time it's creating these, it replies that no group coordinator exists.

You can see this by issuing a metadata request against your broker for all topics: the __consumer_offsets topic will not yet exist.

If you leave the client running, does it eventually recover? I just looked at the franz-go code and I'm actually suspicious that I cache the invalid coordinator (under ID -1). I'll skim this a bit more, but if you can vet that too, that'd be great.

from franz-go.

I left my consumer running for an hour but it didn't recover. On a side note, I noticed that kafka-console-consumer.sh from the Kafka cli works fine with a new group id.

from franz-go.

Cool, I'm going to patch this quickly, I'm confident the issue is because the invalid node ID is cached permanently.

With that side note, do you mean that the kafka-console-consumer does not run into this problem when a cluster first starts? If so, yeah I'd expect that, I definitely am seeing a caching bug within the client.

from franz-go.

Correct. I didn't see any issues with kafka-console-consumer.

from franz-go.

I think 528f007 (and the quick patch from the laptop I can actually test this on, ef9a16a) should fix this. I tested this by turning off my cluster, creating a new cluster, and immediately integration testing -- things pass now. Historically they did not, so thank you for bringing this to my attention in a different way! I always thought this was something else funky going on and that my local setup was busted.

Would you mind giving the latest commit a shot? If it fixes this, I can tag 0.7.1.

from franz-go.

I tried 528f007 locally. I don't see coordinator -1 error anymore, but it seems the client gets stuck while joining the group.

DEBUG	kgoclient	opening connection to broker	{"addr": "kafka:9092", "broker": -2147483648}
DEBUG	kgoclient	connection opened to broker	{"addr": "kafka:9092", "broker": -2147483648}
DEBUG	kgoclient	issuing api versions request	{"broker": -2147483648, "version": 3}
DEBUG	kgoclient	wrote ApiVersions v3	{"broker": -2147483648, "bytes_written": 29, "write_wait": "184.8µs", "time_to_write": "139.6µs", "err": null}
DEBUG	kgoclient	read ApiVersions v3	{"broker": -2147483648, "bytes_read": 352, "read_wait": "92.9µs", "time_to_read": "11.524ms", "err": null}
DEBUG	kgoclient	connection initialized successfully	{"addr": "kafka:9092", "broker": -2147483648}
DEBUG	kgoclient	wrote Metadata v9	{"broker": -2147483648, "bytes_written": 34, "write_wait": "16.057ms", "time_to_write": "256.5µs", "err": null}
DEBUG	kgoclient	read Metadata v9	{"broker": -2147483648, "bytes_read": 108, "read_wait": "157.5µs", "time_to_read": "3.1607ms", "err": null}
INFO	kgoclient	beginning to manage the group lifecycle
INFO	kgoclient	joining group
DEBUG	kgoclient	wrote FindCoordinator v3	{"broker": -2147483648, "bytes_written": 52, "write_wait": "113.8µs", "time_to_write": "80.5µs", "err": null}
DEBUG	kgoclient	read FindCoordinator v3	{"broker": -2147483648, "bytes_read": 59, "read_wait": "69.8µs", "time_to_read": "57.0043ms", "err": null}
DEBUG	kgoclient	reaped connections	{"time_since_last_reap": "20.011064s", "reap_dur": "149.9µs", "num_reaped": 1}

And just like before, restarting the client fixed the problem.

from franz-go.

Sorry, I phrased that confusingly, can you try ef9a16a?

What I meant in that prior comment is that 528f007 was the initial fix, and ef9a16a was a patch on that fix. I do most of my development on a mac, but docker doesn't play too well with a mac, so I when I went to integration test (on a linux laptop), I noticed that my initial bugfix needed a small patch itself.

from franz-go.

Great, ef9a16a did fix the issue. 🎉

from franz-go.

Cool! I'll go ahead and tag this as 0.7.1

from franz-go.

Alright, I've pushed v0.7.1!

from franz-go.

Cool, thanks for the help!

from franz-go.