Comments (11)
I've noticed this as well, this happens for me immediately when I start a cluster. What I know so far is that Kafka actually does not create a group coordinator (nor transaction coordinator) until a group (or txn id) is used at least once. During the period of time it's creating these, it replies that no group coordinator exists.
You can see this by issuing a metadata request against your broker for all topics: the __consumer_offsets
topic will not yet exist.
If you leave the client running, does it eventually recover? I just looked at the franz-go code and I'm actually suspicious that I cache the invalid coordinator (under ID -1). I'll skim this a bit more, but if you can vet that too, that'd be great.
from franz-go.
I left my consumer running for an hour but it didn't recover. On a side note, I noticed that kafka-console-consumer.sh
from the Kafka cli works fine with a new group id.
from franz-go.
Cool, I'm going to patch this quickly, I'm confident the issue is because the invalid node ID is cached permanently.
With that side note, do you mean that the kafka-console-consumer
does not run into this problem when a cluster first starts? If so, yeah I'd expect that, I definitely am seeing a caching bug within the client.
from franz-go.
Correct. I didn't see any issues with kafka-console-consumer
.
from franz-go.
I think 528f007 (and the quick patch from the laptop I can actually test this on, ef9a16a) should fix this. I tested this by turning off my cluster, creating a new cluster, and immediately integration testing -- things pass now. Historically they did not, so thank you for bringing this to my attention in a different way! I always thought this was something else funky going on and that my local setup was busted.
Would you mind giving the latest commit a shot? If it fixes this, I can tag 0.7.1.
from franz-go.
I tried 528f007
locally. I don't see coordinator -1 error anymore, but it seems the client gets stuck while joining the group.
DEBUG kgoclient opening connection to broker {"addr": "kafka:9092", "broker": -2147483648}
DEBUG kgoclient connection opened to broker {"addr": "kafka:9092", "broker": -2147483648}
DEBUG kgoclient issuing api versions request {"broker": -2147483648, "version": 3}
DEBUG kgoclient wrote ApiVersions v3 {"broker": -2147483648, "bytes_written": 29, "write_wait": "184.8µs", "time_to_write": "139.6µs", "err": null}
DEBUG kgoclient read ApiVersions v3 {"broker": -2147483648, "bytes_read": 352, "read_wait": "92.9µs", "time_to_read": "11.524ms", "err": null}
DEBUG kgoclient connection initialized successfully {"addr": "kafka:9092", "broker": -2147483648}
DEBUG kgoclient wrote Metadata v9 {"broker": -2147483648, "bytes_written": 34, "write_wait": "16.057ms", "time_to_write": "256.5µs", "err": null}
DEBUG kgoclient read Metadata v9 {"broker": -2147483648, "bytes_read": 108, "read_wait": "157.5µs", "time_to_read": "3.1607ms", "err": null}
INFO kgoclient beginning to manage the group lifecycle
INFO kgoclient joining group
DEBUG kgoclient wrote FindCoordinator v3 {"broker": -2147483648, "bytes_written": 52, "write_wait": "113.8µs", "time_to_write": "80.5µs", "err": null}
DEBUG kgoclient read FindCoordinator v3 {"broker": -2147483648, "bytes_read": 59, "read_wait": "69.8µs", "time_to_read": "57.0043ms", "err": null}
DEBUG kgoclient reaped connections {"time_since_last_reap": "20.011064s", "reap_dur": "149.9µs", "num_reaped": 1}
And just like before, restarting the client fixed the problem.
from franz-go.
Sorry, I phrased that confusingly, can you try ef9a16a?
What I meant in that prior comment is that 528f007 was the initial fix, and ef9a16a was a patch on that fix. I do most of my development on a mac, but docker doesn't play too well with a mac, so I when I went to integration test (on a linux laptop), I noticed that my initial bugfix needed a small patch itself.
from franz-go.
Great, ef9a16a
did fix the issue. 🎉
from franz-go.
Cool! I'll go ahead and tag this as 0.7.1
from franz-go.
Alright, I've pushed v0.7.1!
from franz-go.
Cool, thanks for the help!
from franz-go.
Related Issues (20)
- Confluent: Unable to create a topic due to policy violation HOT 3
- How to fetch zero records without waiting HOT 1
- GroupTransactSession Close hangs, preventing restart HOT 4
- Use EOS sessions with BlockRebalanceOnPoll HOT 3
- ProduceSync returning "context canceled" error when parent context hasn't been cancelled HOT 5
- Is there a way to limit consumer PollRecords buffered fetches by size? HOT 1
- Question: Why does kgo.StopProducerOnDataLossDetected() exist? HOT 2
- Incorrect documentation regarding sarama partitioner compatibility SaramaCompatHasher HOT 7
- Feature request: expose metadata to the user HOT 2
- Documentation request / question HOT 1
- RequestTimeoutOverhead appears to override Context cancellation HOT 5
- Broker Throttling
- Retry fetched records on handle error HOT 1
- github.com/twmb/franz-go/pkg/sr#Client.AllSchemas 404 not found HOT 3
- Deadlock in Produce() / TryProduce() when kgo.MaxBufferedBytes() is configured HOT 2
- Unable to specify LZ4 compression level HOT 3
- deadLock in produce HOT 2
- Error on kadm/UserSCRAMs calls with redpanda HOT 3
- Question: Is it safe to poll after receiving ErrGroupSession on poll when a broker node restarts in the cluster? HOT 1
- Data Race: concurrent access to recBatch.canFailFromLoadErrs during retry errors HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from franz-go.