Comments (8)
I believe the correct fix for this is:
- have
reenqueue
return non-nil whenmsg.retried
so we know it hasn't been delivered - have
flushRequest
callerrorCb(nil)
whenoverlimit == 0
On inspection there are two other issues:
- duplicate of this code around line 378 of producer.go, as above
- if the response is well formed but empty (no blocks, or blocks but no partitions) then we return without ever calling errorCb - to fix this we should probably set a flag when we see a block-partition, and call errorCb if the flag is unset after the main loop
from sarama.
@burke @fw42 @wvanbergen does this make sense to you?
from sarama.
I am not quite sure what the initial problem was here? :/
from sarama.
Yeah ditto, no clue what this is about :-) Can you elaborate?
from sarama.
It's a deadlock, read the inserted comments in the code block in order (1, 2, 3). #90 also includes a stack trace which I believe is the same issue.
from sarama.
Basically: enqueue
expects that flushRequest calls errorCb
exactly once, so that there is something on the errs
channel for it to return. There are some cases where we never call errorCb
(or call errorCb
multiple times) which both cause a deadlock reading or writing to the errs
channel.
from sarama.
OK, now it makes sense to me. Your solutions makes sense to me. I wonder whether we can build tests around this.
from sarama.
https://groups.google.com/forum/#!topic/sarama-users/8smSO57Ubzg found a way to consistently reproduce the hang, but it requires some dns hackery - I will use that to at least test a local copy of the fix
from sarama.
Related Issues (20)
- unaligned 64-bit atomic operation on i386 HOT 4
- Bug while adding new ACLs HOT 5
- sarama.CreateAclsRequest: Ignoring TextMarshaler & TextUnmarshaler for members HOT 1
- Kafka: client has run out of available brokers HOT 3
- Should share the max retries of metadata? HOT 1
- Topic retention settings causes consumer to skip consuming new messages that arrived after deleted messages HOT 5
- A topic with multiple partitions won't get dedicated consumer members in consumer group for each partition HOT 6
- kafka: Failed to produce message to topic myRsp: write tcp [ip1]:[port1]->[ip2]:[port2]: write: broken pipe HOT 7
- Enabling idempotency still allows producing same message
- No throughput difference between sync and async producers HOT 1
- High CPU usage in `newBrokerProducer.func2` HOT 3
- Incorrect rate5 metrics values
- Error and data race using transaction example from the library HOT 2
- kafka server: The producer attempted to update a transaction while another concurrent operation on the same transaction was ongoing HOT 3
- Will handling response this way result in subsequent requests all being mishandled? HOT 1
- unsubscribe w/o passing whole list HOT 2
- Sincerely ask, is this redundant? HOT 1
- How can consumer group consume new partition without losing data using OffsetNewest config?
- Setting a too high value for Producer.Retry.Max can lead to excessive memory usage.
- Docs with examples using tls/ssl by cert files creating a producer and consumer HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sarama.