Comments (9)
Sorry for the noise – I reproduced the situation above, and I think the issue is exactly it – deadlock on Centrifugo level when processing PUB/SUB message due to trying to acquire the same lock acquired before Subscribe/Unsubscribe calls.
from rueidis.
Hi @Mikephii,
A client doesn't and shouldn't share a mutex with a dedicated client. However, if you use the same dedicated client inside its own OnMessage callback directly, it will deadlock indeed. If the callback blocks, the whole dedicated client blocks.
from rueidis.
Hi @dangngoctam00,
Using the same client in the message handler directly will indeed block the whole client. You need to at least defer the usage to another goroutine. For example:
err := client.Receive(context.Background(), client.B().Psubscribe().Pattern("trafficCh").Build(), func(msg rueidis.PubSubMessage) {
go func() {
if err := client.Do(timeout, client.B().Hset().Key(fmt.Sprintf(numsMsgHash, turnIp)).FieldValue().FieldValue(allocationId, strconv.Itoa(1)).Build()).Error(); err != nil {
// error: context deadline exceed
}
}()
})
from rueidis.
I think I have the idea – I am acquiring the same lock before subscribe and in PUB/SUB message processing. So if the lock acquired, then subscribe called, then message received from PUB/SUB (separate goroutine), again same lock is trying to be acquired thus deadlock happens. I have not done this in previous Centrifugo versions – so this may explain why it worked before.
Do I understand right that if I am not processing PUB/SUB hooks properly (block in processing) - this can cause subscribing Do to hang on channel receive?
from rueidis.
I think I've run into a very similar deadlock
I'm using the Pubsub hooks api and subbing and pubbing dynamically later on
I've got a shared resource with a mutex and so in my on message callback I'll grab that mutex. Problem is if another routing has that resource already it will block, and if that other routine is trying to client.Do it will block there too and deadlock.
I'm assuming this means that the client and dedicated client are locked behind a mutex?
Assuming client has a mutex
A) onMessage callback triggered, client mutex acquired?
B) routine B aquires shared resource mutex
A) onMessage calls fn inside callback that attempts to Aquire shared resource mutex - blocks while holding client mutex
B) routine B tries to client.do() and attempts aquire client mutex but blocks while holding shared resource mutex
I'll test tomorrow if this is what's causing me problems.
from rueidis.
Thank you!
"If the callback blocks, the whole dedicated client blocks."
i think this is whats causing my deadlocks. my callblack is definitely blocking on a mutex that i think will never be released as another goroutine is holding the mutex and blocking on the client.
should be simple enough to fix with smaller lock hold periods
from rueidis.
thanks that was it: making sure im not holding any locks before executing with the dedicated client. i have to aquire a lock in the onMessage but i can make sure im not holding that same lock before using that client to add and remove subs
from rueidis.
hello @rueian , @FZambia , I currently have a case like this without using lock.
I'm using the code like this:
go func() {
err := redis.Receive(context.Background(), redis.B().Psubscribe().Pattern(trafficCh).Build(), func(msg rueidis.PubSubMessage) {
if err := redis.Do(timeout, redis.B().Hset().Key(fmt.Sprintf(numsMsgHash, turnIp)).FieldValue().FieldValue(allocationId, strconv.Itoa(1)).Build()).Error(); err != nil {
// error: context deadline exceed
}
}
}
I see that the command was processed successfully by Redis and there is a response but Do
function has not released.
Here is the goroutine dump file
goroutine.txt
Rueidis version: v1.0.41
Client options:
clientOption := rueidis.ClientOption{
Dialer: net.Dialer{
Timeout: time.Millisecond * time.Duration(60000),
KeepAlive: time.Millisecond * time.Duration(10000),
},
InitAddress: addresses,
Username: "",
Password: "turnserver",
SelectDB: 0,
DisableRetry: true,
}
Do you have any idea. Thank you.
from rueidis.
@rueian thank you very much.
from rueidis.
Related Issues (20)
- "Unknown command" error when connecting to Redis cluster with authentication HOT 3
- Add IsParseErr helper HOT 8
- Initialize the Redis connection lazily HOT 2
- Remove the`oklog/ulid` dependency from the `rueidisaside` submodule HOT 7
- Replace SET NX GET in rueidisaside with Lua when using Redis < 7 HOT 5
- Fallback MGetCache to MGet when DisableCache=true HOT 8
- Add IsParseError() documentation HOT 2
- Send `DISCARD` automatically when a dedicated connection is recycled
- Make DedicatedClient.Do return errors after being recycled HOT 3
- Error updating the cluster architecture HOT 25
- Index Out of Range issue in DoMultiCache function HOT 5
- Sample code for Generic Object Mapping hangs indefinitely HOT 3
- rueidislock CPU highload HOT 13
- Need to improve the test coverage on rueidiscompat HOT 1
- Add HExpire, HPExpire, HTTL, and HPTTL functions to rueidiscompat
- Add FTSearch, FTCreate, and FTDropIndex functions to rueidiscompat HOT 3
- Could Hset example be added to cheatsheet ? HOT 2
- Hashrepository with nested json, indexes and search HOT 4
- Client-side caching with Amazon ElastiCache Redis OSS HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rueidis.