Deadlock during Do method about rueidis HOT 9 CLOSED

Sorry for the noise – I reproduced the situation above, and I think the issue is exactly it – deadlock on Centrifugo level when processing PUB/SUB message due to trying to acquire the same lock acquired before Subscribe/Unsubscribe calls.

from rueidis.

Hi @Mikephii,

A client doesn't and shouldn't share a mutex with a dedicated client. However, if you use the same dedicated client inside its own OnMessage callback directly, it will deadlock indeed. If the callback blocks, the whole dedicated client blocks.

from rueidis.

Hi @dangngoctam00,

Using the same client in the message handler directly will indeed block the whole client. You need to at least defer the usage to another goroutine. For example:

	err := client.Receive(context.Background(), client.B().Psubscribe().Pattern("trafficCh").Build(), func(msg rueidis.PubSubMessage) {
		go func() {
			if err := client.Do(timeout, client.B().Hset().Key(fmt.Sprintf(numsMsgHash, turnIp)).FieldValue().FieldValue(allocationId, strconv.Itoa(1)).Build()).Error(); err != nil {
				// error: context deadline exceed
			}	
		}()
	})

from rueidis.

I think I have the idea – I am acquiring the same lock before subscribe and in PUB/SUB message processing. So if the lock acquired, then subscribe called, then message received from PUB/SUB (separate goroutine), again same lock is trying to be acquired thus deadlock happens. I have not done this in previous Centrifugo versions – so this may explain why it worked before.

Do I understand right that if I am not processing PUB/SUB hooks properly (block in processing) - this can cause subscribing Do to hang on channel receive?

from rueidis.

I think I've run into a very similar deadlock

I'm using the Pubsub hooks api and subbing and pubbing dynamically later on

I've got a shared resource with a mutex and so in my on message callback I'll grab that mutex. Problem is if another routing has that resource already it will block, and if that other routine is trying to client.Do it will block there too and deadlock.

I'm assuming this means that the client and dedicated client are locked behind a mutex?

Assuming client has a mutex

A) onMessage callback triggered, client mutex acquired?

B) routine B aquires shared resource mutex

A) onMessage calls fn inside callback that attempts to Aquire shared resource mutex - blocks while holding client mutex

B) routine B tries to client.do() and attempts aquire client mutex but blocks while holding shared resource mutex

I'll test tomorrow if this is what's causing me problems.

from rueidis.

Thank you!

"If the callback blocks, the whole dedicated client blocks."

i think this is whats causing my deadlocks. my callblack is definitely blocking on a mutex that i think will never be released as another goroutine is holding the mutex and blocking on the client.

should be simple enough to fix with smaller lock hold periods

from rueidis.

thanks that was it: making sure im not holding any locks before executing with the dedicated client. i have to aquire a lock in the onMessage but i can make sure im not holding that same lock before using that client to add and remove subs

from rueidis.

hello @rueian , @FZambia , I currently have a case like this without using lock.
I'm using the code like this:

go func() {
    err := redis.Receive(context.Background(), redis.B().Psubscribe().Pattern(trafficCh).Build(), func(msg   rueidis.PubSubMessage) {
         if err := redis.Do(timeout, redis.B().Hset().Key(fmt.Sprintf(numsMsgHash, turnIp)).FieldValue().FieldValue(allocationId, strconv.Itoa(1)).Build()).Error(); err != nil {
             // error: context deadline exceed
         }
    }
}

I see that the command was processed successfully by Redis and there is a response but Do function has not released.
Here is the goroutine dump file
goroutine.txt

Rueidis version: v1.0.41
Client options:

clientOption := rueidis.ClientOption{
		Dialer: net.Dialer{
			Timeout:   time.Millisecond * time.Duration(60000),
			KeepAlive: time.Millisecond * time.Duration(10000),
		},
		InitAddress:  addresses,
		Username:     "",
		Password:     "turnserver",
		SelectDB:     0,
		DisableRetry: true,
}

Do you have any idea. Thank you.

from rueidis.

@rueian thank you very much.

from rueidis.