Comments (10)
atbe
commented on May 16, 2024
Note: this is beginning to happen much more frequently since server version v0.5.1
Even on server reboots I'm getting clients stuck connecting.
DMing @davidzhao a url with a token that he can use to reproduce.
from livekit.
atbe
commented on May 16, 2024
Hold off on debugging this. I believe it might be an issue on our side.
from livekit.
atbe
commented on May 16, 2024
Can confirm that this issue is occurring very frequently on server version v0.5.1
from livekit.
atbe
commented on May 16, 2024
Switching the server to v0.5.0 resolved the issue.
from livekit.
atbe
commented on May 16, 2024
Due to misconfiguration on our end.
from livekit.
atbe
commented on May 16, 2024
Reopening this issue as its happening more frequently now.
@davidzhao believes it could be related to the multi-node configuration where a room is routed to a node that might no longer exist.
On a different note, we are seeing the following in our livekit-server logs quite often and I believe it might be related to this issue:
"<14>1 2021-02-20T17:44:20.139749Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
--
"<14>1 2021-02-20T17:44:20.139275Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:20.140081Z - - - - - 2021-02-20T17:44:20.139Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:20.140117Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:20.139235Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:20.139185Z - - - - - 2021-02-20T17:44:20.139Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:20.140110Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:21.140547Z - - - - - 2021-02-20T17:44:21.140Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:21.140613Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:21.141449Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:21.141440Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:21.140808Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:21.140605Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:21.141415Z - - - - - 2021-02-20T17:44:21.141Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:21.140787Z - - - - - 2021-02-20T17:44:21.140Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:21.140828Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:21.141191Z - - - - - 2021-02-20T17:44:21.140Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:21.141352Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:21.642852Z - - - - - 2021-02-20T17:44:21.642Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:21.141346Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:21.141325Z - - - - - 2021-02-20T17:44:21.141Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:21.642915Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:21.141241Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:21.141250Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:21.642925Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:21.642979Z - - - - - 2021-02-20T17:44:21.642Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:21.643000Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:21.643007Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:21.642470Z - - - - - 2021-02-20T17:44:21.642Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:21.642539Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:21.643478Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:21.643443Z - - - - - 2021-02-20T17:44:21.643Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:21.642547Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:21.643471Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:21.643211Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:21.643204Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:21.643168Z - - - - - 2021-02-20T17:44:21.643Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:22.645568Z - - - - - 2021-02-20T17:44:22.645Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:22.646530Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:22.645620Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:22.645628Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:22.646539Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:22.646499Z - - - - - 2021-02-20T17:44:22.646Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:22.645795Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:22.645716Z - - - - - 2021-02-20T17:44:22.645Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:22.645784Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:23.147478Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:22.645871Z - - - - - 2021-02-20T17:44:22.645Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:22.646123Z - - - - - 2021-02-20T17:44:22.645Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:22.645961Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:23.147448Z - - - - - 2021-02-20T17:44:23.147Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:23.147429Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:22.646140Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:23.147341Z - - - - - 2021-02-20T17:44:23.147Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:23.147415Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:23.147486Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:22.645953Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:22.646148Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:23.148287Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:23.148276Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:23.148240Z - - - - - 2021-02-20T17:44:23.148Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:23.147933Z - - - - - 2021-02-20T17:44:23.147Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:23.147993Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:23.148079Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:23.148087Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:23.649668Z - - - - - 2021-02-20T17:44:23.649Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:23.148006Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:23.148050Z - - - - - 2021-02-20T17:44:23.147Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:23.650520Z - - - - - 2021-02-20T17:44:23.650Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:23.650555Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:23.650547Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:23.649740Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:23.649731Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:23.649814Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:23.649766Z - - - - - 2021-02-20T17:44:23.649Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:24.151637Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:23.650261Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:23.650005Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:24.151015Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:23.649827Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:24.151024Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:24.151602Z - - - - - 2021-02-20T17:44:24.151Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:23.649995Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:23.650226Z - - - - - 2021-02-20T17:44:23.650Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:23.649965Z - - - - - 2021-02-20T17:44:23.649Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:23.650251Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:24.150957Z - - - - - 2021-02-20T17:44:24.150Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:24.151645Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:24.151763Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:24.151737Z - - - - - 2021-02-20T17:44:24.151Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:24.151754Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:24.416352Z - - - - - 2021-02-20T17:44:24.416Z\tINFO\tserver/main.go:169\texit requested, shutting down\t{\"signal\": \"terminated\"}\n"
"<14>1 2021-02-20T17:44:24.416564Z - - - - - 2021-02-20T17:44:24.416Z\tDEBUG\trouting/redisrouter.go:286\tfinishing redisWorker\t{\"node\": \"ND_GN-HXa7T\"}\n"
"<14>1 2021-02-20T17:44:24.416399Z - - - - - 2021-02-20T17:44:24.416Z\tDEBUG\trouting/redisrouter.go:239\tstopping RedisRouter\n"
"<14>1 2021-02-20T17:44:24.151471Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:24.151264Z - - - - - 2021-02-20T17:44:24.151Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
"<14>1 2021-02-20T17:44:24.151464Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:24.151325Z - - - - - \t/workspace/pkg/routing/redisrouter.go:305\n"
"<14>1 2021-02-20T17:44:24.151317Z - - - - - github.com/livekit/livekit-server/pkg/routing.(*RedisRouter).redisWorker\n"
"<14>1 2021-02-20T17:44:24.151445Z - - - - - 2021-02-20T17:44:24.151Z\tERROR\trouting/redisrouter.go:305\terror processing signal message\t{\"error\": \"channel is full\"}\n"
from livekit.
atbe
commented on May 16, 2024
Another note on this.
When I attempted to connect to room A, the issue occurred.
Then I attempted to connect to room B and the connection worked fine.
The major difference I found between the two attempts was the server log here:
Here are the logs when attemption to connect to room A
2021-02-20T22:44:23.717Z INFO server/main.go:145 configured key provider {"num_keys": 1}
2021-02-20T22:44:23.830Z INFO server/main.go:178 using multi-node routing via redis {"address": "livespot-lk-redis-production:6379"}
2021-02-20T22:44:23.834Z DEBUG routing/redisrouter.go:288 starting redisWorker {"node": "ND_0wBdyiA9"}
2021-02-20T22:44:23.834Z INFO service/server.go:110 starting LiveKit server {"address": ":7880", "nodeId": "ND_0wBdyiA9", "version": "0.5.1"}
2021-02-20T22:47:10.196Z INFO service/rtcservice.go:113 new client WS connected {"connectionId": "CO_X4n4KvLiBLp3", "room": "RM_iSavWZzUm6qU", "roomName": "8fb5bc1e-1918-4240-9e05-b1c01420152c", "name": "0xFFC80bd2A413f37E125D39C281Cc85B88dcebF20"}
2021-02-20T22:47:28.587Z INFO service/rtcservice.go:97 WS connection closed {"participant": "0xFFC80bd2A413f37E125D39C281Cc85B88dcebF20", "connectionId": "CO_X4n4KvLiBLp3"}
Then right after I attempted to connect to room B
Here are the logs following the attempt to connect to room B
2021-02-20T22:47:32.051Z DEBUG service/roommanager.go:78 selected node for room {"room": "a62ac02a-54e4-4254-9667-989c40855d24", "node": "ND_0wBdyiA9"}
2021-02-20T22:47:32.052Z INFO service/rtcservice.go:113 new client WS connected {"connectionId": "CO_sTD2tGrYtYJn", "room": "RM_ZR6Smi97AmkN", "roomName": "a62ac02a-54e4-4254-9667-989c40855d24", "name": "0xFFC80bd2A413f37E125D39C281Cc85B88dcebF20"}
2021-02-20T22:47:32.053Z DEBUG service/roommanager.go:176 starting RTC session {"room": "a62ac02a-54e4-4254-9667-989c40855d24", "node": "ND_0wBdyiA9", "participant": "0xFFC80bd2A413f37E125D39C281Cc85B88dcebF20", "num_participants": 0}
Notice the message 2021-02-20T22:47:32.051Z DEBUG service/roommanager.go:78 selected node for room {"room": "a62ac02a-54e4-4254-9667-989c40855d24", "node": "ND_0wBdyiA9"} in the second set of logs.
This is all with a single node running.
from livekit.
davidzhao
commented on May 16, 2024
the logs on signal connection not found is interesting.. though it shouldn't be the cause of the issue.
strangely, in the logs that you've dumped. for room A I'm not seeing anything regarding: selected node for room.
this kind of confirms my theory that the room was an old room, and was assigned to a different node_id that no longer exists. I think the node selection logic could be a bit more robust in checking if the node still exists when a new participant is connecting.
from livekit.
atbe
commented on May 16, 2024
this kind of confirms my theory that the room was an old room, and was assigned to a different node_id that no longer exists. I think the node selection logic could be a bit more robust in checking if the node still exists when a new participant is connecting.
Yes, I agree.
from livekit.
davidzhao
commented on May 16, 2024
I have a fix for this in 0.5.3, it passed my tests for this scenario. Give it a shot!
from livekit.
Related Issues (20)
- On Android device, when a user first joining the room, its resolution is very low until 10-15 seconds. HOT 1
- VM Deploy Requires Domain HOT 1
- Share screen functionality not working while building a client application using livekit HOT 1
- audio issue if user dont have ipv6 HOT 4
- Livekit randomly crashing in between meetings HOT 1
- Docker hub has incorrect tag v1.10.6 published HOT 1
- How to get remote is not presenting? HOT 1
- request 'DeleteRoom Service' timeout HOT 2
- LiveKit runs into timeouts after a Redis restart HOT 7
- Could not fetch region settings: livekit HOT 2
- Missing Change Speaker Functionality HOT 1
- api_key is required to use webhooks HOT 1
- Connection failing on Safari when connecting via TCP HOT 1
- No documentation about what redis ACL rules are needed by LiveKit server
- 1.5.3 release was not completed HOT 2
- LiveKit and redis > 7 support
- FEATURE REQUEST HOT 2
- Noticeable lag when trying to share a local video window HOT 3
- How to switch to the rear camera of the mobile phone instead of using the default front camera of the mobile phone HOT 1
- ERROR livekit routing/redisrouter.go:225 status update delayed, possible deadlock HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from livekit.