Comments (6)
@zmstone
I have figured out that the issue is related to the health check on port 5370 that we implemented outside of the cluster. Upon removing the health check, the error log no longer appears.
from emqx.
@zhupeijun did such logs repeat until the service restart ?
from emqx.
@zmstone yes, it repeats infinitely. but the service seems still alive
uptime : 1 days, 1 hours, 42 minutes, 1 seconds
from emqx.
@zhupeijun thank you for your reply.
How many nodes are you trying to cluster, could you share logs from the other nodes which are trying to connect this node?
Although I cannot reproduce the enotconn
error code, I am able to get a similar error of code einval
.
This seems to be a race condition.
from emqx.
@zmstone we have 2 nodes, both printed similar logs.
Node1
2024-01-17T10:29:21.788662+08:00 [error] crasher: initial call: gen_rpc_server:init/1, pid: <0.21589.23>, registered_name: gen_rpc_server_tcp, error: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gen_rpc_sup,<0.2185.0>], message_queue_len: 0, messages: [], links: [#Port<0.991414>,#Port<0.991418>,<0.2186.0>], dictionary: [], trap_exit: false, status: running, heap_size: 4185, stack_size: 28, reductions: 8015; neighbours:
2024-01-17T10:29:21.789082+08:00 [error] Supervisor: {local,gen_rpc_sup}. Context: child_terminated. Reason: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gen_rpc_server_tcp,pid=<0.21589.23>.
2024-01-17T10:29:22.060430+08:00 [error] State machine gen_rpc_server_tcp terminating. Reason: {badmatch,{error,enotconn}}. Stack: [{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]. Last event: {internal,accept}. State: {waiting_for_connection,{state,#Port<0.991419>,tcp,gen_rpc_driver_tcp}}.
2024-01-17T10:29:22.060794+08:00 [error] crasher: initial call: gen_rpc_server:init/1, pid: <0.21591.23>, registered_name: gen_rpc_server_tcp, error: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gen_rpc_sup,<0.2185.0>], message_queue_len: 0, messages: [], links: [#Port<0.991419>,#Port<0.991422>,<0.2186.0>], dictionary: [], trap_exit: false, status: running, heap_size: 4185, stack_size: 28, reductions: 8017; neighbours:
2024-01-17T10:29:22.061251+08:00 [error] Supervisor: {local,gen_rpc_sup}. Context: child_terminated. Reason: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gen_rpc_server_tcp,pid=<0.21591.23>.
2024-01-17T10:29:23.438408+08:00 [error] State machine gen_rpc_server_tcp terminating. Reason: {badmatch,{error,enotconn}}. Stack: [{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]. Last event: {internal,accept}. State: {waiting_for_connection,{state,#Port<0.991423>,tcp,gen_rpc_driver_tcp}}.
Node2
2024-01-17T10:31:24.015710+08:00 [error] crasher: initial call: gen_rpc_server:init/1, pid: <0.18306.23>, registered_name: gen_rpc_server_tcp, error: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gen_rpc_sup,<0.2241.0>], message_queue_len: 0, messages: [], links: [#Port<0.985692>,#Port<0.985696>,<0.2242.0>], dictionary: [], trap_exit: false, status: running, heap_size: 4185, stack_size: 28, reductions: 8017; neighbours:
2024-01-17T10:31:24.016086+08:00 [error] Supervisor: {local,gen_rpc_sup}. Context: child_terminated. Reason: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gen_rpc_server_tcp,pid=<0.18306.23>.
2024-01-17T10:31:24.792451+08:00 [error] State machine gen_rpc_server_tcp terminating. Reason: {badmatch,{error,enotconn}}. Stack: [{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]. Last event: {internal,accept}. State: {waiting_for_connection,{state,#Port<0.985697>,tcp,gen_rpc_driver_tcp}}.
2024-01-17T10:31:24.792745+08:00 [error] crasher: initial call: gen_rpc_server:init/1, pid: <0.18307.23>, registered_name: gen_rpc_server_tcp, error: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gen_rpc_sup,<0.2241.0>], message_queue_len: 0, messages: [], links: [#Port<0.985697>,#Port<0.985699>,<0.2242.0>], dictionary: [], trap_exit: false, status: running, heap_size: 4185, stack_size: 28, reductions: 8015; neighbours:
2024-01-17T10:31:24.793063+08:00 [error] Supervisor: {local,gen_rpc_sup}. Context: child_terminated. Reason: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gen_rpc_server_tcp,pid=<0.18307.23>.
2024-01-17T10:31:25.118951+08:00 [error] State machine gen_rpc_server_tcp terminating. Reason: {badmatch,{error,enotconn}}. Stack: [{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]. Last event: {internal,accept}. State: {waiting_for_connection,{state,#Port<0.985700>,tcp,gen_rpc_driver_tcp}}.
Our environment has no v4 ip and routing
$ ifcofnig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet6 < ipv6 addr #### > prefixlen 64 scopeid 0x0<global>
ether <##### > txqueuelen 0 (Ethernet)
from emqx.
@zhupeijun
I will need to reproduce this issue.
Could you please share how the ipv6-only environment is created, and how EMQX is started?
It seems you are running EMQX in docker?
from emqx.
Related Issues (20)
- Auto subscribe based on client id pattern HOT 4
- 管理端访问 主题 带 / 的功能时报错 HOT 2
- Emqx manual clustering fail when windows emqx broker attempting to join the linux broker HOT 4
- AWK not found in docker HOT 3
- cannot pass parameters in connector with url or path HOT 5
- closed HOT 1
- Add option to configure pod labels via Helm Chart HOT 3
- The is_superuser field set by the JWT authenticator is invalid. HOT 2
- emqx crashing on startup after upgrade to 5.8.0 HOT 11
- websocket terminated due to fail in matching function HOT 7
- Node Crash Continuously in a Cluster of 3 Nodes Due to emqx_dashboard_listener Failure HOT 2
- The 5.8.0 version of emqx crashes upon startup HOT 2
- The webhook page doesn't work when accessing emqx's web panel through a reverse proxy HOT 19
- Confusing response message when publishing message to a denied topic HOT 3
- use image emqx:5 in20240909 HOT 1
- Can not parse float data with 13 digits after the decimal point like: 31.3338467752013 in rule sql expression HOT 3
- v5.8.0 WS connect kepp disconnect Befroe v5.7.2 is good. HOT 4
- All connections of emqx single point are frequently interrupted and reconnected HOT 2
- shared_subscription_strategy: local 共享策略问题 HOT 2
- Http 连接器创建页面,TLS 显示是关闭,但实际 是开启的,导致为配置证书的情况下,连通性测试一直不过 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from emqx.