Giter Club home page Giter Club logo

Comments (6)

zhupeijun avatar zhupeijun commented on September 23, 2024 2

@zmstone
I have figured out that the issue is related to the health check on port 5370 that we implemented outside of the cluster. Upon removing the health check, the error log no longer appears.

from emqx.

zmstone avatar zmstone commented on September 23, 2024

@zhupeijun did such logs repeat until the service restart ?

from emqx.

zhupeijun avatar zhupeijun commented on September 23, 2024

@zmstone yes, it repeats infinitely. but the service seems still alive

uptime    : 1 days, 1 hours, 42 minutes, 1 seconds

from emqx.

zmstone avatar zmstone commented on September 23, 2024

@zhupeijun thank you for your reply.
How many nodes are you trying to cluster, could you share logs from the other nodes which are trying to connect this node?

Although I cannot reproduce the enotconn error code, I am able to get a similar error of code einval.
This seems to be a race condition.

from emqx.

zhupeijun avatar zhupeijun commented on September 23, 2024

@zmstone we have 2 nodes, both printed similar logs.

Node1

2024-01-17T10:29:21.788662+08:00 [error] crasher: initial call: gen_rpc_server:init/1, pid: <0.21589.23>, registered_name: gen_rpc_server_tcp, error: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gen_rpc_sup,<0.2185.0>], message_queue_len: 0, messages: [], links: [#Port<0.991414>,#Port<0.991418>,<0.2186.0>], dictionary: [], trap_exit: false, status: running, heap_size: 4185, stack_size: 28, reductions: 8015; neighbours:
2024-01-17T10:29:21.789082+08:00 [error] Supervisor: {local,gen_rpc_sup}. Context: child_terminated. Reason: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gen_rpc_server_tcp,pid=<0.21589.23>.
2024-01-17T10:29:22.060430+08:00 [error] State machine gen_rpc_server_tcp terminating. Reason: {badmatch,{error,enotconn}}. Stack: [{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]. Last event: {internal,accept}. State: {waiting_for_connection,{state,#Port<0.991419>,tcp,gen_rpc_driver_tcp}}.
2024-01-17T10:29:22.060794+08:00 [error] crasher: initial call: gen_rpc_server:init/1, pid: <0.21591.23>, registered_name: gen_rpc_server_tcp, error: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gen_rpc_sup,<0.2185.0>], message_queue_len: 0, messages: [], links: [#Port<0.991419>,#Port<0.991422>,<0.2186.0>], dictionary: [], trap_exit: false, status: running, heap_size: 4185, stack_size: 28, reductions: 8017; neighbours:
2024-01-17T10:29:22.061251+08:00 [error] Supervisor: {local,gen_rpc_sup}. Context: child_terminated. Reason: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gen_rpc_server_tcp,pid=<0.21591.23>.
2024-01-17T10:29:23.438408+08:00 [error] State machine gen_rpc_server_tcp terminating. Reason: {badmatch,{error,enotconn}}. Stack: [{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]. Last event: {internal,accept}. State: {waiting_for_connection,{state,#Port<0.991423>,tcp,gen_rpc_driver_tcp}}.

Node2

2024-01-17T10:31:24.015710+08:00 [error] crasher: initial call: gen_rpc_server:init/1, pid: <0.18306.23>, registered_name: gen_rpc_server_tcp, error: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gen_rpc_sup,<0.2241.0>], message_queue_len: 0, messages: [], links: [#Port<0.985692>,#Port<0.985696>,<0.2242.0>], dictionary: [], trap_exit: false, status: running, heap_size: 4185, stack_size: 28, reductions: 8017; neighbours:
2024-01-17T10:31:24.016086+08:00 [error] Supervisor: {local,gen_rpc_sup}. Context: child_terminated. Reason: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gen_rpc_server_tcp,pid=<0.18306.23>.
2024-01-17T10:31:24.792451+08:00 [error] State machine gen_rpc_server_tcp terminating. Reason: {badmatch,{error,enotconn}}. Stack: [{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]. Last event: {internal,accept}. State: {waiting_for_connection,{state,#Port<0.985697>,tcp,gen_rpc_driver_tcp}}.
2024-01-17T10:31:24.792745+08:00 [error] crasher: initial call: gen_rpc_server:init/1, pid: <0.18307.23>, registered_name: gen_rpc_server_tcp, error: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gen_rpc_sup,<0.2241.0>], message_queue_len: 0, messages: [], links: [#Port<0.985697>,#Port<0.985699>,<0.2242.0>], dictionary: [], trap_exit: false, status: running, heap_size: 4185, stack_size: 28, reductions: 8015; neighbours:
2024-01-17T10:31:24.793063+08:00 [error] Supervisor: {local,gen_rpc_sup}. Context: child_terminated. Reason: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gen_rpc_server_tcp,pid=<0.18307.23>.
2024-01-17T10:31:25.118951+08:00 [error] State machine gen_rpc_server_tcp terminating. Reason: {badmatch,{error,enotconn}}. Stack: [{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]. Last event: {internal,accept}. State: {waiting_for_connection,{state,#Port<0.985700>,tcp,gen_rpc_driver_tcp}}.

Our environment has no v4 ip and routing

$ ifcofnig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 < ipv6 addr #### >   prefixlen 64  scopeid 0x0<global>
        ether <##### > txqueuelen 0  (Ethernet)

from emqx.

zmstone avatar zmstone commented on September 23, 2024

@zhupeijun
I will need to reproduce this issue.
Could you please share how the ipv6-only environment is created, and how EMQX is started?
It seems you are running EMQX in docker?

from emqx.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.