[Bug]: netdata plugins segfaults after update 1.45.3 to 1.45.4 about netdata HOT 7 CLOSED

I wasn't able to reproduce this issue as well, but I have spent some time with the Code line from dmesg yesterday before logs and coredumps were available:

[Wed May 15 20:41:19 2024] Code: e8 e7 8d 03 00 48 83 ec 08 4c 89 f6 4c 89 ef 8b 54 24 14 44 0f b6 c8 52 4c 8b 44 24 28 4c 89 fa 48 8b 4c 24 20 e8 52 7c 02 00 <80> 60 68 fd 49 89 c6 58 5a 4d 85 e4 0f 84 40 01 00 00 41 80 3c 24

It disassembles to this:

 0x00000000      e8e78d0300     call 0x38dec
 0x00000005      4883ec08       sub rsp, 8
 0x00000009      4c89f6         mov rsi, r14
 0x0000000c      4c89ef         mov rdi, r13
 0x0000000f      8b542414       mov edx, dword [rsp + 0x14]
 0x00000013      440fb6c8       movzx r9d, al
 0x00000017      52             push rdx
 0x00000018      4c8b442428     mov r8, qword [rsp + 0x28]
 0x0000001d      4c89fa         mov rdx, r15
 0x00000020      488b4c2420     mov rcx, qword [rsp + 0x20]
 0x00000025      e8527c0200     call 0x27c7c
>0x0000002a      806068fd       and byte [rax + 0x68], 0xfd ; [0xfd:1]=255 ; 253
 0x0000002e      4989c6         mov r14, rax
 0x00000031      58             pop rax
 0x00000032      5a             pop rdx

I tried to search in my netdata binary and I have found only one instruction with the same parameters:

 0x00199a08      e8e38f0300     call dbg.rrd_algorithm_id
 0x00199a0d      4883ec08       sub rsp, 8
 0x00199a11      4c89fe         mov rsi, r15                ; char *arg2
 0x00199a14      4c89f7         mov rdi, r14                ; int64_t arg1
 0x00199a17      4155           push r13
 0x00199a19      4c8b442428     mov r8, qword [var_18h]     ; int64_t arg5
 0x00199a1e      440fb6c8       movzx r9d, al               ; int64_t arg_50h
 0x00199a22      488b4c2420     mov rcx, qword [var_10h]    ; int64_t arg4
 0x00199a27      488b542418     mov rdx, qword [var_8h]     ; int64_t arg3
 0x00199a2c      e8ef7c0200     call dbg.rrddim_add_custom
>0x00199a31      806068fd       and byte [rax + 0x68], 0xfd ; [0xfd:1]=0
 0x00199a35      4989c5         mov r13, rax
 0x00199a38      58             pop rax
 0x00199a39      5a             pop rdx

It is from pluginsd_dimension function, it crashes on this line

everything fits, 0xfd represents ~(1 << 1), where (1 << 1) is value of RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS. 0x68 is offset of collector.options in rd structure and the rd pointer is stored in rax register.
rrddim_add (rrddim_add_custom in the assembly) can return NULL in some situations, but this case is not checked here and this is the reason of the crash. At least I think so :)

from netdata.

Hi, @ekexcello. As you can guess, the information provided is not enough to identify the problem.

• Does Netdata crash if you disable your plugin?
• Can you provide the output of your plugin? Multiple data collection intervals will do.
• Can you get coredump?

from netdata.

Hello.
Sorry for the delay.
Netdata doesn't crash with plugins disabled. Will provide other details soon

from netdata.

Please find plugin output and core dumps pid 19500 and pid 26213
Note, uncompressed they will have significant size.

from netdata.

While taking core dumps it was found that another custom plugin we have also crashes, but it doesn't render netdata itself to crash. So the problem seems to be a bit more wide.

from netdata.

Hey, @ekexcello. The problem is missing DIMENSION id

CHART qmail.limit_maxconnip '' 'Qmail SMTPD maxconnip limit' '# reaches' 'tcpserver' 'qmail.qmail_smtpd_limits' line
DIMENSION  '' absolute 1 1

BEGIN qmail.limit_maxconnip 1000016
SET  = 1
END

The fact that Netdata crashes is a bug, we will fix it. After the fix, Netdata will stop your plugin because of incorrect output, so you'll need to fix this.

from netdata.

RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS. 0x68 is offset of collector.options in rd structure and the rd pointer is stored in rax register. rrddim_add (rrddim_add_custom in the assembly) can return NULL in some situations, but this case is not checked here and this is the reason of the crash. At least I think so :)

@arkamar You are correct !!! Thanks for the detailed investigation (and the assembly code 😃 )

from netdata.