runq_overload alert on using MongoDB for authz/authn and also alert gets stuck for days sometimes about emqx HOT 9 OPEN

I do not believe this is a bug, hence we should continue the discussion in the original discussion thread:
#13188

from emqx.

just to be clear, runq_overload is for monitoring/alerting.
We do not expect runq_overload alert from a healthy system.
We do expect raising/clearing runq_overload alert on a resource tight system and it is a sign of resource saturation.

This issue report is about runq_overload alert is not cleared when the system is not overloaded. refer to #13188 (reply in thread)

from emqx.

Yes, but also I feel like there is some problem with mongoDB authn/authz plugins, because when I was using EMQX 4, there was no issue like this. Even when I scaled up Im getting alerts only when I enable authn and authz on MongoDB only. Please check if there is any issue in auth plugins with MongoDB, this is very critical for our system.

from emqx.

@chaymankala Pls create another issue/discussion for performance issue regarding mongoDB and provide your configuration to help understand if it is a concern

from emqx.

V4 has no runq overload alarms.
Is there monitoring data to support the hypothesis that v5 has performance degraded?

from emqx.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

from emqx.

Okay, you are saying that runq_overload might have happened in v4 too, but as we dont have any alerting, we didnt see that. I get it. But why am I getting runq_overload and its not clearing from alerts. Its been in alerting state since 45 days?

And as of today, whenever I run this command I get 0

emqx eval 'length(lists:filter(fun(Pid) -> case process_info(Pid,current_function) of {current_function,{qlc,wait_for_request,3}} -> true; _ -> false end end, processes())).'

from emqx.

@chaymankala, I just want to add that in v4, you don't get alarm but you get logging printout in log file. That you could have a check if you have same perf issue.

For the alerting not cleared, I cannot reproduce it locally but I will create another ticket just track such issue.

from emqx.

We dont have the v4 setup right now, please create another ticket to track the not clearing issue

from emqx.