proxySQL can not properly distribute slow queries after linux update about proxysql HOT 2 OPEN

Hi @ukyanj0 .
Thank you for being part of our community!

Do you have evidence that ProxySQL is sending more traffic to DB 4,5 ?

proxySQL catches that DB 4,5 have better specification

No, ProxySQL doesn't check backend specification, but based on response time it may act differently.
If multiplexing is enabled, ProxySQL tends to send more traffic to the servers that respond faster to queries because they have free connections.
But in your case, it seems that:

• multiplexing get disabled
• DB 4,5 is actually slower

Therefore I do not think that ProxySQL is sending load balancing more traffic to DB 4,5 .
But because your traffic disables multiplexing, it is possible that just few connections are responsible for most of the load on DB 4,5 .

On the other end, the load average on DB 4,5 is very small. The highest read is 0.43 on 5m average.
The server DB 4,5 is not overloaded at all!
It is "only" swapping a lot.

I suspect that during the OS upgrade one of two things happened:

1. your database server configuration was changed too (for example, the InnoDB buffer pool size was changed)
2. the available RAM was reduced

If I should make a guess, I would guess that RAM was reduced.
A server with 32 CPU cores and only 16GB of RAM is a very weird configuration: too little RAM for a server with 32 cores.

So far, I don't believe this is a ProxySQL issue, neither a MariaDB issue. I suspect it is either a misconfiguration, or someone removed some RAM.

Thanks,
René

from proxysql.

It is my bad that I didn't make it clear that we only updated WAS, not DB server.
WAS is now Rocky Linux 8/MariaDB 10.11/ProxySQL 2.3.2, while DB server is CentOS 7/MaridDB 10.0 and there is very little chance we changed DB server config. The multiplexing option seems to be off as you said.

By the way the disk IO increases to a few hundreads on DB4,5 from time to time that might be caused by high-core/low-memory as you said. We will check it first then figure out if there is actual traffic monitoring tools.
Thank you for your support.

Sincerely
Ukyan

from proxysql.