Comments (11)
The fix mentioned in the comments is not in place yet because it is not complete.
from proxysql.
Is your client using JDBC with auto-reconnect enabled?
from proxysql.
Is your client using JDBC with auto-reconnect enabled?
Do you mean the client our Users or Services are using to access to Proxysql cluster? if that's the case, I'm not sure, as we have many services from different sources and maybe not all them use the same client config.
from proxysql.
I talked to developers and they said:
- for C++ services, they use "Maria DB C Connector 3.3.1"
- for NodeJs services, they use this library https://www.npmjs.com/package/mysql/v/2.18.1
from proxysql.
Hi @cmejiat104 , the important bit in my previous question is actually "with auto-reconnect enabled?"
I have seen this problem (I will describe it shortly) with several customers using JDBC and auto-reconnect enabled, and it is possible that other libraries have the same problem.
For reference, JDBC auto-reconnect documentation says that the feature is not recommended:
https://dev.mysql.com/doc/connector-j/en/connector-j-connp-props-high-availability-and-clustering.html#cj-conn-prop_autoReconnect
Now, what is the issue that crashed ProxySQL?
The client tries to execute a prepared statement that doesn't exist. ProxySQL currently asserts when this happen (this is your backtrace).
When a client tries to execute a prepared statement that doesn't exist?
We have seen many scenarios in which a client is connected to a proxyql instance, it has a prepared statement prepared, then it lose connectivity with proxysql (for whatever reason), it then reconnects (because of autoReconnect
) and tries to execute a prepared statement that doesn't exist anymore: proxysql asserts and crashes.
What happens when you have multiple proxysql instances, and a bugged client with autoReconnect tries to execute a prepared statement that doesn't exist? After crashing the first proxysql, the client will auto-reconnect to the 2nd proxysql instance and try to execute the non-existing prepared statement there too, crashing the 2nd proxysql instance as well.
If you have more proxysql instance, this client will crash them all.
I don't know the exact implementation in the libraries you are using, but if you have autoReconnect (or an equivalent named property) please disable it.
For further reference, auto-reconnect is now deprecated also in MySQL C API:
https://dev.mysql.com/doc/c-api/8.0/en/c-api-auto-reconnect.html
That said, I acknowledge that this must also be considered a serious bug in proxysql, because a buggy application/driver shouldn't lead to a proxysql crash.
We have this in our todo list.
from proxysql.
Thank you for your replies.
Unfortunately, modifications in Services' client is not possible at this moment, this is a Prod environment.
We have a UAT environment that was also impacted; although, in this case we only have two instances of ProxySQL, proxysql01 and proxysql02, but Mysql backend has three nodes (mysql01, mysql02 and mysql03) as Prod. In the case of UAT, proxysql01 crashed only once and proxysql02 twice, after that both stayed up and running; of course when this happened, for UAT was out of business hours, so only a few batch processes were running, but for Prod was in the middle of peak hours.
During the network issue, connections to & from proxysql02 and mysql02 dropped (between them were fine), in the case of Prod mysql02 was a Mysql Replica and for UAT mysql02 was the Master. BTW, I don't have any query rule configured, so all the transactions go always to the Mysql Master.
from proxysql.
Might this another issue also related to my Proxysql crashes?
from proxysql.
It is the same issue, yes
from proxysql.
Ok thanks!, but the fix mentioned in the comments is not in place yet, right?
from proxysql.
This issue is closed by #4481
from proxysql.
Thank you!
from proxysql.
Related Issues (20)
- Question : when a node from reader hostgroup and lost his READ_ONLY HOT 1
- Implement setting in `mysql_hostgroup_attributes` to override global `mysql-monitor_slave_lag_when_null`
- proxySQL can not properly distribute slow queries after linux update HOT 2
- Unable to parse query. If correct, report it as a bug: SET SESSION wsrep_sync_wait=? HOT 2
- [Question] Does ProxySQL Provide Metrics for User Maximum Connections and Current Connection Usage? HOT 2
- Evaluate exporting `stats_mysql_users` using the Prometheus exporter HOT 3
- Every uneven traffic distribution for nodes with same weight after upgrading from v2.0.10-1 to v2.5.5-1
- Ever increasing 'stats_mysql_query_digest' results in sporadic client timings HOT 5
- Official documentation has missing introduced version information for mysql-connection_warming
- Implement setting in `mysql_hostgroup_attributes` to define custom load balancing algorithm
- ProxySQL errorlog not getting picked up from proxysql.cnf in docker HOT 5
- crashes during cluster shutdown in CI testing HOT 1
- Support some form of partial cleanup of `stats_mysql_query_digest`
- Shunned writer node not becoming online HOT 8
- Intermittent "Access denied" errors HOT 1
- Different server_version per port
- Wrong column order in table mysql_servers HOT 3
- aarch64 / i686 builds fail of v2.6.2 (narrowing conversion of timeout in clickhouse) HOT 2
- Can not set servers_defaults field of mysql_hostgroup_attributes table from config file HOT 4
- Clients seeing "Lost connection to MySQL server during query" errors HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from proxysql.