MysqlRestarted will never fire about awesome-prometheus-alerts HOT 4 CLOSED

This alert has a misleading name as the whole purpose is to notify the operator when mysql is crash looping and not just restarting (restart is a normal thing and shouldn't be alerted on). In crashlooping scenario mysql_global_status_uptime will rise from 0 to some low value, but due to mysql restart it will be reset to 0 and the cycle will repeat. If the cycle repeats itself for longer time (for: 5m), then it should be of concern to the operator.

Essentially, crash looping mysql will cause a saw-tooth pattern on mysql_global_status_uptime where metric never goes 60 (set threshold).

Note: You should never alert on things that are normal to the operation of the application. So creating alerts that will always fire because application restarted once is a no-go.

from awesome-prometheus-alerts.

I agree with your comment @paulfantom, except for 1 thing:

I don't care if a simple API instance restarts. But in a 3 tier architecture, a MySQL short downtime can lead to a major outage. It should not be considered as "normal" and more investigations are advised.

from awesome-prometheus-alerts.

In such a scenario, info alert is fine, but not anything with higher severity, as it is something to help in troubleshooting. "Main" alert still should come from the application which has problems and not from MySQL server. Otherwise, this is alerting on cause and not symptom and as such, it is considered a bad practice.

from awesome-prometheus-alerts.

I removed the for parameter + set severity to info.

Closing.

from awesome-prometheus-alerts.