To address the toil in the ticketing system, several potential solutions or products can be considered:
Tools like Ansible or Kubernetes Operator Pattern can help automate repetitive tasks. For example, you could codify actions based upon incoming events.
Jira, Zendesk, or ServiceNow. These systems have features for prioritizing tickets and managing workflows.
Tools like PagerDuty or Opsgenie can help manage alerts, reducing noise and improving prioritization.
Make the alert review as a practive would help to identify areas of improvement. This could involve redefining alert conditions, improving documentation, create defects for development team track or providing additional training to staff.