Allow users to set up a special alert when their Monitor is erroring about alerting HOT 6 CLOSED

The context passed to your scripts (both trigger script and action script) has details of the error in the ctx.error field. You can send a different alert text if there is an error by writing your mustache template as follows:

{{^ctx.error}}
  The action ran successfully!
{{/ctx.error}}
{{#ctx.error}}
   Action failed. Here is the error message: {{ctx.error}}
{{/ctx.error}}

When evaluating a trigger script (painless) the ctx.error variable will contain the error capturing data if there was one. When evaluating an action script (mustache) ctx.error will contain either the error capturing data or the error evaluating the script (in that order).

from alerting.

OK, I get it that I can write a conditional in my script. However, I have to know that I can do that. And it's ugly to have to conditionalize every alert.

More than that, I would like separate notifications for errors in my monitoring system. The team that responds to those issues could be different than the team that responds to the monitored condition.

from alerting.

OK, I get it that I can write a conditional in my script. However, I have to know that I can do that. And it's ugly to have to conditionalize every alert.

We went back and forth on having separate actions for errors vs. using the inline conditional in the script. Ultimately, we decided to go with the inline conditional as it provides flexibility to programmatically implement different policies vs. a pre-defined error notification. I agree that this needs to be better documented (perhaps in a blog or wiki post?)
/cc @aetter and @elfisher to explore how we can better surface this technique in documentation and UI.

More than that, I would like separate notifications for errors in my monitoring system. The team that responds to those issues could be different than the team that responds to the monitored condition.

I agree that this should be possible, but currently isn't. One way to allow for this is suppress notificatinos for actions if they produce no output (right now it throws an error, if there isn't any output). You can then achieve what you describe by splitting the conditional in the script into separate actions with different destinations for errors vs. normal monitoring. Place the normal action in a {{^ctx.error}} block and the error action template in a {{#ctx.error}} block. /cc'ing @elfisher for his thoughts.

from alerting.

I'd way rather maintain 2 different systems for alerting than maintain 2 different actions per trigger.

That gets really messy if I have even a moderate amount of triggers.

from alerting.

ctx.error is mainly useful to deal with errors that are query and trigger specific (for e.g. not enough data, script error etc.). I'm not sure I understand what you mean by "2 different systems for alerting".

For errors related to the "monitoring system" itself (I assume you're thinking of something like being unable to send notifications or unable to save alerts to the alerting indices, multiple monitors being delayed), I agree that these would normally be handled by a different team than the monitor creator. That's why we logs those as errors/warnings to the ES log with the expectation that cluster administrators will be monitoring the logs. We also have a detailed system status API that can be used by administrators to monitor the overall status of the alerting plugin: https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/api/#monitor-stats . This is intended to be integrated into whatever external cluster health monitoring is being used (similar to /_cluster/health). So it's not expected that ctx.error would be used to handle this.

Going back to your original issue: Would having a per-action flag to suppress running the action if the monitor is in an ERROR state solve your issue? Under the hood this could just check ctx.error similar to the conditional above...

from alerting.

Closing in favor of opensearch-project/alerting#29

from alerting.