[Meta] UI ability to assign alert actions per action group

[mikecote]

An alert can use one or many action groups to fire actions.
Currently the UI is limited to assign alert actions to defaultActionGroupId only.
This is a meta issue to support different action groups like the API already supports.

Individual Issues:

• Define actions under a specific Action Group in the Add / Edit flyouts [Alerts] There is no way to assign actions to a specific Action Group in the Add / Edit flyouts #82274
• Include Action Group in Event Log when Instances are activated and display Action Group on Alert Instances in Alert Details page Display Action Group in Alert Details page #82275
• Conditions fields for configurable Action Group params There's no framework level UI for defining conditions for Action Groups #82412

TBD

• Include the transition of an Alert Instance from one Action Group to another in the event log Transition of alert instance from one action group to another #82792

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

One of the questions that was raised when we previously looked at supporting multiple action groups, was how the parameters might be changed to deal with action groups. For example, say you have an threshold-styled alert type that has two action groups - warning and critical. Presumably you want two threshold values there. I believe we were trying to figure out how to put the relevant threshold picker inputs near the action group settings. Which then raised the question - what other parts of the alert parameters would change? Could you pick another index, or time field, or comparator? How would these elements be associated with the action groups?

Now thinking that trying to put the parameters near the action groups isn't right. There is only one set of config/parameters for an alert, they should be grouped together in terms of input controls. For the case of a threshold-style alert with two action groups, the simplest case is just to two threshold values - and they should probably be named relevant to the action group. So, warningThreshold and criticalThreshold. And since they are part of the complete set of parameters, they should be in the "parameter" section of the UI.

A more complicated example could be a threshold-style alert with two action groups that took different thresholds AND expressions. In that case, the parameters could have paired parameter values for the threshold, the comparator, the field compared, etc. Still all in the "parameter" section of the UI, and again named relevant to their actionGroup.

This makes the UI a bit wonky, with all the params in one section, and then the action groups in another section, but I'm not sure a pleasing UI is possible where you split these kind of "paired" parameter values so they are closer to the action group they related to. I think this is the simplest conceptual model though (all parameters together), so we should try a design like that to see if we think it's reasonable.

[pmuellr]

I'm wondering if we could convert the existing index threshold alert to allow multiple action groups, in a "progressive" manner. Eg, add an optional second threshold value and second action group. Seems do-able, kinda wondering how confusing it will be though ...

Some interesting UI considerations as well - will need potentially two threshold lines in the preview graph.

In general we need to figure out the constraints between comparators and thresholds. If comparator is >, then presumably threshold 1 > threshold 2. And if threshold 2 fires, then threshold 1 should not fire, I guess. For comparator <, I guess it's kinda reversed? And I what about the between comparator? Do you have an "outer" threshold and "inner" one? That doesn't seem like it makes sense, maybe only one threshold is available for between.

[mdefazio]

Below are some questions I keep running into as I try to create concepts for this.

1. Will there be certain parameters that we can create a 'group' off of? And others we cannot?
2. Will the group only be based on a single parameter, or can it be multiple parameters?
3. Am I required to setup separate actions for the group? Or is it also just as likely that I would simply want to run the same actions?
4. If I have 3 actions setup for the main group, and I add another group, can I choose to run only 1 of those actions?
5. How common is it to setup these groupings? Would this need to be an option for the solutions to show the option or not, so it would show for some types and not others?
6. Are there set types of groups? (Critical, Alert, Warning, ... ) And/or can the user create their own label for these? I know we also talked about acknowledgment (or 'Unacknowledged) as a possible group.

I know some of these overlap Patrick's comments, but was just outlining the ones I had

[pmuellr]

1. Will there be certain parameters that we can create a 'group' off of? And others we cannot?

This would be defined by the alert type, but it seems like it would be hard to allow the UI of an alert type to specify a set of parameters applicable to a subset of action groups. So it feels like for now, we shouldn't consider "grouping" the params around certain action groups - let's just leave them all in the same visual location.

2. Will the group only be based on a single parameter, or can it be multiple parameters?

Again, up to the alert type, but could certainly be multiple.

3. Am I required to setup separate actions for the group? Or is it also just as likely that I would simply want to run the same actions?

Yes, looking at it from the alerting point of view, the actions in each group are separate. Howevers, it seems likely that folks would want to define the same connector to run in multiple groups. Maybe in one condition (for one action group) you do X, but as the condition worsens you want to do an additional thing (for another action group). It would be nice to "copy" the connector from one group to another. Another way of looking at this is that you could have a list of all the connectors you wanted to run, across all action groups, and then for each connector you could indicate which action group it ran in. Except that doesn't quite work as the connectors in the action groups are ordered, so there wouldn't be a way to indicate the order. I think for now, the simplest thing in this case is to force the customer to create the same connector in all the action groups. We can optimize the UX for this later when we find out how this stuff is used in practice.

4. If I have 3 actions setup for the main group, and I add another group, can I choose to run only 1 of those actions?

Per my notes in 3. ^^^, the customer will have to add a new connector for that group, and they can choose whatever connectors they want, including connectors not in the main group. Every connector would be under a single action group, and an action group can have multiple connectors. Also, in case it wasn't clear, the alert type defines how many action groups it supports (it's static), and also indicates the "default" one to use, if there are multiple.

5. How common is it to setup these groupings? Would this need to be an option for the solutions to show the option or not, so it would show for some types and not others?

Not clear how common it will be for alerts to have multiple action groups, but as action groups are defined by the alert type, some will have 1, some will have multiple. We could optimize the case for an alert with a single action group defined, and not show any grouping, just as we do today. If the alert does support multiple action groups, then the area under the graph, where you list the actions, would now have a separate section for each action group (the default one first, and the remaining ones are ordered I believe), where actions can be added to each action group, just like today.

6. Are there set types of groups? (Critical, Alert, Warning, ... ) And/or can the user create their own label for these? I know we also talked about acknowledgment (or 'Unacknowledged) as a possible group.

Critical/Alert/Warning was the original reason for creating action groups. The user cannot add/change/rename them - they're defined by the alert type. Ack/unack is - I think - out of scope here. I can see how it could potentially fit in here, but there's a lot of other stuff to think about with acks. I've also been thinking about having "built in" action groups - action groups every alert would have - "resolved" and "no data" are the two that I've been thinking of. They're basically common behaviours/states an alert / alert instance could be in, that we'd like to handle generically, if possible. Jury is still out though, it's not coming in 7.8 anyway.

As a somewhat contrived example, but based on a real need, there's a new SSL certificate check alert. It will fire when a cert is going to expire within 30 days. The thought is to allow for additional actions once it's going to expire in 7 days. So, think two action groups "early warning" and "about to expire". As alert parameters, you'd be able to set the 30 and 7 values for those, in the usual parameter section. In the alert groups, you add an email action to "early warning", and an email AND slack action to "about to expire" - the idea being to bug them a little bit more the closer you get to the expiration date. You could also just decide you don't want the 30 day warning at all, and not have any actions associated with it. If you had email connectors in both groups, they're independent - they can have different messages.

[mikecote]

@mdefazio to paste latest mockups.

@arisonl to ensure these mockups align with what solution teams are expecting.

[mdefazio]

Here is the current iteration of the wireframes we've been discussing. Please let me know if anyone sees misalignments with what we've agreed upon. @arisonl

[arisonl]

@mdefazio Some early feedback from Observability: Metrics expect to need two or three levels in most cases, with two (alert and warning) being the most usual use case and three (e.g. major, minor and warning) being also possible, yet less usual. They would also like the option to receive an alert for each transition, hence not just "resolved" but also once the value falls from warning to normal, from major to minor etc.

[mdefazio]

Some initial thoughts on this:
Depending on what statuses/conditions are added in the trigger section, if the user checks "Run actions when resolved", a dropdown will provide options for 'Resolved', 'Alerting -> Warning', or 'Major --> Minor'.

I'm assuming that 'Warning --> Resolved' or 'Alerting --> Resolved' are the same as simply 'Resolved'. Is this correct?

If an alert goes from Warning to Alert, will it simply run the actions that are associated with 'Alert'? Or do we also need to provide transitional actions here as well?

[pmuellr]

Depending on what statuses/conditions are added in the trigger section, if the user checks "Run actions when resolved", a dropdown will provide options for 'Resolved', 'Alerting -> Warning', or 'Major --> Minor'.

Technically, these are "action groups" - [alert, warning], [major, minor], where resolved may end up being a "free" action group you get (and maybe "no data") (eg, [alert, warning, resolved, no-data]. Of course, that's a terrible phrase; status is not a great one either maybe, as we want to have an alert-level thing called "status" which would indicate no-data, error, actively firing kind of info.

I'm not sure a toggle for "Run actions when resolved" makes sense as a toggle, if it's just another action group.

I'm assuming that 'Warning --> Resolved' or 'Alerting --> Resolved' are the same as simply 'Resolved'. Is this correct?

That was my understanding. There is only "resolved", we won't have "minor->resolved" and "major->resolved" as separate things. I think technically, we could, but not sure we need it, and it makes things more complicated, so I'd say at best we defer that (and open a new issue if we think we need that).

If an alert goes from Warning to Alert, will it simply run the actions that are associated with 'Alert'? Or do we also need to provide transitional actions here as well?

Likewise, my understanding is that we won't have transitional actions like that, so you'd only see the 'Alert' actions run in that case. And as before, we could, but it's just more stuff, so worthy of a new issue (probably one issue for this and the previous note ^^^).

[arisonl]

Say 1 is threshold for warning and 2 is threshold for alert. The current thinking is that, you will get:

• a warning notification if you get from 0.5 to 1.5 and from 2.5 to 1.5
• an alert notification if you get from 0.5 to 2.5 or from 1.5 to 2.5
• a resolved notification if you get from 2.5 to 0.5

What I am hearing might be needed is a resolved notification when you get from 1.5 to 0.5, i.e. drop from warning to normal or from any other level to normal (e.g. from minor to normal).