[ML] Investigate how aggregate_metric_double fields affect datafeeds

[droberts195]

Since #87849 and #88534 it is possible that anomaly detection jobs and datafeeds will encounter the aggregate_metric_double type.

We need to investigate how this will affect certain aspects of their behaviour:

1. Specifying detector fields when the datafeed is using an aggregation
2. Specifying detector fields when the datafeed is not using an aggregation
3. Ability to switch a datafeed between using an aggregation and not using an aggregation without deleting and recreating the job
4. Correct identification of whether to use _source or doc values to get values from aggregate_metric_double fields
5. How to appropriately set summary_count_field when using aggregate_metric_double

Based on this investigation decide what changes we should make to either code or documentation.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[dimitris-athanasiou]

Using aggregated_metric_double without aggregations

This is not currently possible.

Let us assume we have such a field called presum and that it has all possible metrics: [ min, max, sum, value_count ]. Datafeeds use the _field_caps api to discover fields in the source indices. The _field_caps response for our field will be:

"aggregate_metric_double" : {
  "type" : "aggregate_metric_double",
  "metadata_field" : false,
  "searchable" : true,
  "aggregatable" : true
}

There are no fields for its subfields presum.min, presum.max, etc.

This means we cannot retrieve the values from _source.

Even if we tried to retrieve presum as a docvalue field:

GET my_index/_search
{
    "docvalue_fields": ["presum"]
}

, we'd get the error:

String representation of doc values for [aggregate_metric_double] fields is not supported

Note that it is possible to get the value using the fields api but that is not how datafeeds currently work:

GET my_index/_search
{
    "fields": ["responsetime"]
}

...
"hits": [
  {
    ...
    "fields": {
        "responsetime": [
            {
                "value_count": 42,
                "max": 325.255,
                "sum": 100.0,
                "min": 1.0
            }
        ]
    }
  }
]

Using aggregated_metric_double with aggregations

Users can use aggregated_metric_double fields in datafeeds with aggs. The key here is to also retrieve the value_count in another aggregation and use that as the summary_count_field_name.

Example config:

{
  "analysis_config": {
    "bucket_span": "1h",
    "detectors": [
      {
        "function": "avg",
        "field_name": "my_avg"
      }
    ],
    "summary_count_field_name": "my_count"
  },
  "data_description": {
    "time_field": "timestamp"
  },
  "datafeed_config": {
    "indices": [
      "my_index"
    ],
    "datafeed_id": "datafeed-id",
    "aggregations": {
      "buckets": {
        "date_histogram": {
          "field": "time",
          "fixed_interval": "360s",
          "time_zone": "UTC"
        },
        "aggregations": {
            "timestamp": {  
                "max": {"field": "timestamp"}
            },
            "my_avg": {  
                "avg": {
                    "field": "presum"
                }
             },
             "my_count": {
                 "value_count": {
                     "field": "presum"
                 }
             }
          }
        }
     }
  }
}

[droberts195]

Thanks for the detailed analysis @dimitris-athanasiou.

It seems to me we should do two things.

1. Add a section to the docs explaining how to use aggregate_metric_double in an aggregation in a datafeed and giving an example.
2. Change the validation for datafeeds so that we return a clear error if somebody attempts to create a datafeed that doesn't use aggregations but references an aggregate_metric_double field. Ideally the UI will also check for this and error at an early stage, because most datafeeds created via the UI don't use aggregations. That is perhaps the most troubling aspect of the findings - we'll be forcing users who have aggregate_metric_double fields out of our friendly UI and into the world of typing out aggregations in JSON form in dev console. 😬

[dimitris-athanasiou]

I also checked what the ML UI does with those fields and it ignores them from the dropdowns in the wizards. So a user would have to configure the aggs in the datafeed using the json editor.

[dimitris-athanasiou]

@szabosteve Has documented the limitation in #92139