New mapping parameters to annotate dimensions and metrics in timeseries data

[csoulios]

As Elasticsearch embraces time series data, we must make sure that fundamental timeseries concepts become first class citizens. We propose that we implement two new mapping parameters to be used for annotating dimension and metric fields in the index mapping.

Mapping dimensions

To mark a field as a dimension we will create a mapping paramenter named time_series_dimension that can take boolean values.

Mapping metrics

To mark a field as a metric we must create a mapping paramenter named time_series_metric. Its value will be a string that can take one of the following values: gauge, counter, histogram and summary.

For each metric type there should be a set of supported downsampling aggregations. However, there are cases that users want to override the downsampling aggregations that are supported by default. To allow them to override the default aggregations, we can allow the time_series_metric to be an object containing a type and a aggregations field.

An example to illustrate the index mapping can be found below:

PUT my-index-000001
{
  "mappings": {
    "properties": {
      "hostname": {
        "type": "keyword",
        "time_series_dimension": true
      },
      "free_memory": {
        "type": "double",
        "time_series_metric": "gauge"
      },
      "free_memory": {
        "type": "double",
        "time_series_metric": {
            "type" : "counter",
            "aggregations": ["min". "max", "value_count", "sum"]
        }
      }
    }
  }
}

This issue deprecates the support for metric_type key in the field mapping meta (#72536).

Also, we should expose those parameters through the field capabilities API so that Kibana can access this information. However, this feature will be described in a separate issue.

[elasticmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[imotov]

I feel like timeseries_dimension and timeseries_metric use case is pretty generic, but the use case for aggregations is pretty specific to rollups. So, I wonder if we should do:

• timeseries_dimension -> dimension
• timeseries_metric -> metric_type or even metric
• aggregations -> rollup_aggregations

[nik9000]

I like dimension and metric_type!

[axw]

I feel like timeseries_dimension and timeseries_metric use case is pretty generic, but the use case for aggregations is pretty specific to rollups.

As long as it's only used for downsampling/rollups, and doesn't bleed into restricting storage or aggregations generally. There's an issue with summary metrics I'd like to raise.

In Elastic APM (maybe eventually Metricbeat's Prometheus module? CC @exekias), we would like to store pre-aggregated "summary" metrics: always sum/count, and optionally min/max and possibly other quantiles. These metrics would always support value_count, sum, and avg; possibly min/max. Ideally we would use aggregate_metric_double, but you have to explicitly state up front which metric sub-fields to store. We don't necessarily know this up front.

[csoulios]

@imotov I am fine with renaming timeseries_dimension to dimension and timeseries_metric to just metric. They are less verbose and simpler.

However, I think that renaming aggregations to rollup_aggregations is too restrictive. Although any aggregation can be computed on the raw data, after the data has been rolled up only the set of aggregations listed for the metric will be supported. So, this field will be used for adveritising to clients (kibana, metrics etc) what aggregations are supported.

Finally, in my view the most important part is that aggregations is a subfield of the metric parameter and not another parameter. Thisway there will not be any confusion that aggregations refers to the metric property of the field.

[imotov]

@csoulios had a chat about the aggregation field. The aggregations field in metrics plays 2 different roles here. Before role up it indicates fields that we want to keep, after the rollup it indicates the fields that were kept. We agreed that rollup_aggregations while is appropriate for the first role, doesn't reflect the second role correctly. So, we should probably keep something generic like aggregations there.

[csoulios]

In Elastic APM (maybe eventually Metricbeat's Prometheus module? CC @exekias), we would like to store pre-aggregated "summary" metrics: always sum/count, and optionally min/max and possibly other quantiles. These metrics would always support value_count, sum, and avg; possibly min/max. Ideally we would use aggregate_metric_double, but you have to explicitly state up front which metric sub-fields to store. We don't necessarily know this up front.

@axw I have created a separate issue #74145 to discuss about relaxing the constraints in aggregate_metric_double

[axw]

A related problem has has been brewing around histograms. I had been putting it off, but @benwtrent has just come across the same problem (see #74213)

The histogram field docs say:

The histogram field is "algorithm agnostic" and does not store data specific to either T-Digest or HDRHistogram. While this means the field can technically be aggregated with either algorithm, in practice the user should chose one algorithm and index data in that manner (e.g. centroids for T-Digest or intervals for HDRHistogram) to ensure best accuracy.

The problem with this is that a consumer doesn't necessarily know how the data was stored. This certainly applies to custom application metrics, e.g. so Lens could automatically decide the most appropriate algorithm to use for calculating percentiles. I imagine this would also apply when rolling up histogram metrics. Moreover, for rollups you might want to reduce the histogram resolution to reduce storage cost.

Would it make sense to extend the metric mapping parameter with information about histograms? i.e. which algorithm (and parameters) were used to record them? If this is not the right place to discuss, maybe let's take it to #74213.