[Change Proposal] Support customization of routing_path

[jsoriano]

In #357 we added a flag to enable TSDB, this sets index.mode: "time_series". This mode requires to configure a routing_path, when it isn't, it uses the configured keyword dimensions.

When collecting data from unknown sources, the dimensions may not be known beforehand, so it is not possible to rely on the default behavior. For these cases it is possible to use wildcards in the routing path, and dynamic mappings, to configure the dimensions.

For example for prometheus, it would be possible to configure an index template with the following settings, to use prometheus.labels.* as routing path, and as dimensions:

{
  ...
  "template": {
    "settings": {
      "index.mode": "time_series",
      "index.routing_path": [
        "prometheus.labels.*"
      ]
    },
    "mappings": {
      "dynamic_templates": [
        {
          "labels": {
            "path_match": "prometheus.labels.*",
            "mapping": {
              "type": "keyword",
              "time_series_dimension": true
            }
          }
        },

To support this, the package spec needs to expose some way to configure the routing path. There are some options for this:

• Option 1: Directly configure it as an elasticsearch setting, this would allow to provide any configuration for this setting, but would complicate validation of fields.

  elasticsearch:
    index_mode: "time_series"
    routing_path: ["prometheus.labels.*"]
  

• Option 2: Configure it as a setting in the field definition. This would be less flexible, but safer for validations.

  - name: prometheus.labels.*
    type: keyword
    routing_path: true
    dimension: true
  

• Option 3: Implement both previous options, what would be more flexible, but the less safe option regarding validations.
• Option 4: Don't add an explicit setting for the routing_path, and configure all dimensions as routing paths. This is what Elasticsearch would do in any case, but we would also include the dynamic mappings with their wildcards.
  • [Fleet] Add dynamic mappings for dimension fields to the routing path for TSDB data streams kibana#141796

If options 1-3 are chosen, we also need to decide what to do with other dimensions when configuring routing_path. Without configured routing path, keyword dimensions would be used as routing_path, but if a routing path is added, should other dimensions be also used as routing path? There are two options:

• Option 1: If a routing path is configured, use only these fields, and not the dimensions fields.
• Option 2: If a routing path is configured, combine them with the rest of dimension fields.

[ruflin]

Option 4 is basically this one here?

dynamic_templates": [
        {
          "labels": {
            "path_match": "prometheus.labels.*",
            "mapping": {
              "type": "keyword",
              "time_series_dimension": true
            }
          }
        },

The reason I like Option 4 is that we never expose routing_path to the package dev. I consider this an implementation detail of TSDB, ideally it would not even be exposed to us ;-) The part I worry about option 4 is that it might need quite a bit of magic in Fleet to make it work?

[jsoriano]

Yes, option 4 would be that for a configuration like the following one:

- name: prometheus.labels.*
  type: keyword
  dimension: true

• A dynamic template is added for prometheus.labels.* with time_series_dimension: true.
• index.routing_path setting includes prometheus.labels.*.

This would be similar to current behaviour, but it would also work for wildcard fields. This is probably the most intuitive approach for developers. I agree that it would be nice to don't expose an additional concept unless it is needed.

The part I worry about option 4 is that it might need quite a bit of magic in Fleet to make it work?

Yes, Fleet would need to add support for this, but this doesn't seem too magic 🪄, it would be to loop over all fields with time_series_dimension, and add them to the routing_path. Maybe only if there are dynamic mappings configured as dymensions.

[jsoriano]

Another advantage of option 4 is that it doesn't need any change on current packages, or the package spec itself.

[jsoriano]

In the future we could add something like option 2 if we find that for some reason we need to don't include a dimension in the routing path.

[jsoriano]

I have created elastic/kibana#141796 to start discussion about option 4.

[jsoriano]

Option 4 implemented in Elasticsearch (elastic/elasticsearch#90528), so closing this, no need to add anything in the spec by now.