Support DLM in the Logging Track

[inqueue]

Data Lifecycle Management (DLM) will be optional for stateful Elasticsearch and required for serverless. This commit aims to add DLM support to the Logging track:

• Add a lifecycle parameter for Index Lifecycle Management (ilm) or Data Lifecycle Management (dlm).
• Add a new component template named track-lifecycle.json. ILM or DLM will be enabled based on the value of the lifecycle parameter.
• Remove the ILM configuration from individual data stream component templates.
• Reference the track-lifecycle component template in each data stream composable template.

Unlike ILM, DLM does not use policies. Instead, data retention is specified per data stream (as applied in the templates) and the criteria for rollover is specified in cluster settings. For this iteration, the track will rely on DLM defaults in cluster settings:

GET /_cluster/settings?include_defaults=true&filter_path=**.dlm.*

{
  "defaults": {
    "cluster": {
      "dlm": {
        "default": {
          "rollover": "max_age=auto,max_primary_shard_size=50gb,min_docs=1,max_primary_shard_docs=200000000"
        }
      }
    },
    "indices": {
      "dlm": {
        "poll_interval": "10m"
      }
    }
  }
}

@gmarouli, I hope you don't mind me adding you to review how DLM is configured. Please indicate if I have overlooked any assumptions. Thanks!

[gmarouli]

LGTM in general, I really like having one component template that defines contains the lifecycle configuration, because it might still change in the future and this way we only have to touch one place <3.
I have two nits that I wanted to bring up, but none are blocking

1. If I am not mistaken the logs ILM policy is :

{
  "policy" : {
    "phases" : {
      "hot" : {
        "min_age" : "0ms",
        "actions" : {
          "rollover" : {
            "max_size" : "50gb",
            "max_age" : "30d"
          }
        }
      }
    }
  }
}

I don't think it makes much of a different in practice, because this performance tests are not going to run for too long, but if you want to have DLM being as close as possible with the configuration of ILM, you might want to have an empty lifecycle, this represents that the data stream is managed, so rollover and merging will be applied, but the retention time is infinite:

"lifecycle": { }

Again in practice, it will not make a difference.

2. Maybe, if you do not mind a longer name, rename track-lifecycle.json to track-data-stream-lifecycle.json. The only reason I saying this, is because if in the future someone thinks it's a good idea to copy paste it they need to know that DLM will work only with data streams and they cannot use it with plain indices.

[gmarouli]

We are in the process of rebranding DLM. We decided that it will be introduced as a new feature but as the built-in lifecycle of data streams. We believe this will be easier for the users to digest and differentiate eith ILM.

We are just in the process of that. I don't think this should be a blocker for this PR, but I am mentioning that we might have to change the wording later to match the new "brand". Would that be ok?

[inqueue]

Thank you, Mary, for your recommendations. Your two suggestions have been implemented in 7f83fce.

We are just in the process of that. I don't think this should be a blocker for this PR, but I am mentioning that we might have to change the wording later to match the new "brand". Would that be ok?

This is ok with me and I do not think it should block this PR.

[dliappis]

LGTM! Looks great and it seems to rollover with the default 50gb value in ES cluster settings:

esbench start --use-case=simple --params=./jasonbdlm.json --user-tags='project:serverless'

{
    "cloud.region": "eu-central-1",
    "loaddriver.instance.name": "aws::i4i_16xlarge-64-512-4x3750GB_nvme_local",
    "elasticsearch.instance.name": "aws::c6gd_8xlarge-32-64-1x1900GB_nvme_local",
    "elasticsearch.remote.repo.url":"https://github.com/elastic/elasticsearch",
    "elasticsearch.remote.repo.revision":"main@now",
    "elasticsearch.node.count": 3,
    "client.options": {
      "default": {
        "timeout": 240
      }
    },
    "track.repository": "inqueue",
    "track.revision": "logs-dlm",
    "track.name": "elastic/logs",
    "track.challenge": "logging-indexing",
    "track.params": {
      "bulk_start_date": "2020-01-01",
      "bulk_end_date": "2020-01-04",
      "bulk_indexing_clients": 32,
      "bulk_size": 5000,
      "data_generation_clients": 16,
      "raw_data_volume_per_day": "100GB",
      "max_generated_corpus_size": "1000GB",
      "max_total_download_gb": 2000,
      "number_of_replicas": 1,
      "number_of_shards": 1,
      "wait_for_status": "green",
      "lifecycle": "dlm"
    },
    "exclude.tasks": [
      "compression-stats"
    ],
    "stats.telemetry": [
      "node-stats"
    ],
    "stats.telemetry.params": {
      "node-stats-sample-interval": 10,
      "node-stats-include-indices": true
    }
  }

$ curl -k -u esbench:redacted https://elasticsearch-0:9200/_cat/indices
green open .ds-logs-k8-application.log-default-2023.06.13-000001 -GJm02onRrKGq3-FnjZWbg 1 1 229191838 0    93gb  46.5gb
green open .ds-logs-k8-application.log-default-2023.06.13-000002 A3Yf-fslQuSKmEb-jmj_YA 1 1  61779925 0  25.8gb  12.8gb
green open .ds-logs-mysql.error-default-2023.06.13-000001        cEkzrsgoROaqcIcN6gEgjA 1 1   5695945 0 977.3mb 488.6mb
green open .ds-logs-apache.error-default-2023.06.13-000001       Tm1sc3F8Q3KnbiGJBMpM9w 1 1   6831250 0   1.8gb 970.6mb
green open .ds-logs-redis.log-default-2023.06.13-000001          ewiUqXFcQsCAGHvukQ9Qiw 1 1  16132148 0   2.6gb   1.3gb
green open .ds-logs-system.syslog-default-2023.06.13-000001      CsYvlljQS7mdMuBK09ANlw 1 1  32233317 0  11.4gb   5.7gb
green open .ds-logs-nginx.error-default-2023.06.13-000001        5_EVt0ZJTu2hJvqF_R5fxw 1 1   6306669 0   2.1gb     1gb
green open .ds-logs-nginx.access-default-2023.06.13-000001       L0gKnQm5ScC5neqvnIygcg 1 1  25235327 0  18.5gb   9.2gb
green open .ds-logs-apache.access-default-2023.06.13-000001      Rb6DUTkiS9a7fr_SMkVByQ 1 1  10148637 0   4.3gb   2.1gb
green open .ds-logs-mysql.slowlog-default-2023.06.13-000001      mO5xq_YtQMGOgbb6hKyA-A 1 1   8127095 0   1.7gb 918.5mb
green open .ds-logs-redis.slowlog-default-2023.06.13-000001      2lhfAUsHRUimgn5uQ3Y95g 1 1  20882385 0   8.7gb   4.3gb
green open .ds-logs-kafka.log-default-2023.06.13-000001          E3u4xzUNS-q0H1uSJEMbhA 1 1  11923038 0   2.5gb   1.2gb
green open .ds-logs-postgresql.log-default-2023.06.13-000001     LMKIy91QTByyM1z00A5gnw 1 1   4194528 0 993.8mb 501.7mb
green open .ds-logs-system.auth-default-2023.06.13-000001        EQcgjExIS7G8IdCrSr1pdg 1 1  15734750 0   5.9gb   2.9gb