Generate and validate ecs@mapping component template.

[Mpdreamz]

Related to #2584 the official ecs mappings are still hard mappings while Elasticsearch ships with ecs@mapping which is primarily conventions based mapping for ECS.

See: https://github.com/elastic/elasticsearch/blob/3011ff68ffe3e57111a3bb01492c956567ab7c8b/x-pack/plugin/core/template-resources/src/main/resources/ecs%40mappings.json#L4

and

https://www.elastic.co/observability-labs/blog/future-proof-your-logs-with-ecs-mappings-template

Questions:

• Does this repo validate the ecs@mapping?
  • this doesn't get updated with every release.
• Should this repo start generating an ecs@mapping ?
• If not should it generate a more explicit alternative (that does enforce types for instance?).

Do beats, agent, integrations now rely on ecs@mappings or are they using the component templates this repo generates?

The component templates now sit at 2200+ fields which is tad much for each backing index in a datastream when most usecases do not use all fields.

Tagging a few folks who I know might have some thoughts:

@eyalkoren @felixbarny @gregkalapos @P1llus @trisch-me @andrewkroh @mjwolf

Feel free to ignore my @mention or forward to someone else who may have the answer(s)? 🙏

[P1llus]

@Mpdreamz The bundled ecs@mapping is a dynamic template and not a field mapping. So the fields are not mapped until its used.

The dynamic template was introduced so that all integrations would map ECS to their correct type, as integrations before had to manually define all the ECS fields they used, which would end up with human errors here and there.

For security-integrations at least we have stopped defining the ECS fields manually and rely mostly on the dynamic template with a few exceptions. However for those exceptions the fields are still defined manually rather than using all at the same time.

The total fields limit should not be related to that specific component template.

Ref dynamic templates: https://www.elastic.co/docs/manage-data/data-store/mapping/dynamic-templates

They produce very little overhead and provides a big advantage in that we don't need to define mappings for all ECS fields beforehand, but rather only once the field has been used at least 1 in that specific index.

For the generation of the template I would redirect that to @eyalkoren and @felixbarny

[Mpdreamz]

@P1llus The field limit is triggered by the component templates this repository generates: https://github.com/elastic/ecs/tree/main/generated/elasticsearch/composable

The question was whether this repository should switch to generating dynamic templates as well? and if it does should that mimic ecs@mapping ? In effect take ownership of keeping that up to date and validated with the spec?

I have no clue what the process for ecs@mapping is, hence these questions :)

For security-integrations at least we have stopped defining the ECS fields manually and rely mostly on the dynamic template with a few exceptions.

Does that mean ecs@mapping ?

They produce very little overhead and provides a big advantage in that we don't need to define mappings for all ECS fields beforehand, but rather only once the field has been used at least 1 in that specific index.

Aware, which is why I am posing these questions hoping to move over and the process behind ecs@mapping 😸

[P1llus]

Ah I understand what you mean now. Yes that means security integrations do mainly use ecs@mapping but I cannot speak for observability.

On the topic of who should generate it, I believe it was implemented on the Elasticsearch side simply because it was something that we controlled and had activity as well.
I believe we had hoped to move it to be generated somehwere else for the long term, and this would be a perfect place, though we would have to look at the test requirements needed for that and find a way to include the correct versions to be bundled with Elasticsearch for each release.

Eyal would most likely be able to answer that better though as I was only part of the initial development and implementation of it.

[eyalkoren]

Does this repo validate the ecs@mapping?
this doesn't get updated with every release.

No, it gets validated in the Elasticsearch repo through an integration test suite that runs daily and reports through email and Slack when the templates get out of sync with ECS. We already have experience with that and so far it worked well (even helped identifying ECS bugs a couple of times, e.g. - invalid field type used 🙂 )

Should this repo start generating an ecs@mapping ?

For maintenance, this would be ideal. It would make the synching process much better as we could then couple template updates with schema changes.
The cons:

• adding integration tests here that are as comprehensive as the ones we now have, may be more complicated to write and maintain. By having them within Elasticsearch, we take advantage of the existing infrastructure for it
• we tried to make the number of templates and the number of patterns minimal. However, these templates were introduced in the pre-genai era, I can easily see now an automatic process that minimizes the templates and iterates over potential changes until the result is both valid (passes all tests) and ~minimal (more difficult to define)

If not should it generate a more explicit alternative (that does enforce types for instance?).

What do you mean? We do use match_mapping_type and unmatch_mapping_type wherever makes sense.

I don't know to which extent integrations/agent use it, @ruflin probably knows better

[felixbarny]

The ecs@mappings template is used in every integration - both security and observability. But especially on the o11y side, I think there are still a lot of integrations that also explicitly define all fields, even if that's not necessary. Even the OTel templates use ecs@mappings which works well due to the big overlap and similar naming conventions between ECS and SemConv.

I'd prefer this repo to be the source of truth for ecs@mappings so that if new fields get added, we can automatically check that ecs@mappings is up-do-date in the same PR. I like the idea of simplify the curation of ecs@mappings with gent AI. This could be a cursor rule or something similar.

[Mpdreamz]

No, it gets validated in the Elasticsearch repo through an integration test suite that runs daily and reports through email and Slack when the templates get out of sync with ECS. We already have experience with that and so far it worked well (even helped identifying ECS bugs a couple of times, e.g. - invalid field type used 🙂 )

Awesome! I was 90% sure we did but could not spot the evidence from a quick scan.

The ecs@mappings template is used in every integration - both security and observability.

Sweet! Thats another key thing I was looking for evidence for :)

It's pretty clear we should all be using ecs@mapping and we should probably stop generating: https://github.com/elastic/ecs/tree/main/generated/elasticsearch/composable @mjwolf do these still have a practical use / known usages?

What do you mean? We do use match_mapping_type and unmatch_mapping_type wherever makes sense.

I mean they don't enforce the structure of _source as the composable templates do. E.g the go structs for libbeat are generated from the spec in this repo. .NET does the same so they are interoperable. Now not all ECS is created equally by all producing it. But I think that ship has long sailed away and I'm good with that :)

Thanks @felixbarny and @eyalkoren for all the intel!