Dynamic templates being replaced instead of appending

[nemonster]

Elasticsearch version: 5.x (5.0 -5.4)

Plugins installed: none

JVM version (java -version): 1.8.0_11

OS version (uname -a if on a Unix-like system): Darwin Kernel Version 15.6.0: Tue Apr 11 16:00:51 PDT 2017; root:xnu-3248.60.11.5.3~1/RELEASE_X86_64 x86_64

Steps to reproduce:
Took a simple dynamic template example from the Definitive Guide and broke it into two parts. Created the index with one, and then submitted the other one subsequently.

http://localhost:9200/my_index
{
  "mappings": {
    "my_type": {
        "dynamic_templates": [
            { "es": {
                  "match":              "*_es", 
                  "match_mapping_type": "string",
                  "mapping": {
                      "type":           "string",
                      "analyzer":       "spanish"
                  }
            }}
        ]
}}}

http://localhost:9200/my_index/_mappings/my_type
{
  "dynamic_templates": [
    { "en": {
          "match":              "*", 
          "match_mapping_type": "string",
          "mapping": {
              "type":           "string",
              "analyzer":       "english"
          }
    }}
  ]
}

In 2.2 when you check the mapping you see:

{
  "my_index" : {
    "mappings" : {
      "my_type" : {
        "dynamic_templates" : [ {
          "es" : {
            "mapping" : {
              "analyzer" : "spanish",
              "type" : "string"
            },
            "match" : "*_es",
            "match_mapping_type" : "string"
          }
        }, {
          "en" : {
            "mapping" : {
              "type" : "string",
              "analyzer" : "english"
            },
            "match" : "*",
            "match_mapping_type" : "string"
          }
        }
        } ]
      }
    }
  }
}

However when executing exactly the same steps in 5.x you get:

{
  "my_index" : {
    "mappings" : {
      "my_type" : {
        "dynamic_templates" : [
          {
            "en" : {
              "match" : "*",
              "match_mapping_type" : "string",
              "mapping" : {
                "analyzer" : "english",
                "type" : "string"
              }
            }
          }
        ]
      }
    }
  }
}

It doesn't appear to be appending new templates to the mapping, but replacing any existing ones, despite the documentation being the same between 2.2 and 5.x.

[PnPie]

I've had a look at the code and it seems that the dynamic templates is designed to be totally replaced while updating but not "merged" for 5.x version ?

[clintongormley]

I'm not sure if that change was intentional, but it may actually be a better default. Then we can forget about things like #11627

[arboliveira]

We're upgrading from Elasticsearch 2 to Elasticsearch 6. Our application needs to add new dynamic templates on the fly to an existing index, and we rely on the PUT mapping API for that. At time of first implementation, we followed Elastic documentation and everything worked as described under Elasticsearch 2:

https://www.elastic.co/guide/en/elasticsearch/reference/6.0/dynamic-templates.html#dynamic-templates

"Templates are processed in order — the first matching template wins. New templates can be appended to the end of the list with the PUT mapping API. If a new template has the same name as an existing template, it will replace the old version."

Notice the documentation above hasn't changed in Elasticsearch 5 or 6. However, behavior since 5 doesn't match it anymore and breaks backward compatibility for our application.

I was directed to this issue after filing a ticket with Elastic Support. Would you please consider restoring backward compatibility for the PUT mapping API in Elasticsearch 6? Even if a deliberate choice is made to not "merge" dynamic templates anymore, I'd also ask to please drive it through proper deprecation/breaking changes/announcement process and update documentation so we can prepare to implement merging on our side. On the other hand, having instead a fully granular API to configure dynamic templates would be really convenient.

[clintongormley]

Hi @arboliveira

I'm sorry that you have run into this change, especially as the documentation hasn't been updated. That said, I think the change is a good one and greatly simplifies management of dynamic templates, including the ability to unset templates (#11627). These templates are usually set at index creation and seldom touched again - having a "fully granular API to configure dynamic templates" seems like overkill to me. With the current situation, you can implement this client side by retrieving, editing, and reputting the templates.

Would you please consider restoring backward compatibility for the PUT mapping API in Elasticsearch 6? Even if a deliberate choice is made to not "merge" dynamic templates anymore, I'd also ask to please drive it through proper deprecation/breaking changes/announcement process and update documentation so we can prepare to implement merging on our side.

I agree that we missed the proper deprecation cycle for this change, and for that I apologise. That said, given that this breaking change happened two major versions ago, I don't think it makes sense to have another breaking change (in 7.0) to put it back the way it was, only with the intention of breaking it again in 8.0.

[arboliveira]

Thank you, sounds reasonable. Would anyone kindly be able to direct us to the source code in 2.x where "merging" used to be performed? ("replace dynamic template if present, add to end if not") We may be able to salvage some of the logic when writing our client side implementation.

[clintongormley]

thanks @arboliveira

The PR in question is #20119