[BUG] Validation-Error: collision of indexpatterns with differences after a wildcard

[micha558]

(initially posted as opendistro-for-elasticsearch/index-management#446)

Describe the bug
We have indexes named app-test-*-some-* and app-test-*-some_other-*
Therefore we create two indexpatterns which should be clearly distinguishable.

Unfortunately this does not work as the patterns are supposed to not be unique

But the patterns are clearly distinguisable in my eyes because of the character after 'some' (underscore vs. dash)

It seems that this is not only when a difference is between underscore and dash but that only the part before the first wildcard is checked:

Other plugins installed
Opendistro 1.13.2 without only the standard plugins

To Reproduce
Steps to reproduce the behavior:

1. Go to Dev-Tools and send the request 1:

PUT /_index_template/app-test-some-
{
  "index_patterns": ["app-test-*-some-*" ]
}

2. Go to Dev-Tools and send the request 2:

PUT /_index_template/app-test-some_other-
{
  "index_patterns": ["app-test-*-some_other-*" ]
}

3. This gives error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "index template [app-test-some_other-] has index patterns [app-test-*-some_other-*] matching patterns from existing templates [app-test-some-] with patterns (app-test-some- => [app-test-*-some-*]) that have the same priority [0], multiple index templates may not match during index creation, please use a different priority"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "index template [app-test-some_other-] has index patterns [app-test-*-some_other-*] matching patterns from existing templates [app-test-some-] with patterns (app-test-some- => [app-test-*-some-*]) that have the same priority [0], multiple index templates may not match during index creation, please use a different priority"
  },
  "status" : 400
}

also:

'''
PUT /_index_template/app-test-some_other-
{
"index_patterns": ["app-test--completedifferent-" ]
}
'''
gives this error

Expected behavior
The patterns should be both accepted as a dash and an underscore are two different characters. Parts after a wildcard should be respected

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):
Browser, curl and Python-Http-call (problem not client related)

Additional context
This worked using the old api (_template) in previous version (1.12 ?)

[CEHENKLE]

I've transferred this to OpenSearch Index Management. If this is not the right place, @ me :)

Thanks,
/C

[ketanv3]

Thinking out loud, we have the following two index patterns with us:

1. Pattern A = app-test-*-some-*
2. Pattern B = app-test-*-some_other-*

Let's say we create an index with the name app-test-foo-some-some_other-bar. It would then match both these index patterns. Notice that the part that matches the wildcard (*) is shown in square brackets.

Pattern A = app-test-[foo]-some-[some_other-bar]
Pattern B = app-test-[foo-some]-some_other-[bar]

As the index matches multiple index patterns, one template must define a greater priority value for this to work.

[dbbaughe]

@CEHENKLE This is not related to our Index Management feature set, index_templates is a core OpenSearch API/feature.

[CEHENKLE]

Thanks Drew. My mistake.

[micha558]

@ketanv3 thanks for this example. Indeed there is a collision in this case (which is an academic case at our site).

So ok - I accept that these two patterns are rejected without priority.

Unfortunately this does not solve my problem: and I have to find an alternative. Currently I am assigning a random value but this still produces collision errors from time to time...

At our site such a collision is not possible due to other reasons so the exact priority/order is not important.

How about an option in the api like "autoassign priority" which assigns an appropriate priority so that the last added pattern would win in case of a collision. Or would it be possible to assing a "currentTimeInMillies" as priority (does this accept long values)?

One other possibility I see is to do a count templates first and then use count + 1 as priority for the next one (but this would need an extra call before)

Does someone have other suggestings?

[pgtgrly]

@CEHENKLE I would like to take a shot at this feature request.