Add support for `wildcard` field type

[epiphone]

Elasticsearch added the wildcard field type in v7.9. Are there any plans to support the field type in OpenSearch?

Thanks!

[tlfeng]

A clarification for others, Elasticsearch added wildcard field type as an x-pack feature in version 7.9.0, which is not an open-source feature. See the user guide of Elasticsearch 7.9 https://www.elastic.co/guide/en/elasticsearch/reference/7.9/keyword.html#wildcard-field-type

[macrakis]

@epiphone Could you describe your use case?
I do understand that it speeds up wildcard searches, but why is wildcard search performance critical in your application? How many unique values are in your dataset and how big are they?
Are there any good workarounds?

[josefschiefer27]

Some good use cases for wildcard:

• Matching error messages and stack traces
• Matching URL and file paths
• Matching fields that have encoded content (e.g. "(8-10)||86128||Women's Apparel||...")

[epiphone]

@macrakis my use case is a large index of user-submitted names where most names are short (<100 characters) and unique, and I want to query the names by arbitrary substrings.

As a workaround I'm using an ngram tokenizer which works well enough but is more complicated to set up than the wildcard field type.

[macrakis]

Josef, Epiphone, thanks for your answers -- very helpful!

So it sounds like you need to find arbitrary substrings in your corpus, not just strings starting at token boundaries.

That would be not just "clerc" in "Leclerc", but also "ecle", not just "org/open" in "server/src/main/java/org/opensearch/index/query" but also "ense" in that pathname.

Could the problems be solved by different tokenization?

[josefschiefer27]

What makes the 'wildcard' data-type nice is that it is optimized for fields with large values or high cardinality for wildcard and regexp queries without changing the search experiences (e.g. searching via *ense*) and without worrying about tokenization.

[stevesimpson418]

We have a use case for this where we need to index large XML documents that are > 32766 bytes. Our users want to be able to search for a string in an XML document eg *failuremessage122* or just *failure* or even *fail*. A keyword field type would make sense (despite the poor leading wildcard performance) but this is not possible due to exceeding the 32766 byte limit.

Tokenisation is also problematic with XML docs and we also get issues where we have token explosion with > 10000 terms generated when using most of the analyzers.

The XML logstash filter was considered but has similar issues with large documents producing a huge amount of fields. We don't always know ahead of time which elements we need to search for so that pre-processing of data isn't really an option.

Support for a "wildcard" field type would really improve our user experience