Add field type to query shape and extensive unit tests

[deshsidd]

Description

Add field type to query shape and integrate with the setting changes for field_name and field_type: #135

Example

Field Mapping:

{
    "index1": {
        "mappings": {
            "properties": {
                "field1": { "type": "keyword" },
                "field2": { "type": "text" },
                "field3": { "type": "text" },
                "field4": { "type": "long" }
            }
        }
    }
}

Query:

{
    "query": {
        "bool": {
            "must": [
                {
                    "term": {
                        "field1": "example_value"
                    }
                }
            ],
            "filter": [
                {
                    "match": {
                        "field2": "search_text"
                    }
                },
                {
                    "range": {
                        "field4": {
                            "gte": 1,
                            "lte": 100
                        }
                    }
                }
            ],
            "should": [
                {
                    "regexp": {
                        "field3": ".*"
                    }
                }
            ]
        }
    }
}

Query Shape:

bool []
  must:
    term [field1, keyword]
  filter:
    match [field2, text]
    range [field4, long]
  should:
    regexp [field3, text]

Local Sanity Testing Logs

1. Tested all combinations of field_type and field_name enabled/disabled
2. Tested multiple types of search queries including complex queries with aggregation, sort, pipeline agg
3. Tested invalid fields not contained in the mapping

[2024-10-09T18:33:16,662][INFO ][o.o.p.i.c.l.QueryInsightsListener] [integTest-0] Query Shape: match [name, text]

[2024-10-09T18:33:18,673][INFO ][o.o.p.i.c.l.QueryInsightsListener] [integTest-0] Query Shape: term [email, keyword]

[2024-10-09T18:33:20,721][INFO ][o.o.p.i.c.l.QueryInsightsListener] [integTest-0] Query Shape: range [age, integer]

[2024-10-09T18:33:23,045][INFO ][o.o.p.i.c.l.QueryInsightsListener] [integTest-0] Query Shape: 
bool []
  must:
    match [name, text]
    range [age, integer]

[2024-10-09T18:33:25,295][INFO ][o.o.p.i.c.l.QueryInsightsListener] [integTest-0] Query Shape: 
wildcard [email, keyword]

[2024-10-09T18:33:27,630][INFO ][o.o.p.i.c.l.QueryInsightsListener] [integTest-0] Query Shape: 
aggregation:
  avg [age, integer]

[2024-10-09T18:33:30,132][INFO ][o.o.p.i.c.l.QueryInsightsListener] [integTest-0] Query Shape: 
nested []
  must:
    match [address.city]

[2024-10-09T18:35:43,292][INFO ][o.o.p.i.c.l.QueryInsightsListener] [integTest-0] Query Shape: 
bool []
  must:
    match [name, text]
  filter:
    range [age, integer]
aggregation:
  terms [age, integer]
    aggregation:
      max [join_date, date]
      pipeline aggregation:
        bucket_sort
sort:
  desc [join_date, date]

Updating [search.insights.top_queries.grouping.attributes.field_type] from [true] to [false]

[2024-10-09T18:39:14,158][INFO ][o.o.p.i.c.l.QueryInsightsListener] [integTest-0] Query Shape: 
bool []
  must:
    match [name]
  filter:
    range [age]
aggregation:
  terms [age]
    aggregation:
      max [join_date]
      pipeline aggregation:
        bucket_sort
sort:
  desc [join_date]

[2024-10-09T18:39:39,482][INFO ][o.o.c.s.ClusterSettings  ] [integTest-0] updating [search.insights.top_queries.grouping.attributes.field_name] from [true] to [false]
[2024-10-09T18:39:39,482][INFO ][o.o.c.s.ClusterSettings  ] [integTest-0] updating [search.insights.top_queries.grouping.attributes.field_type] from [false] to [true]
[2024-10-09T18:39:42,616][INFO ][o.o.p.i.c.l.QueryInsightsListener] [integTest-0] Query Shape: bool []
  must:
    match [text]
  filter:
    range [integer]
aggregation:
  terms [integer]
    aggregation:
      max [date]
      pipeline aggregation:
        bucket_sort
sort:
  desc [date]

[2024-10-09T18:39:48,314][INFO ][o.o.c.s.ClusterSettings  ] [integTest-0] updating [search.insights.top_queries.grouping.attributes.field_type] from [true] to [false]
[2024-10-09T18:39:50,451][INFO ][o.o.p.i.c.l.QueryInsightsListener] [integTest-0] Query Shape: bool
  must:
    match
  filter:
    range
aggregation:
  terms
    aggregation:
      max
      pipeline aggregation:
        bucket_sort
sort:
  desc

resolves: #69

[jainankitk]

Looks much better now. Minor comments to improve the readability

[opensearch-trigger-bot]

The backport to 2.17 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/query-insights/backport-2.17 2.17
# Navigate to the new working tree
pushd ../.worktrees/query-insights/backport-2.17
# Create a new branch
git switch --create backport/backport-140-to-2.17
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 dcf52c28a6bd7ed169c71fd8c006e56db7b73f13
# Push it to GitHub
git push --set-upstream origin backport/backport-140-to-2.17
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/query-insights/backport-2.17

Then, create a pull request where the base branch is 2.17 and the compare/head branch is backport/backport-140-to-2.17.

[ansjcy]

@deshsidd don't backport this to 2.17, this is for open source 2.18.