[FEATURE] Additional options for "Group By"

[simonsolts-bark]

For the top-n query insights plugin, I would like to group the queries, so that I can visualise whether certain searches are taking longer than expected on the cluster

I tried to use the "Group by" configuration, but the only option is "similarity" -- this is not appropriate for my usecase

I would like to be able to group / visualise the query latency by properties I define on the query
For example, I have added some stats keywords to the query, these would be ideal to group on.

My usecase is an (AWS Managed) OpenSearch 3.1 cluster; storing documents with certain attributes.
Part of the customer journey involves them setting some filters in a search UI, our backend turns this into an OpenSearch query and ranks and retrieves the documents from the cluster.

[kkhatua]

@simonsolts-bark since the current group by feature does not support user specific attributes, there will need to be another way to extract that information. We have an exporter that writes to a local index and can probably serve the purpose. I'll let @ansjcy comment on this.

[ansjcy]

Hi @simonsolts-bark , thanks for reaching out!

This is an interesting problem. IMO there are 2 solutions to your use case: exportered indices based grouping vs query time grouping.

1. As @kkhatua mentioned, the top queries are exported to local indices, you can do grouping / aggregations on the data with customized search queries. Also, the plugin already captured x-opaque-id in the LABELS attribute, see

   query-insights/src/main/java/org/opensearch/plugin/insights/core/listener/QueryInsightsListener.java

   Line 369 in 60a657b

   labels.put(Task.X_OPAQUE_ID, userProvidedLabel);

   , so if you sent your queries with x-opaque-id header, you can later group / aggregate with the exported index. The caveat here is you are actually grouping the "Top queries", instead of "all the queries".
   for example, one query on the exported data to grouped by customized label:

  GET /top_queries-*/_search
  {
    "size": 0,
    "aggs": {
      "by_label": {
        "terms": {
          "field": "labels.X-Opaque-Id.keyword",
          "size": 100
        },
        "aggs": {
          "avg_latency": {
            "avg": { "field": "measurements.latency.number" }
          },
          "max_latency": {
            "max": { "field": "measurements.latency.number" }
          },
          "p95_latency": {
            "percentiles": {
              "field": "measurements.latency.number",
              "percents": [95]
            }
          }
        }
      }
    }
  }

3. If you need a native way to group all the queries at query time, we need to actually add another group_by dimention (need to make code changes for that), potentially by "label" (like the x-opaque-id), "user name" or even "customized attribute path on the query" etc.
   Which do you prefer?