[META] Generic Query Insights Framework

[ansjcy]

Is your feature request related to a problem? Please describe.
This meta issue tracks all the features and tasks within the scope of the Query Insights framework. Currently, there are multiple areas we want to invest in to improve query visibility in OpenSearch:

• Query Insights plugin with collect, process, and export capabilities.
• Top queries insights by resource usage / latency.
• Metrics improvements, grouped by query categories / similar queries.
• Query Insights Dashboards with profiling capabilities.
• Recommendations based on query insights to improve search performance.

Describe the solution you'd like

Related OS RFCs:

• [RFC] Query insights framework  #11429
• [RFC] Real-time Insights into Top N Queries by Latency and Resource Usage #11186
• [RFC] Query Insights - Recommendation framework #12292
• [RFC] Query Visibility #11008
• [META] Search Query Categorization #11596
• [RFC] User Behavior Insights #12084 (client side insights (UBI))
• [Feature Request] [RFC] Grouping similar Top N Queries by Latency and Resource Usage #13357

Related OSD RFCs:

• [RFC] Query Insights Dashboard OpenSearch-Dashboards#5571
• Search Profiler visualization OpenSearch-Dashboards#571 (We can potentially integrate the Query Insights Dashboard with Profiler UX experience.)

Child Meta Issues:

• [META] Grouping similar Top N Queries by Latency and Resource Usage #13419

Major Pending Issues:

• Logic duplication in SearchRequestContext and TimeProvider #11762
• Support for dynamically adding SearchRequestOperationsListener #11520
• Top N queries by Latency - aggregator implementation #11295
• Query insight - identify important metrics in search workflow #11431
• [Query Insights] Capture query-level resource usage metrics #12399
• Asynchronous processor and exporter for query insights data #11296
• [Query Insights] Add user info in top queries results #12740
• [Query Insghts] Memory budgeting for Query Insights services #13369
• [Feature Request] Move the Query Insights plugin out from core repo #13881
• [Feature Request] [RFC] Grouping similar Top N Queries by Latency and Resource Usage #13357
• [Feature Request] Grouping Top N queries by user labels #14197
• [Feature Request] Optimizing Resource Usage Header Performance for Search Requests #17407
• [FEATURE] Default labeling rule for security based tenancy labeling security#4402
• [DOC] Query Insights with Top N Queries documentation-website#6180
• [DOC] Query Insights Exporters documentation-website#6945
• [DOC] Add document for top n queries by CPU and Memory documentation-website#7325

Major Ongoing Tasks:

• Add support for dynamically adding SearchRequestOperationsListener #11526
• Query insights plugin implementation #11903
• Top N Queries by latency implementation #11904
• Query insights exporters implementation #12982
• Query-level resource usages tracking #13172
• consume query level cpu and memory usage in query insights #13739
• Add X-Opaque-Id to search request metadata for query insights #13374
• Admin role for Query insights plugin security#4006
• Add index permissions for query insights exporters security#4229
• Add default request labeling rules in security plugin security#4403
• Add Query Insights documentation documentation-website#6261
• Add document for top n queries improvements in 2.15 documentation-website#7326

POC / Experimental PRs

• [Draft] Query Insight Plugin with Top Queries feature #11506
• [POC] Query-level resource usages instrumentation with SearchPhaseResult #12449
• [POC] Accurate query-level resource usages instrumentation on Cluster Manager #12473
• [POC] Add ThreadContext user info in top queries #12529

Additional context
Please add any tickets in the comments in case I'm missing anything.

[peternied]

[Triage - attendees 1 2 3 4]
@ansjcy Thanks for filing this issue