Skip to content

Top N queries by Latency - aggregator implementation #11295

@ansjcy

Description

@ansjcy

Is your feature request related to a problem? Please describe.
Currently, OpenSearch lacks a direct means of providing insights into “top queries” that have a significant impact on latency, we want to build the in-memory aggregator to capture those queries on the coordinator node level.
(Parent rfc: #11186)

Describe the solution you'd like
Implement a priority queue-based in-memory data store, with configurable window size, on the coordinator node, designed to efficiently store the top N queries. The data model of the stored query attributes should be highly extensible for different types of resources and metrics.

Describe alternatives you've considered
We can instead export the data and offload the query instrumentation logic to external sinks. But it would require us to retain all necessary data for every request. Additionally, as of now we lack a state-of-the-art on-node store capable of holding all request information for aggregation and calculating the top N when user requested. Moreover, we refrain from introducing dependencies on a specific sink in this feature.

Additional context
Please see parent rfc: #11186

Metadata

Metadata

Assignees

Labels

Search:Query InsightsenhancementEnhancement or improvement to existing feature or requestuntriagedv2.12.0Issues and PRs related to version 2.12.0

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions