Skip to content

Optimize doc level monitors - only query indices with new documents in data streams #1290

@eirsep

Description

@eirsep

Doc level monitors intend to track sequence number of indices shards and query only documents ingested since last monitor execution. The problem is that users configure doc level monitors to run against index patterns with rollover or data streams.
More often than not doc level monitors are querying an index patter which resolves to , say 1000 concrete indices, of which 999 are read-only and 1 write index.
We really don't need to query all the 999 read-only indices as they might not have any documents ingested since the previous execution.

we can optimize to query only those indices which are writeable and all indices which have been marked as read-only AFTER the previous monitor execution
for first run of monitor, when creating doc level queries it should only consider write index and the read indices which were created after the enabled_time of monitor

This will drastically reduce the CPU consumption of the doc level monitor and provide a significant optimization for the plugin's resource consumption

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions