[Feature Request] Intra-segment Concurrent Search

[asimmahmood1]

Is your feature request related to a problem? Please describe

As of OpenSearch 3 with auto search mode, we now support concurrent segment search by default. This trades hgher resource use for the benefit of lower latency gains, as long there are mutltiple segments to search. With a single segment or very large segment, Lucene now provides intra-segment slicing: apache/lucene#13542

Current this is not enabled by default in Lucene because of some regressions: apache/lucene#13745. This issue will track the Lucene progress in parallel, but there are many optimizations withing OpenSearch that assume we can slice a segment cleanly, which isn't true, e.g. BKD trees.

Describe the solution you'd like

1. Track down which queries show regression when intra-segment search is enabled
2. Come up with a plan to tackle the regressiona and take advantage of intra-segment search
3. Clean up tech debt from OS concurrent segment search work (PR TBD)

Related component

Search:Performance

Describe alternatives you've considered

No response

Additional context

No response

[asimmahmood1]

@jainankitk Please add more details as needed.

@bowenlan-amzn You mentioned some tech debt from previous concurrent segment search work, can you add some details or links?

[asimmahmood1]

Also following up on apache/lucene#14485

[asimmahmood1]

Based on my reading of the code, I propose the following plan.

1. Pick a simple strategy to use intra segment: currently the slices are distributed in FFD manner.
   a. One simple strategy if there are more slices than segments, or one segment is 2x all other segmentd combined, then split the largest.
   b. This will handle the case of single segments, or one very large segment
2. Test the changes with big5, http and pmc benchmarks
   a. goal is to find out which queries take advantage of the split, my guess is any that do not use optimization techniques.
   b. come up with mechanism to disable intra segment concurrentcy for those queries
3. Based on the benchmark results, come up with heuristic to use intra-segment. Also consider putting it behind a feature flag, e.g. concurrent_segment_search:intra
4. Enhance profile result: provide which segment(s) were intrasliced, plus look to see if we can integrate https://github.com/apache/lucene/pull/14413/files