Expand fetch phase profiling to multi-shard queries

[andrevandeven]

Description

Fetch phase profiling has been implemented for the single-shard case. When there is only one shard, the search executes the fetch phase immediately after the query phase and the profile is rebuilt right away.

Focusing on the single-shard case first allowed for isolated development and validation of the core fetch profiling logic before tackling the more complex distributed coordination required for multi-shard searches.

When the search involves multiple shards, executeQueryPhase previously only returned the query-phase results. The coordinating node later sends a ShardFetchRequest for each shard, which runs executeFetchPhase(ShardFetchRequest ...). This method performed the fetch but does not rebuild or attach new profiling information.

The solution I've implemented allows the fetch phase to send profile results back to the coordinator node in the multi shard case. These fetch profile results are then merged with query profile results in the coordinator node.

Related Issues

Resolves #18863

#18864

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 2ad9894: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for ab05ba0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 7dc097c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for ac1fd12: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for e4c029b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 06731f0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for a81e6f8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 3c4904c: SUCCESS

[codecov]

Codecov Report

❌ Patch coverage is 88.37209% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.83%. Comparing base (8d40807) to head (0af0a6d).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
...org/opensearch/search/fetch/FetchSearchResult.java	62.50%	1 Missing and 2 partials ⚠️
...pensearch/action/search/SearchPhaseController.java	95.00%	0 Missing and 1 partial ⚠️
...arch/search/profile/SearchProfileShardResults.java	90.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18887      +/-   ##
============================================
+ Coverage     72.82%   72.83%   +0.01%     
- Complexity    68868    68899      +31     
============================================
  Files          5600     5600              
  Lines        316247   316288      +41     
  Branches      45866    45874       +8     
============================================
+ Hits         230318   230381      +63     
+ Misses        67327    67255      -72     
- Partials      18602    18652      +50     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bowenlan-amzn]

Overral looks great! Just try increase the coverage and get this merged, so we have the full fetch profile out for 3.2!

[github-actions]

❌ Gradle check result for 6a84ced: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[msfroh]

Thanks, looks good @andrevandeven! I only have one nitpicky comment, but otherwise I think this is ready to ship.

[github-actions]

❌ Gradle check result for 0dd6430: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 0af0a6d: SUCCESS

[andrevandeven]

@msfroh, I implemented the change you suggested. Should be good to go!