Change INDEX_SEARCHER threadpool to resizable to support task resource tracking

[jed326]

Description

Change the INDEX_SEARCHER threadpool type to QueueResizableOpenSearchThreadPoolExecutor to support task resource tracking for concurrent segment search.

• Added new ConcurrentSearchTasksIT test to validate that multiple threads are reporting resource stats for the same task with concurrent search enabled. Note that this is separate from [Concurrent Segment Search] Enable Search ITs with Concurrent Segment Search #7440.
• Added new UT testStartingTrackingHandlesMultipleThreadsPerTask to TaskResourceTrackingServiceTests to check that TaskResourceTrackingService properly handles multiple threads per task.

We already have existing tests for passing thread context between threadpools so I did not add any more tests related to TaskAwareRunnable.

Related Issues

#7425

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Resolves #7425

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15166/
• CommitID: c9c0dbb
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15168/
• CommitID: cd88594
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15169/
• CommitID: 68beb41
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/15171/
• CommitID: 1480aba

[codecov-commenter]

Codecov Report

Merging #7502 (3e11e38) into main (27dbcd5) will decrease coverage by 0.11%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##               main    #7502      +/-   ##
============================================
- Coverage     70.54%   70.44%   -0.11%     
+ Complexity    59713    59674      -39     
============================================
  Files          4896     4896              
  Lines        286798   286798              
  Branches      41331    41331              
============================================
- Hits         202334   202033     -301     
- Misses        67761    68098     +337     
+ Partials      16703    16667      -36     

Impacted Files	Coverage Δ
...ain/java/org/opensearch/threadpool/ThreadPool.java	82.47% <0.00%> (-0.58%)	⬇️

... and 534 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/15172/
• CommitID: 222c08f

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/15174/
• CommitID: e8f742a
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[jed326]

Seems like a known flaky test #7401

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15446/
• CommitID: 6c013b5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15444/
• CommitID: 51e7892
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[reta]

Ah I see. If numThreads is large then the same thread may pick up multiple Runnables for the same task id. The other problem is task.getResourceStats() has a max size of the number of threads in the threadpool.

Have we figure out (and fixed) the flakiness part? thank you

[jed326]

@reta Yep the flakiness and your last round of comments are both addressed now.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15577/
• CommitID: 6be87cb
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/15579/
• CommitID: 6bad124

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testReplicaHasDiffFilesThanPrimary
      1 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.classMethod

• URL: https://build.ci.opensearch.org/job/gradle-check/15594/
• CommitID: 3e11e38
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.