[Backport 2.x] Add threadpool wait time metric (#9681)

[jed326]

Description

Backport PR for #9681

To be merged at the same time as #10004

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25422/
• CommitID: e1c3c31
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change 5a30acc

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25430/
• CommitID: 42edfad
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25443/
• CommitID: f16bd34
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ABORTED ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25452/
• CommitID: 9db6ded
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25459/
• CommitID: 53467d7
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[jed326]

Will take a look at the flakey concurrent search stats test

--

Ah the slice count setting is not dynamic in 2.x. Let me see if I can rework the test then, if not will just add an @ignore for now.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25464/
• CommitID: 87cc5da
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25465/
• CommitID: ec8fe05
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/25468/
• CommitID: 5a30acc

[codecov]

Codecov Report

Merging #10000 (5a30acc) into 2.x (e4ae8fe) will increase coverage by 0.04%.
Report is 4 commits behind head on 2.x.
The diff coverage is 51.72%.

@@             Coverage Diff              @@
##                2.x   #10000      +/-   ##
============================================
+ Coverage     70.77%   70.81%   +0.04%     
+ Complexity    58209    58205       -4     
============================================
  Files          4810     4810              
  Lines        275125   275150      +25     
  Branches      40433    40439       +6     
============================================
+ Hits         194723   194851     +128     
+ Misses        63814    63656     -158     
- Partials      16588    16643      +55     

Files Changed	Coverage Δ
...dination/UnsafeBootstrapClusterManagerCommand.java	0.00% <0.00%> (ø)
...ensearch/rest/action/cat/RestThreadPoolAction.java	11.94% <0.00%> (-0.19%)	⬇️
...ensearch/common/util/concurrent/TimedRunnable.java	63.33% <33.33%> (-3.34%)	⬇️
...ava/org/opensearch/threadpool/ThreadPoolStats.java	77.64% <54.54%> (-3.69%)	⬇️
...ain/java/org/opensearch/threadpool/ThreadPool.java	84.68% <80.00%> (+0.67%)	⬆️
.../util/concurrent/OpenSearchThreadPoolExecutor.java	85.10% <100.00%> (+0.32%)	⬆️
...ent/QueueResizingOpenSearchThreadPoolExecutor.java	58.24% <100.00%> (+1.42%)	⬆️

... and 477 files with indirect coverage changes