Concurrent Search Tasks Response Updates

[jed326]

Description

After #7502 has been merged we will start tracking the resource usage of the concurrent search threadpool. This change adds average,min,max resource usage to the tasks API response which are more meaningful in the concurrent search use case. These stats will only be shown whenever thread executions > 1.

{
          "node" : "0h_eKuenQgmRxhGGtDwsDg",
          "id" : 63,
          "type" : "direct",
          "action" : "indices:data/read/search[phase/query]",
          "description" : "shardId[[my-index-000001][0]]",
          "start_time_in_millis" : 1685467779105,
          "running_time_in_nanos" : 11345102263,
          "cancellable" : true,
          "cancelled" : false,
          "parent_task_id" : "0h_eKuenQgmRxhGGtDwsDg:62",
          "headers" : { },
          "resource_stats" : {
            "average" : {
              "cpu_time_in_nanos" : 13312500,
              "memory_in_bytes" : 1059688
            },
            "total" : {
              "cpu_time_in_nanos" : 26625000,
              "memory_in_bytes" : 2119376
            },
            "min" : {
              "cpu_time_in_nanos" : 29000,
              "memory_in_bytes" : 480
            },
            "max" : {
              "cpu_time_in_nanos" : 26596000,
              "memory_in_bytes" : 2118896
            },
            "thread_info" : {
              "thread_executions" : 2,
              "active_threads" : 2
            }
          }
        }

This is an implementation of solution 1 in the design from #7425.

Related Issues

#7425
Doc update Issue: opensearch-project/documentation-website#4206

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15976/
• CommitID: 00831b2
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15980/
• CommitID: cbe7e82
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15982/
• CommitID: 1efa636
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15990/
• CommitID: 62c234c
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16102/
• CommitID: 96946d7
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16103/
• CommitID: 03b7df5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16104/
• CommitID: 8922ed8
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16105/
• CommitID: c400861
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16113/
• CommitID: 7c2d1d8
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16108/
• CommitID: 3217749
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16116/
• CommitID: ebb342e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[jed326]

Failing test seems unrelated to my changes - https://build.ci.opensearch.org/job/gradle-check/16116/testReport/

org.opensearch.index.query.IntervalQueryBuilderTests > testToQuery FAILED
    java.lang.IllegalArgumentException: Too many disjunctions to expand
        at __randomizedtesting.SeedInfo.seed([5334F0F28503703F:A4CFF2CCF480B5D5]:0)
        at org.apache.lucene.queries.intervals.Disjunctions.pullUp(Disjunctions.java:45)
        at org.apache.lucene.queries.intervals.OrderedIntervalsSource.pullUpDisjunctions(OrderedIntervalsSource.java:91)
        at org.apache.lucene.queries.intervals.FilteredIntervalsSource.maxGaps(FilteredIntervalsSource.java:33)

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/16607/
• CommitID: 4cc8b78

[jed326]

@reta Not sure why it gave me a merge conflict on that last commit but I think we should be good to squash and merge now. It's rerunning the tests but as you can see the last change has no diff.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/16617/
• CommitID: e90d8ca

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu

• URL: https://build.ci.opensearch.org/job/gradle-check/16620/
• CommitID: 5489641
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure
      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=tasks.list/10_basic/tasks_list test}

• URL: https://build.ci.opensearch.org/job/gradle-check/16627/
• CommitID: f35c232
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[jed326]

Looks like that latest test failure could be related to my change, diving into it.

[jed326]

There was an NPE issue in the constructor, should be fixed with the latest commit.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/16628/
• CommitID: ea2b9dd