[Backport 2.x] Adding concurrent search versions of query count and time metrics (#9622) by jed326 · Pull Request #9653 · opensearch-project/OpenSearch

jed326 · 2023-08-31T01:12:17Z

Description

2.x Backport PR for concurrent segment search related metrics

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-08-31T01:39:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23896/
CommitID: 6372960
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-31T01:43:49Z

Compatibility status:

Checks if related components are compatible with change 141a5f5

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git]

github-actions · 2023-08-31T01:46:50Z

Compatibility status:

Checks if related components are compatible with change 6372960

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

github-actions · 2023-08-31T01:59:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23897/
CommitID: 141a5f5

codecov · 2023-08-31T02:00:00Z

Codecov Report

Merging #9653 (5cf6492) into 2.x (58ae69d) will decrease coverage by 0.08%.
Report is 6 commits behind head on 2.x.
The diff coverage is 62.94%.

@@             Coverage Diff              @@
##                2.x    #9653      +/-   ##
============================================
- Coverage     70.80%   70.73%   -0.08%     
- Complexity    57868    57871       +3     
============================================
  Files          4793     4794       +1     
  Lines        273397   273606     +209     
  Branches      40216    40259      +43     
============================================
- Hits         193583   193530      -53     
- Misses        63235    63521     +286     
+ Partials      16579    16555      -24

Files Changed	Coverage Δ
...rg/opensearch/common/settings/ClusterSettings.java	`93.18% <ø> (ø)`
...mmon/util/concurrent/BufferedAsyncIOProcessor.java	`67.74% <0.00%> (-2.26%)`	⬇️
...rc/main/java/org/opensearch/index/IndexModule.java	`82.35% <ø> (ø)`
...ch/repositories/blobstore/BlobStoreRepository.java	`58.84% <0.00%> (-0.39%)`	⬇️
.../opensearch/rest/action/cat/RestIndicesAction.java	`45.00% <0.00%> (-0.78%)`	⬇️
...rg/opensearch/rest/action/cat/RestNodesAction.java	`50.96% <0.00%> (-1.70%)`	⬇️
...g/opensearch/rest/action/cat/RestShardsAction.java	`84.41% <0.00%> (-3.82%)`	⬇️
...rg/opensearch/repositories/s3/S3BlobContainer.java	`78.66% <33.33%> (-1.47%)`	⬇️
...opensearch/index/SegmentReplicationShardStats.java	`39.13% <40.00%> (+0.10%)`	⬆️
...replication/common/SegmentReplicationLagTimer.java	`40.00% <40.00%> (ø)`
... and 15 more

... and 466 files with indirect coverage changes

jed326 · 2023-08-31T13:32:39Z

@reta would you mind double checking the bwc related changes in this backport PR? Thanks!

github-actions · 2023-08-31T13:57:43Z

Compatibility status:

Checks if related components are compatible with change 3cac1e5

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

github-actions · 2023-08-31T14:19:34Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23976/
CommitID: 3cac1e5

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java

jed326 · 2023-08-31T19:06:16Z

Will use this PR to backport #9670 as well.

…ensearch-project#9622) Signed-off-by: Jay Deng <jayd0104@gmail.com>

github-actions · 2023-09-01T21:21:24Z

Compatibility status:

Checks if related components are compatible with change 4a4b5ee

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git]

github-actions · 2023-09-01T21:33:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24167/
CommitID: 4a4b5ee
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

jed326 · 2023-09-04T06:46:53Z

Hey @reta now that #9670 is merged I updated this PR to do the backport for both concurrent segment search metrics PRs.
As requested, I also opened #9722 as well to apply these bwc changes on main at the same time. Could you please help review & merge both? Thanks!

github-actions · 2023-09-04T06:53:56Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24306/
CommitID: d15943a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-04T06:59:43Z

Compatibility status:

Checks if related components are compatible with change d15943a

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git]

…pensearch-project#9670) Signed-off-by: Jay Deng <jayd0104@gmail.com>

github-actions · 2023-09-04T07:29:21Z

Compatibility status:

Checks if related components are compatible with change f54ada3

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

github-actions · 2023-09-04T07:45:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication

URL: https://build.ci.opensearch.org/job/gradle-check/24311/
CommitID: f54ada3
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

github-actions · 2023-09-04T14:34:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24347/
CommitID: 5cf6492
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-04T15:00:44Z

Compatibility status:

Checks if related components are compatible with change 5cf6492

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git]

github-actions · 2023-09-04T15:45:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24355/
CommitID: 5cf6492
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-04T16:29:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/24356/
CommitID: 5cf6492

jed326 force-pushed the backport-9622-to-2.x branch from 6372960 to 141a5f5 Compare August 31, 2023 01:13

tlfeng added backport PRs or issues specific to backporting features or enhancments v2.10.0 labels Aug 31, 2023

jed326 marked this pull request as ready for review August 31, 2023 13:31

jed326 requested review from Bukhtawar, CEHENKLE, Rishikesh1159, VachaShah, anasalkouz, andrross, dblock, dbwiddis, dreamer-89, gbbafna, kartg, kotwanikunal, mch2, nknize, owaiskazi19, reta, ryanbogan, saratvemulapalli, setiah, shwetathareja and tlfeng as code owners August 31, 2023 13:31

jed326 requested a review from msfroh as a code owner August 31, 2023 13:31

reta reviewed Aug 31, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java Show resolved Hide resolved

reta reviewed Aug 31, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java Show resolved Hide resolved

reta approved these changes Aug 31, 2023

View reviewed changes

jed326 marked this pull request as draft August 31, 2023 17:48

Adding concurrent search versions of query count and time metrics (op…

4a4b5ee

…ensearch-project#9622) Signed-off-by: Jay Deng <jayd0104@gmail.com>

jed326 force-pushed the backport-9622-to-2.x branch from 3cac1e5 to 4a4b5ee Compare September 1, 2023 20:43

This was referenced Sep 4, 2023

Add average query concurrency metric for concurrent segment search #9670

Merged

Fix bwc version for concurrent segment search stats #9722

Merged

jed326 marked this pull request as ready for review September 4, 2023 06:44

Add average query concurrency metric for concurrent segment search (o…

f54ada3

…pensearch-project#9670) Signed-off-by: Jay Deng <jayd0104@gmail.com>

jed326 force-pushed the backport-9622-to-2.x branch from d15943a to f54ada3 Compare September 4, 2023 07:00

Merge branch '2.x' into backport-9622-to-2.x

5cf6492

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

reta merged commit b3b1091 into opensearch-project:2.x Sep 4, 2023

jed326 deleted the backport-9622-to-2.x branch September 13, 2023 19:10

Conversation

jed326 commented Aug 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Check List

Uh oh!

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

Uh oh!

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

Uh oh!

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

codecov bot commented Aug 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jed326 commented Aug 31, 2023

Uh oh!

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

Uh oh!

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Uh oh!

Uh oh!

jed326 commented Aug 31, 2023

Uh oh!

github-actions bot commented Sep 1, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

Uh oh!

github-actions bot commented Sep 1, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

jed326 commented Sep 4, 2023

Uh oh!

github-actions bot commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Sep 4, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

Uh oh!

github-actions bot commented Sep 4, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

Uh oh!

github-actions bot commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Sep 4, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

jed326 commented Aug 31, 2023 •

edited

Loading

codecov bot commented Aug 31, 2023 •

edited

Loading