Adding warn logging for timed out remote state read tasks by gargharsh3134 · Pull Request #18019 · opensearch-project/OpenSearch

gargharsh3134 · 2025-04-22T00:26:04Z

Description

This change focuses on improving logging for remoteState read tasks which might get timed out and lead to node drops. Absence of logging hinders the ability to identify the task/blob entity which is leading to timeout issues while reading diffed cluster state from remote.
This change introduces a new dynamic cluster setting to identify the threshold for logging the details of a slow task.


[2025-02-17T11:31:01,012][ERROR][o.o.g.r.RemoteClusterStateService] [b2c6c40ba58ebb56b0020b8a15c196a4] Failure in downloading diff cluster state.
org.opensearch.gateway.remote.RemoteStateTransferException: Timed out waiting to read cluster state from remote within timeout 40s
        at org.opensearch.gateway.remote.RemoteClusterStateService.readClusterStateInParallel(RemoteClusterStateService.java:1397)
        at org.opensearch.gateway.remote.RemoteClusterStateService.getClusterStateUsingDiff(RemoteClusterStateService.java:1623)
        at org.opensearch.cluster.coordination.PublicationTransportHandler.handleIncomingRemotePublishRequest(PublicationTransportHandler.java:289)
        at org.opensearch.cluster.coordination.PublicationTransportHandler.lambda$new$1(PublicationTransportHandler.java:136)

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-04-22T01:06:59Z

❌ Gradle check result for 643f969: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-04-24T04:01:02Z

❌ Gradle check result for 5b024f7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/common/remote/RemoteWriteableEntityBlobStore.java

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java

server/src/main/java/org/opensearch/common/remote/RemoteWriteableEntityBlobStore.java

server/src/test/java/org/opensearch/gateway/remote/RemoteClusterStateServiceTests.java

server/src/main/java/org/opensearch/common/remote/RemoteWriteableEntityBlobStore.java

github-actions · 2025-05-27T05:43:47Z

❌ Gradle check result for 8d22e1d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/common/remote/ReadBlobWithMetrics.java

...r/src/main/java/org/opensearch/cluster/routing/remote/InternalRemoteRoutingTableService.java

github-actions · 2025-07-08T14:18:22Z

❌ Gradle check result for 683e211: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

...r/src/main/java/org/opensearch/cluster/routing/remote/InternalRemoteRoutingTableService.java

server/src/main/java/org/opensearch/gateway/remote/model/RemoteReadResultsVerbose.java

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java

server/src/main/java/org/opensearch/common/remote/RemoteWriteableEntityBlobStore.java

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java

github-actions · 2025-07-30T13:07:44Z

❌ Gradle check result for c0e71c6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-31T09:05:49Z

❌ Gradle check result for 68287d8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-31T22:03:07Z

❌ Gradle check result for 1870449: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

rajiv-kv

Changes LGTM ! Please fix the flaky tests.

rajiv-kv · 2025-08-01T05:53:29Z

server/src/test/java/org/opensearch/gateway/remote/RemoteClusterStateServiceTests.java

        );
-        assertEquals("Timed out waiting to read cluster state from remote within timeout " + readTimeOut + "s", exception.getMessage());
+        // All lists and maps are passed as empty, while other boolean variables are set to true.
+        // So, for readClusterStateInParallel() total read tasks would be 7


Can we add reason as to why there are 7 files to be read ?

rajiv-kv · 2025-08-01T05:57:47Z

server/src/main/java/org/opensearch/common/remote/RemoteWriteableEntityBlobStore.java

-                listener.onResponse(read(entity));
+                RemoteReadResult<T> result = readWithMetrics(entity, component, componentName);
+                listener.onResponse(result);
+                final long executionTimeMS = Math.max(0, TimeValue.nsecToMSec(System.nanoTime() - queueStartTimeNS));


Should we log before calling the listener#onResponse to be consistent with exception block ?

Can we dedupe the logging code to a private method ?

Signed-off-by: Harsh Garg <gkharsh@amazon.com>

github-actions · 2025-08-01T10:56:37Z

✅ Gradle check result for 4e3fac1: SUCCESS

codecov · 2025-08-01T10:56:58Z

Codecov Report

❌ Patch coverage is 56.12245% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.88%. Comparing base (9b22c9b) to head (4e3fac1).
⚠️ Report is 740 commits behind head on main.

Files with missing lines	Patch %	Lines
...arch/gateway/remote/RemoteClusterStateService.java	38.09%	21 Missing and 5 partials ⚠️
.../common/remote/RemoteWriteableEntityBlobStore.java	63.41%	12 Missing and 3 partials ⚠️
...nsearch/gateway/remote/model/RemoteReadResult.java	60.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18019      +/-   ##
============================================
+ Coverage     72.77%   72.88%   +0.10%     
- Complexity    68690    68826     +136     
============================================
  Files          5582     5587       +5     
  Lines        315456   315797     +341     
  Branches      45778    45825      +47     
============================================
+ Hits         229568   230154     +586     
+ Misses        67290    66989     -301     
- Partials      18598    18654      +56

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

opensearch-trigger-bot · 2025-08-31T15:22:03Z

This PR is stalled because it has been open for 30 days with no activity.

gargharsh3134 force-pushed the remoteLogging branch from 643f969 to 5b024f7 Compare April 24, 2025 03:51

SwethaGuptha reviewed May 2, 2025

View reviewed changes

server/src/main/java/org/opensearch/common/remote/RemoteWriteableEntityBlobStore.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java Show resolved Hide resolved

rajiv-kv reviewed May 12, 2025

View reviewed changes

rajiv-kv reviewed Jun 7, 2025

View reviewed changes

server/src/main/java/org/opensearch/common/remote/ReadBlobWithMetrics.java Outdated Show resolved Hide resolved

...r/src/main/java/org/opensearch/cluster/routing/remote/InternalRemoteRoutingTableService.java Outdated Show resolved Hide resolved

rajiv-kv reviewed Jul 28, 2025

View reviewed changes

gargharsh3134 force-pushed the remoteLogging branch from 683e211 to c0e71c6 Compare July 30, 2025 12:17

gargharsh3134 marked this pull request as ready for review July 30, 2025 12:21

gargharsh3134 requested a review from a team as a code owner July 30, 2025 12:21

This was referenced Jul 30, 2025

[AUTOCUT] Gradle Check Flaky Test Report for AzureBlobStoreRepositoryTests #14291

Open

[AUTOCUT] Gradle Check Flaky Test Report for RecoveryWhileUnderLoadIT #14509

Open

gargharsh3134 force-pushed the remoteLogging branch from 68287d8 to 1870449 Compare July 31, 2025 21:09

rajiv-kv reviewed Aug 1, 2025

View reviewed changes

Harsh Garg added 2 commits August 1, 2025 14:59

Improving logging for timed out remote state read tasks

06c213b

Signed-off-by: Harsh Garg <gkharsh@amazon.com>

Retry Build

4e3fac1

Signed-off-by: Harsh Garg <gkharsh@amazon.com>

gargharsh3134 force-pushed the remoteLogging branch from 1870449 to 4e3fac1 Compare August 1, 2025 09:30

This was referenced Aug 13, 2025

[AUTOCUT] Gradle Check Flaky Test Report for SearchOnlyReplicaIT #15812

Open

[AUTOCUT] Gradle Check Flaky Test Report for QueryPhaseTests #14551

Open

opensearch-trigger-bot bot added stalled Issues that have stalled and removed stalled Issues that have stalled labels Aug 31, 2025

opensearch-ci-bot mentioned this pull request Sep 17, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RemoteClusterStateServiceTests #19325

Closed

rajiv-kv approved these changes Oct 10, 2025

View reviewed changes

Conversation

gargharsh3134 commented Apr 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Apr 22, 2025

Uh oh!

github-actions bot commented Apr 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 27, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 30, 2025

Uh oh!

github-actions bot commented Jul 31, 2025

Uh oh!

github-actions bot commented Jul 31, 2025

Uh oh!

rajiv-kv left a comment

Choose a reason for hiding this comment

Uh oh!

rajiv-kv Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

rajiv-kv Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 1, 2025

Uh oh!

codecov bot commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

opensearch-trigger-bot bot commented Aug 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gargharsh3134 commented Apr 22, 2025 •

edited

Loading

codecov bot commented Aug 1, 2025 •

edited

Loading