Optimize remote store refresh performance by avoiding unnecessary metadata uploads when index state is unchanged#20192
Optimize remote store refresh performance by avoiding unnecessary metadata uploads when index state is unchanged#20192sjs004 wants to merge 15 commits into
Conversation
WalkthroughCaches last-uploaded metadata identifiers (primary term, segmentInfos generation, checkpoint version) in Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (1 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
server/src/test/java/org/opensearch/index/shard/RemoteStoreRefreshListenerTests.java (2)
404-405: RelaxedsuccessLatchcounts align with reduced engine calls, but only assert “at least once”Lowering the latch from 3 to 1 in the success‑on‑first/second/third‑attempt tests matches the expectation that
IndexShard.getEngine()is now invoked fewer times in the success path. Note that withCountDownLatchthis still only enforces “called at least once”, not “exactly once”; if you ever need to pin down the exact call count, a mock/spy verification or anAtomicLongassertion would be more precise.Also applies to: 426-427, 499-500
875-888:verifyUploadedSegmentsnow tolerates missing metadata entriesAllowing files to be absent from
uploadedSegmentsand only validating metadata when present makes the helper compatible with runs where metadata upload may have been legitimately skipped. However, this also weakens the check: it no longer guarantees that every non‑excluded local file has corresponding remote metadata, only that any present entries are sane and that a localsegments_Nfile exists.If the intent is to keep strong guarantees for cases where uploads are expected (e.g., the various
testAfter*flows), consider optionally asserting that at least one non‑excluded file appears inuploadedSegmentsor adding a dedicated assertion in those tests while leaving this helper looser for optimization cases.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
CHANGELOG.md(1 hunks)server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java(1 hunks)server/src/test/java/org/opensearch/index/shard/RemoteStoreRefreshListenerTests.java(4 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (210)
- GitHub Check: gradle-check
- GitHub Check: Mend Security Check
- GitHub Check: detect-breaking-change
- GitHub Check: precommit (25, macos-15)
- GitHub Check: precommit (25, ubuntu-latest)
- GitHub Check: precommit (25, macos-15-intel)
- GitHub Check: precommit (21, windows-2025, true)
- GitHub Check: precommit (21, ubuntu-latest)
- GitHub Check: precommit (25, windows-latest)
- GitHub Check: precommit (21, macos-15)
- GitHub Check: precommit (21, windows-latest)
- GitHub Check: Analyze (java)
- GitHub Check: precommit (21, ubuntu-24.04-arm)
- GitHub Check: precommit (25, ubuntu-24.04-arm)
- GitHub Check: assemble (25, ubuntu-latest)
- GitHub Check: assemble (25, windows-latest)
- GitHub Check: precommit (21, macos-15-intel)
- GitHub Check: assemble (21, windows-latest)
- GitHub Check: assemble (25, ubuntu-24.04-arm)
- GitHub Check: assemble (21, ubuntu-latest)
- GitHub Check: assemble (21, ubuntu-24.04-arm)
- GitHub Check: gradle-check
- GitHub Check: Mend Security Check
- GitHub Check: detect-breaking-change
- GitHub Check: precommit (25, macos-15)
- GitHub Check: precommit (25, ubuntu-latest)
- GitHub Check: precommit (25, macos-15-intel)
- GitHub Check: precommit (21, windows-2025, true)
- GitHub Check: precommit (21, ubuntu-latest)
- GitHub Check: precommit (25, windows-latest)
- GitHub Check: precommit (21, macos-15)
- GitHub Check: precommit (21, windows-latest)
- GitHub Check: Analyze (java)
- GitHub Check: precommit (21, ubuntu-24.04-arm)
- GitHub Check: precommit (25, ubuntu-24.04-arm)
- GitHub Check: assemble (25, ubuntu-latest)
- GitHub Check: assemble (25, windows-latest)
- GitHub Check: precommit (21, macos-15-intel)
- GitHub Check: assemble (21, windows-latest)
- GitHub Check: assemble (25, ubuntu-24.04-arm)
- GitHub Check: assemble (21, ubuntu-latest)
- GitHub Check: assemble (21, ubuntu-24.04-arm)
- GitHub Check: gradle-check
- GitHub Check: Mend Security Check
- GitHub Check: detect-breaking-change
- GitHub Check: precommit (25, macos-15)
- GitHub Check: precommit (25, ubuntu-latest)
- GitHub Check: precommit (25, macos-15-intel)
- GitHub Check: precommit (21, windows-2025, true)
- GitHub Check: precommit (21, ubuntu-latest)
- GitHub Check: precommit (25, windows-latest)
- GitHub Check: precommit (21, macos-15)
- GitHub Check: precommit (21, windows-latest)
- GitHub Check: Analyze (java)
- GitHub Check: precommit (21, ubuntu-24.04-arm)
- GitHub Check: precommit (25, ubuntu-24.04-arm)
- GitHub Check: assemble (25, ubuntu-latest)
- GitHub Check: assemble (25, windows-latest)
- GitHub Check: precommit (21, macos-15-intel)
- GitHub Check: assemble (21, windows-latest)
- GitHub Check: assemble (25, ubuntu-24.04-arm)
- GitHub Check: assemble (21, ubuntu-latest)
- GitHub Check: assemble (21, ubuntu-24.04-arm)
- GitHub Check: gradle-check
- GitHub Check: Mend Security Check
- GitHub Check: detect-breaking-change
- GitHub Check: precommit (25, macos-15)
- GitHub Check: precommit (25, ubuntu-latest)
- GitHub Check: precommit (25, macos-15-intel)
- GitHub Check: precommit (21, windows-2025, true)
- GitHub Check: precommit (21, ubuntu-latest)
- GitHub Check: precommit (25, windows-latest)
- GitHub Check: precommit (21, macos-15)
- GitHub Check: precommit (21, windows-latest)
- GitHub Check: Analyze (java)
- GitHub Check: precommit (21, ubuntu-24.04-arm)
- GitHub Check: precommit (25, ubuntu-24.04-arm)
- GitHub Check: assemble (25, ubuntu-latest)
- GitHub Check: assemble (25, windows-latest)
- GitHub Check: precommit (21, macos-15-intel)
- GitHub Check: assemble (21, windows-latest)
- GitHub Check: assemble (25, ubuntu-24.04-arm)
- GitHub Check: assemble (21, ubuntu-latest)
- GitHub Check: assemble (21, ubuntu-24.04-arm)
- GitHub Check: gradle-check
- GitHub Check: Mend Security Check
- GitHub Check: detect-breaking-change
- GitHub Check: precommit (25, macos-15)
- GitHub Check: precommit (25, ubuntu-latest)
- GitHub Check: precommit (25, macos-15-intel)
- GitHub Check: precommit (21, windows-2025, true)
- GitHub Check: precommit (21, ubuntu-latest)
- GitHub Check: precommit (25, windows-latest)
- GitHub Check: precommit (21, macos-15)
- GitHub Check: precommit (21, windows-latest)
- GitHub Check: Analyze (java)
- GitHub Check: precommit (21, ubuntu-24.04-arm)
- GitHub Check: precommit (25, ubuntu-24.04-arm)
- GitHub Check: assemble (25, ubuntu-latest)
- GitHub Check: assemble (25, windows-latest)
- GitHub Check: precommit (21, macos-15-intel)
- GitHub Check: assemble (21, windows-latest)
- GitHub Check: assemble (25, ubuntu-24.04-arm)
- GitHub Check: assemble (21, ubuntu-latest)
- GitHub Check: assemble (21, ubuntu-24.04-arm)
- GitHub Check: gradle-check
- GitHub Check: Mend Security Check
- GitHub Check: detect-breaking-change
- GitHub Check: precommit (25, macos-15)
- GitHub Check: precommit (25, ubuntu-latest)
- GitHub Check: precommit (25, macos-15-intel)
- GitHub Check: precommit (21, windows-2025, true)
- GitHub Check: precommit (21, ubuntu-latest)
- GitHub Check: precommit (25, windows-latest)
- GitHub Check: precommit (21, macos-15)
- GitHub Check: precommit (21, windows-latest)
- GitHub Check: Analyze (java)
- GitHub Check: precommit (21, ubuntu-24.04-arm)
- GitHub Check: precommit (25, ubuntu-24.04-arm)
- GitHub Check: assemble (25, ubuntu-latest)
- GitHub Check: assemble (25, windows-latest)
- GitHub Check: precommit (21, macos-15-intel)
- GitHub Check: assemble (21, windows-latest)
- GitHub Check: assemble (25, ubuntu-24.04-arm)
- GitHub Check: assemble (21, ubuntu-latest)
- GitHub Check: assemble (21, ubuntu-24.04-arm)
- GitHub Check: gradle-check
- GitHub Check: Mend Security Check
- GitHub Check: detect-breaking-change
- GitHub Check: precommit (25, macos-15)
- GitHub Check: precommit (25, ubuntu-latest)
- GitHub Check: precommit (25, macos-15-intel)
- GitHub Check: precommit (21, windows-2025, true)
- GitHub Check: precommit (21, ubuntu-latest)
- GitHub Check: precommit (25, windows-latest)
- GitHub Check: precommit (21, macos-15)
- GitHub Check: precommit (21, windows-latest)
- GitHub Check: Analyze (java)
- GitHub Check: precommit (21, ubuntu-24.04-arm)
- GitHub Check: precommit (25, ubuntu-24.04-arm)
- GitHub Check: assemble (25, ubuntu-latest)
- GitHub Check: assemble (25, windows-latest)
- GitHub Check: precommit (21, macos-15-intel)
- GitHub Check: assemble (21, windows-latest)
- GitHub Check: assemble (25, ubuntu-24.04-arm)
- GitHub Check: assemble (21, ubuntu-latest)
- GitHub Check: assemble (21, ubuntu-24.04-arm)
- GitHub Check: gradle-check
- GitHub Check: Mend Security Check
- GitHub Check: detect-breaking-change
- GitHub Check: precommit (25, macos-15)
- GitHub Check: precommit (25, ubuntu-latest)
- GitHub Check: precommit (25, macos-15-intel)
- GitHub Check: precommit (21, windows-2025, true)
- GitHub Check: precommit (21, ubuntu-latest)
- GitHub Check: precommit (25, windows-latest)
- GitHub Check: precommit (21, macos-15)
- GitHub Check: precommit (21, windows-latest)
- GitHub Check: Analyze (java)
- GitHub Check: precommit (21, ubuntu-24.04-arm)
- GitHub Check: precommit (25, ubuntu-24.04-arm)
- GitHub Check: assemble (25, ubuntu-latest)
- GitHub Check: assemble (25, windows-latest)
- GitHub Check: precommit (21, macos-15-intel)
- GitHub Check: assemble (21, windows-latest)
- GitHub Check: assemble (25, ubuntu-24.04-arm)
- GitHub Check: assemble (21, ubuntu-latest)
- GitHub Check: assemble (21, ubuntu-24.04-arm)
- GitHub Check: gradle-check
- GitHub Check: Mend Security Check
- GitHub Check: detect-breaking-change
- GitHub Check: precommit (25, macos-15)
- GitHub Check: precommit (25, ubuntu-latest)
- GitHub Check: precommit (25, macos-15-intel)
- GitHub Check: precommit (21, windows-2025, true)
- GitHub Check: precommit (21, ubuntu-latest)
- GitHub Check: precommit (25, windows-latest)
- GitHub Check: precommit (21, macos-15)
- GitHub Check: precommit (21, windows-latest)
- GitHub Check: Analyze (java)
- GitHub Check: precommit (21, ubuntu-24.04-arm)
- GitHub Check: precommit (25, ubuntu-24.04-arm)
- GitHub Check: assemble (25, ubuntu-latest)
- GitHub Check: assemble (25, windows-latest)
- GitHub Check: precommit (21, macos-15-intel)
- GitHub Check: assemble (21, windows-latest)
- GitHub Check: assemble (25, ubuntu-24.04-arm)
- GitHub Check: assemble (21, ubuntu-latest)
- GitHub Check: assemble (21, ubuntu-24.04-arm)
- GitHub Check: gradle-check
- GitHub Check: Mend Security Check
- GitHub Check: detect-breaking-change
- GitHub Check: precommit (25, macos-15)
- GitHub Check: precommit (25, ubuntu-latest)
- GitHub Check: precommit (25, macos-15-intel)
- GitHub Check: precommit (21, windows-2025, true)
- GitHub Check: precommit (21, ubuntu-latest)
- GitHub Check: precommit (25, windows-latest)
- GitHub Check: precommit (21, macos-15)
- GitHub Check: precommit (21, windows-latest)
- GitHub Check: Analyze (java)
- GitHub Check: precommit (21, ubuntu-24.04-arm)
- GitHub Check: precommit (25, ubuntu-24.04-arm)
- GitHub Check: assemble (25, ubuntu-latest)
- GitHub Check: assemble (25, windows-latest)
- GitHub Check: precommit (21, macos-15-intel)
- GitHub Check: assemble (21, windows-latest)
- GitHub Check: assemble (25, ubuntu-24.04-arm)
- GitHub Check: assemble (21, ubuntu-latest)
- GitHub Check: assemble (21, ubuntu-24.04-arm)
🔇 Additional comments (3)
CHANGELOG.md (1)
27-27: Changelog entry accurately reflects behavior changeDescription and PR link look correct and match the implementation in
RemoteStoreRefreshListener. No further changes needed.server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java (1)
270-276: Metadata upload is now correctly gated on “any new segments”Reusing
skipUploadto decide whether all post‑refresh files are already present remotely is consistent with the existing upload filter andisRemoteSegmentStoreInSync(). GivencontainsFileis keyed off remote metadata, this preserves the invariant that metadata is uploaded whenever new segments exist, and skipped only when the remote store is already in sync. Looks good.server/src/test/java/org/opensearch/index/shard/RemoteStoreRefreshListenerTests.java (1)
908-927:testUploadsWhenNewSegmentsPresentcorrectly guards against regressions in segment uploadsThis integration test validates that adding a new document leads to additional uploaded segments, using
assertBusyto accommodate async behavior. It’s a good complement to the optimization: ensures we didn’t inadvertently suppress uploads when new segments are present.
| public void testSkipsMetadataUploadWhenNoNewSegments() throws IOException { | ||
| setup(true, 2); | ||
|
|
||
| indexShard.refresh("initial-upload"); | ||
|
|
||
| try (Store remoteStore = indexShard.remoteStore()) { | ||
| RemoteSegmentStoreDirectory remoteSegmentStoreDirectory = | ||
| (RemoteSegmentStoreDirectory) ((FilterDirectory) ((FilterDirectory) remoteStore.directory()).getDelegate()).getDelegate(); | ||
|
|
||
| final int initialCount = remoteSegmentStoreDirectory.getSegmentsUploadedToRemoteStore().size(); | ||
|
|
||
| indexShard.refresh("test-optimization"); | ||
|
|
||
| int afterOptimizationCount = remoteSegmentStoreDirectory.getSegmentsUploadedToRemoteStore().size(); | ||
| assertEquals("No new segments should be uploaded when optimization kicks in", initialCount, afterOptimizationCount); | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
testSkipsMetadataUploadWhenNoNewSegments doesn’t currently prove metadata uploads are skipped
This test asserts that the number of entries in getSegmentsUploadedToRemoteStore() is unchanged across two refreshes with no new docs. That condition should already hold even without the new optimization, since no additional segment blobs are uploaded when there are no new segments.
If you want this test to specifically guard the “skip metadata upload” behavior, you may want to instrument or spy on the remote directory (e.g., verify uploadMetadata is not called on the second refresh) instead of or in addition to checking the segment count.
🤖 Prompt for AI Agents
In
server/src/test/java/org/opensearch/index/shard/RemoteStoreRefreshListenerTests.java
around lines 890 to 907, the test only checks segment count which does not prove
metadata upload was skipped; replace or augment the test to instrument/spy the
remote directory and assert that its metadata-upload method is not invoked on
the second refresh. Specifically, obtain a spy or mock of the
RemoteSegmentStoreDirectory (or the remote directory wrapper), run the first
refresh and verify uploadMetadata was called once, run the second refresh and
verify uploadMetadata was NOT called (or that its invocation count did not
increase); use your test framework’s mocking utilities (e.g., Mockito
spy/verify) to assert the expected call counts and keep the existing
segment-count assertion if desired.
|
❌ Gradle check result for 7d7e4f1: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
I tried to fix Test failures for this PR but I believe changes maybe incomplete. Snapshot State post this change is coming as |
Signed-off-by: Hyunsang Han <gustkd3@gmail.com>
Signed-off-by: Hyunsang Han <gustkd3@gmail.com>
Signed-off-by: Hyunsang Han <gustkd3@gmail.com>
…WhenNewSegmentsPresent Signed-off-by: Hyunsang Han <gustkd3@gmail.com>
Signed-off-by: Hyunsang Han <gustkd3@gmail.com>
|
This PR is stalled because it has been open for 30 days with no activity. |
Description
Optimizes remote store refresh performance by avoiding unnecessary metadata uploads when index state is unchanged
Related Issues
Resolves #17530
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.