[BUG FIX] Add reference count control in NRTReplicationEngine#acquireLastIndexCommit by guojialiang92 · Pull Request #19214 · opensearch-project/OpenSearch

guojialiang92 · 2025-09-02T08:47:10Z

Description

The purpose of this PR is to address the issues #19213.

When obtaining GatedCloseable<IndexCommit> in NRTReplicationEngine#acquireLastIndexCommit, the reference count of the files it contains is incremented by 1. When performing the close operation, the reference count is decremented by 1.

I introduced tests which could reproduce the issue stably before this fix:

SegmentReplicationIT#testAcquireLastIndexCommit
NRTReplicationEngineTests#testAcquireLastIndexCommit

Related Issues

Resolves #[19213]

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-09-02T08:50:41Z

❌ Gradle check result for ff2a581: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-09-02T11:30:28Z

❌ Gradle check result for b069d27: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

github-actions · 2025-09-03T02:01:41Z

❌ Gradle check result for abe8112: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java

github-actions · 2025-09-08T09:01:31Z

❌ Gradle check result for e469570: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

…managed by lock. Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

github-actions · 2025-09-08T10:52:02Z

❕ Gradle check result for e622f98: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

kkewwei · 2025-09-08T12:20:14Z

@guojialiang92 Can you add the CHANGELOG.txt?

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

…ine-acquireLastIndexCommit-add-ref-count-control

github-actions · 2025-09-08T14:24:33Z

❕ Gradle check result for 92a555c: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

guojialiang92 · 2025-09-08T14:28:28Z

@guojialiang92 Can you add the CHANGELOG.txt?

@kkewwei
Yeah, I have already modified the CHANGELOG.txt.
Could you please help review it again?

kkewwei

LGTM

…LastIndexCommit (opensearch-project#19214) * NRTReplicationEngine#acquireLastIndexCommit add ref count control Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * fix test Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * the incRef in NRTReplicationEngine#commitSegmentInfos should also be managed by lock. Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * update test Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * add change log Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> --------- Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

…LastIndexCommit (opensearch-project#19214) * NRTReplicationEngine#acquireLastIndexCommit add ref count control Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * fix test Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * the incRef in NRTReplicationEngine#commitSegmentInfos should also be managed by lock. Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * update test Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * add change log Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> --------- Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> Signed-off-by: Ankit Jain <jainankitk@apache.org>

…LastIndexCommit (opensearch-project#19214) * NRTReplicationEngine#acquireLastIndexCommit add ref count control Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * fix test Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * the incRef in NRTReplicationEngine#commitSegmentInfos should also be managed by lock. Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * update test Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * add change log Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> --------- Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

guojialiang92 mentioned this pull request Sep 2, 2025

[BUG] Traversing the files returned by NRTReplicationEngine#acquireLastIndexCommit may throw NoSuchFileException. #19213

Closed

guojialiang92 force-pushed the dev/NRTReplicationEngine-acquireLastIndexCommit-add-ref-count-control branch from ff2a581 to b069d27 Compare September 2, 2025 11:27

NRTReplicationEngine#acquireLastIndexCommit add ref count control

abe8112

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

guojialiang92 force-pushed the dev/NRTReplicationEngine-acquireLastIndexCommit-add-ref-count-control branch from b069d27 to abe8112 Compare September 3, 2025 00:47

guojialiang92 marked this pull request as draft September 3, 2025 05:32

fix test

0eada7a

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

guojialiang92 marked this pull request as ready for review September 3, 2025 07:53

kkewwei reviewed Sep 5, 2025

View reviewed changes

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java Show resolved Hide resolved

the incRef in NRTReplicationEngine#commitSegmentInfos should also be …

e622f98

…managed by lock. Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

guojialiang92 force-pushed the dev/NRTReplicationEngine-acquireLastIndexCommit-add-ref-count-control branch from e469570 to e622f98 Compare September 8, 2025 09:26

guojialiang92 added 3 commits September 8, 2025 20:45

update test

4935c72

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

add change log

4cb11e5

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

Merge remote-tracking branch 'origin/main' into dev/NRTReplicationEng…

92a555c

…ine-acquireLastIndexCommit-add-ref-count-control

kkewwei approved these changes Sep 9, 2025

View reviewed changes

kkewwei merged commit ceb27a2 into opensearch-project:main Sep 9, 2025
31 checks passed

opensearch-ci-bot mentioned this pull request Sep 8, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreReplicationSourceTests #16683

Open

BrewTestBot mentioned this pull request Oct 15, 2025

opensearch 3.3.0 Homebrew/homebrew-core#249627

Merged

This was referenced Jan 10, 2026

docs: add NRT Replication Engine report for v3.3.0 tkykenmt/opensearch-feature-explorer#1480

Merged

[feature] NRT Replication Engine tkykenmt/opensearch-feature-explorer#1429

Closed

opensearch-ci-bot mentioned this pull request Feb 18, 2026

[AUTOCUT] Gradle Check Flaky Test Report for SegmentReplicationUsingRemoteStoreIT #14329

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG FIX] Add reference count control in NRTReplicationEngine#acquireLastIndexCommit#19214

[BUG FIX] Add reference count control in NRTReplicationEngine#acquireLastIndexCommit#19214
kkewwei merged 6 commits intoopensearch-project:mainfrom
guojialiang92:dev/NRTReplicationEngine-acquireLastIndexCommit-add-ref-count-control

guojialiang92 commented Sep 2, 2025

Uh oh!

github-actions bot commented Sep 2, 2025

Uh oh!

github-actions bot commented Sep 2, 2025

Uh oh!

github-actions bot commented Sep 3, 2025

Uh oh!

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

kkewwei commented Sep 8, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

guojialiang92 commented Sep 8, 2025

Uh oh!

kkewwei left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

guojialiang92 commented Sep 2, 2025

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Sep 2, 2025

Uh oh!

github-actions bot commented Sep 2, 2025

Uh oh!

github-actions bot commented Sep 3, 2025

Uh oh!

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

kkewwei commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

guojialiang92 commented Sep 8, 2025

Uh oh!

kkewwei left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kkewwei commented Sep 8, 2025 •

edited

Loading