[Segment Replication] Refactor file cleanup logic and fix PIT/Scroll with remote store. by mch2 · Pull Request #9111 · opensearch-project/OpenSearch

mch2 · 2023-08-04T04:29:44Z

Description

This change fixes multiple issues around scroll/PIT tests with Segment Replication and remote store. These issues stem from different logic in NRTReplicationEngine around how segments and commit points are preserved between refresh cycles. With node to node we are only performing local commits and preserving the latest on-disk commit but with remote store it was possible for a new incoming commit point to leave a still required commit point available to deletion as its not the "latest commit" on disk.

With this change all replicas with segrep enabled perform local commits when necessary from the incoming SegmentInfos byte[] only and ignore any incoming segments_n from its replication source. This PR also changes the recovery sync with remote store to exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened.

This change also simplifies deletion logic with segment replication to automatically delete when a file is decref'd to 0 to make it easier to reason about when files/cleanup is performed.
Files are Incref'd when they are loaded on to the reader and when committed or when a segmentInfosSnapshot is acquired.
Files are decref'd after a new commit is made, when a reader is closed, or a segmentInfosSnapshot is closed.

Related Issues

Resolves #8850
Resolves #7556
Resolves #8777

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

opensearch-trigger-bot · 2023-08-04T04:56:51Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/reporting.git]

BUILD SUCCESSFUL in 26m 43s

github-actions · 2023-08-04T05:03:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21860/
CommitID: 334ab1d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-04T08:31:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21868/
CommitID: 334ab1d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-07T08:19:11Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21992/
CommitID: 334ab1d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

tlfeng · 2023-08-07T17:23:59Z

The above 2 build having the same test failure

> Task :server:test

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.index.engine.NRTReplicationEngineTests.testUpdateSegments_replicaReceivesSISWithLowerGen" -Dtests.seed=6F54B4D34B49FD4 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-MT -Dtests.timezone=America/St_Vincent -Druntime.java=20

org.opensearch.index.engine.NRTReplicationEngineTests > testUpdateSegments_replicaReceivesSISWithLowerGen FAILED
    java.lang.AssertionError: expected:<5> but was:<7>
        at __randomizedtesting.SeedInfo.seed([6F54B4D34B49FD4:FD75BA2C32425ECF]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.opensearch.index.engine.NRTReplicationEngineTests.testUpdateSegments_replicaReceivesSISWithLowerGen(NRTReplicationEngineTests.java:161)

mch2 · 2023-08-07T17:42:26Z

Thanks @tlfeng will get this cleaned up.

github-actions · 2023-08-08T07:50:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22078/
CommitID: 986f55a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-08T08:19:16Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/security-analytics.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 35m 54s

opensearch-trigger-bot · 2023-08-09T06:20:55Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 26m 54s

github-actions · 2023-08-09T06:25:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22201/
CommitID: 12878c5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-09T06:34:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22202/
CommitID: 2e43482
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-09T06:44:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22200/
CommitID: de713f2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-09T06:52:22Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 29m 45s

opensearch-trigger-bot · 2023-08-09T06:57:15Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 39m 43s

This change removes divergent commit paths for segrep node-node and remote store. All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file. This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened. This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0. Signed-off-by: Marc Handalian <handalm@amazon.com>

Signed-off-by: Marc Handalian <handalm@amazon.com>

…it a new segmentInfos. Signed-off-by: Marc Handalian <handalm@amazon.com>

Signed-off-by: Marc Handalian <handalm@amazon.com>

opensearch-trigger-bot · 2023-08-10T21:32:38Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/reporting.git]
Compatible components: [https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

BUILD SUCCESSFUL in 24m 39s

github-actions · 2023-08-10T21:52:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22373/
CommitID: 1989a2e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

dreamer-89 · 2023-08-10T22:03:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/22373/

CommitID: 1989a2e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Assertion trip here

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2679, name=opensearch[node_s1][flush][T#1], state=RUNNABLE, group=TGRP-RemoteStoreForceMergeIT]
	at __randomizedtesting.SeedInfo.seed([90BCF02533C9A76B:61FEE78AB3B48CC0]:0)
Caused by: java.lang.AssertionError: [remote-store-test-idx-1][0] Expected non-empty readers
	at __randomizedtesting.SeedInfo.seed([90BCF02533C9A76B]:0)
	at org.opensearch.index.translog.RemoteFsTranslog.deleteStaleRemotePrimaryTerms(RemoteFsTranslog.java:430)
	at org.opensearch.index.translog.RemoteFsTranslog.trimUnreferencedReaders(RemoteFsTranslog.java:400)
	at org.opensearch.index.translog.InternalTranslogManager.trimUnreferencedReaders(InternalTranslogManager.java:394)
	at org.opensearch.index.engine.InternalEngine.revisitIndexDeletionPolicyOnTranslogSynced(InternalEngine.java:537)
	at org.opensearch.index.engine.InternalEngine$1.onAfterTranslogSync(InternalEngine.java:265)
	at org.opensearch.index.translog.listener.CompositeTranslogEventListener.onAfterTranslogSync(CompositeTranslogEventListener.java:49)
	at org.opensearch.index.translog.InternalTranslogManager.syncTranslog(InternalTranslogManager.java:193)
	at org.opensearch.index.shard.IndexShard.sync(IndexShard.java:4154)
	at org.opensearch.index.IndexService.maybeFSyncTranslogs(IndexService.java:980)
	at org.opensearch.index.IndexService$AsyncTranslogFSync.runInternal(IndexService.java:1104)
	at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1623)

mch2 · 2023-08-10T22:25:21Z

./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.RemoteStoreForceMergeIT.testRestoreForceMergeSingleIteration" -Dtests.seed=90BCF02533C9A76B -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=hi-IN -Dtests.timezone=Europe/Amsterdam

Not able to repro this locally - @ankitkala @gbbafna wondering if you have context here?

github-actions · 2023-08-10T23:06:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/22388/
CommitID: 1989a2e

opensearch-trigger-bot · 2023-08-10T23:12:37Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-9111-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c30154458a44e91a2f245b2357e69ecc839265a9
# Push it to GitHub
git push --set-upstream origin backport/backport-9111-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9111-to-2.x.

…with remote store. (opensearch-project#9111) * Remove divergent commit logic with segment replication. This change removes divergent commit paths for segrep node-node and remote store. All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file. This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened. This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more NRTReplicationEngineTests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Ensure old commit files are wiped on remote store sync before we commit a new segmentInfos. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more shard level tests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add test ensuring commits are cleaned up on replicas. Signed-off-by: Marc Handalian <handalm@amazon.com> * Self review. Signed-off-by: Marc Handalian <handalm@amazon.com> * Use refresh level sync before recovery Signed-off-by: Marc Handalian <handalm@amazon.com> * PR feedback. Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com>

…with remote store. (opensearch-project#9111) * Remove divergent commit logic with segment replication. This change removes divergent commit paths for segrep node-node and remote store. All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file. This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened. This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more NRTReplicationEngineTests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Ensure old commit files are wiped on remote store sync before we commit a new segmentInfos. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more shard level tests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add test ensuring commits are cleaned up on replicas. Signed-off-by: Marc Handalian <handalm@amazon.com> * Self review. Signed-off-by: Marc Handalian <handalm@amazon.com> * Use refresh level sync before recovery Signed-off-by: Marc Handalian <handalm@amazon.com> * PR feedback. Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com> (cherry picked from commit c301544)

…fix PIT/Scroll with remote store. (#9272) * [Segment Replication] Refactor file cleanup logic and fix PIT/Scroll with remote store. (#9111) * Remove divergent commit logic with segment replication. This change removes divergent commit paths for segrep node-node and remote store. All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file. This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened. This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more NRTReplicationEngineTests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Ensure old commit files are wiped on remote store sync before we commit a new segmentInfos. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more shard level tests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add test ensuring commits are cleaned up on replicas. Signed-off-by: Marc Handalian <handalm@amazon.com> * Self review. Signed-off-by: Marc Handalian <handalm@amazon.com> * Use refresh level sync before recovery Signed-off-by: Marc Handalian <handalm@amazon.com> * PR feedback. Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com> (cherry picked from commit c301544) * Fix test SegmentReplicationIndexShardTests.testPrimaryRestart. This test is specific to remote store and should not be run for node-node replication. Signed-off-by: Marc Handalian <handalm@amazon.com> (cherry picked from commit a33f67e) Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com>

gbbafna · 2023-08-14T05:44:45Z

@mch2 : I saw this failure in relocation tests failures as well . Will create an issue and take a look .

…with remote store. (opensearch-project#9111) * Remove divergent commit logic with segment replication. This change removes divergent commit paths for segrep node-node and remote store. All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file. This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened. This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more NRTReplicationEngineTests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Ensure old commit files are wiped on remote store sync before we commit a new segmentInfos. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more shard level tests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add test ensuring commits are cleaned up on replicas. Signed-off-by: Marc Handalian <handalm@amazon.com> * Self review. Signed-off-by: Marc Handalian <handalm@amazon.com> * Use refresh level sync before recovery Signed-off-by: Marc Handalian <handalm@amazon.com> * PR feedback. Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com>

…with remote store. (opensearch-project#9111) * Remove divergent commit logic with segment replication. This change removes divergent commit paths for segrep node-node and remote store. All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file. This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened. This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more NRTReplicationEngineTests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Ensure old commit files are wiped on remote store sync before we commit a new segmentInfos. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more shard level tests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add test ensuring commits are cleaned up on replicas. Signed-off-by: Marc Handalian <handalm@amazon.com> * Self review. Signed-off-by: Marc Handalian <handalm@amazon.com> * Use refresh level sync before recovery Signed-off-by: Marc Handalian <handalm@amazon.com> * PR feedback. Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

…with remote store. (opensearch-project#9111) * Remove divergent commit logic with segment replication. This change removes divergent commit paths for segrep node-node and remote store. All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file. This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened. This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more NRTReplicationEngineTests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Ensure old commit files are wiped on remote store sync before we commit a new segmentInfos. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more shard level tests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add test ensuring commits are cleaned up on replicas. Signed-off-by: Marc Handalian <handalm@amazon.com> * Self review. Signed-off-by: Marc Handalian <handalm@amazon.com> * Use refresh level sync before recovery Signed-off-by: Marc Handalian <handalm@amazon.com> * PR feedback. Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…with remote store. (opensearch-project#9111) * Remove divergent commit logic with segment replication. This change removes divergent commit paths for segrep node-node and remote store. All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file. This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened. This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more NRTReplicationEngineTests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Ensure old commit files are wiped on remote store sync before we commit a new segmentInfos. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more shard level tests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add test ensuring commits are cleaned up on replicas. Signed-off-by: Marc Handalian <handalm@amazon.com> * Self review. Signed-off-by: Marc Handalian <handalm@amazon.com> * Use refresh level sync before recovery Signed-off-by: Marc Handalian <handalm@amazon.com> * PR feedback. Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

tlfeng added :test Adding or fixing a test skip-changelog labels Aug 4, 2023

mch2 force-pushed the cleanupdeletes branch from 334ab1d to 986f55a Compare August 8, 2023 07:42

mch2 mentioned this pull request Aug 8, 2023

[BUG] Segment_N file cleanup not happening on replicas for indexes with segrep using remote store replication #8777

Closed

mch2 force-pushed the cleanupdeletes branch from 986f55a to de713f2 Compare August 9, 2023 05:53

mch2 force-pushed the cleanupdeletes branch from 12878c5 to 2e43482 Compare August 9, 2023 06:22

mch2 force-pushed the cleanupdeletes branch from 2e43482 to 4494c3a Compare August 9, 2023 06:42

This comment was marked as outdated.

Sign in to view

mch2 marked this pull request as ready for review August 9, 2023 17:14

mch2 requested review from Bukhtawar, CEHENKLE, anasalkouz, andrross, dblock and reta as code owners August 9, 2023 17:14

mch2 added 8 commits August 10, 2023 14:07

Add more NRTReplicationEngineTests.

413bb24

Signed-off-by: Marc Handalian <handalm@amazon.com>

Ensure old commit files are wiped on remote store sync before we comm…

fc1d756

…it a new segmentInfos. Signed-off-by: Marc Handalian <handalm@amazon.com>

Add more shard level tests.

a8106ca

Signed-off-by: Marc Handalian <handalm@amazon.com>

Add test ensuring commits are cleaned up on replicas.

7362c20

Signed-off-by: Marc Handalian <handalm@amazon.com>

Self review.

b555a26

Signed-off-by: Marc Handalian <handalm@amazon.com>

Use refresh level sync before recovery

078b35b

Signed-off-by: Marc Handalian <handalm@amazon.com>

PR feedback.

1989a2e

Signed-off-by: Marc Handalian <handalm@amazon.com>

mch2 force-pushed the cleanupdeletes branch from 488404b to 1989a2e Compare August 10, 2023 21:07

dreamer-89 approved these changes Aug 10, 2023

View reviewed changes

mch2 merged commit c301544 into opensearch-project:main Aug 10, 2023

mch2 added the backport 2.x Backport to 2.x branch label Aug 10, 2023

mch2 deleted the cleanupdeletes branch August 10, 2023 23:11

mch2 mentioned this pull request Aug 11, 2023

[Backport 2.x] [Segment Replication] Refactor file cleanup logic and fix PIT/Scroll with remote store. #9272

Merged

mch2 mentioned this pull request Aug 21, 2023

[Segment Replication + Remote Store] Remove divergent logic between node-node and remote store implementations #9016

Closed

Conversation

mch2 commented Aug 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

opensearch-trigger-bot bot commented Aug 4, 2023

Uh oh!

github-actions bot commented Aug 4, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Aug 4, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Aug 7, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

tlfeng commented Aug 7, 2023

Uh oh!

mch2 commented Aug 7, 2023

Uh oh!

github-actions bot commented Aug 8, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

opensearch-trigger-bot bot commented Aug 8, 2023

Uh oh!

opensearch-trigger-bot bot commented Aug 9, 2023

Uh oh!

github-actions bot commented Aug 9, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Aug 9, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Aug 9, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

opensearch-trigger-bot bot commented Aug 9, 2023

Uh oh!

opensearch-trigger-bot bot commented Aug 9, 2023

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

opensearch-trigger-bot bot commented Aug 10, 2023

Uh oh!

github-actions bot commented Aug 10, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

dreamer-89 commented Aug 10, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

mch2 commented Aug 10, 2023

Uh oh!

github-actions bot commented Aug 10, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

opensearch-trigger-bot bot commented Aug 10, 2023

Uh oh!

gbbafna commented Aug 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mch2 commented Aug 4, 2023 •

edited

Loading