[2.19] Fix segment replication failure during rolling restart

[cuonghm2809]

Description

This PR backports the fix for issue #19234 to the 2.19 branch.

Problem:
During rolling restart with segment replication enabled, replica shards fail to re-assign with error "Rejecting stale metadata checkpoint" after 5 retries. This occurs because:

1. Replica restarts and loads its last persisted checkpoint (e.g., version 100)
2. During restart, primary relocates to another node and advances to a newer checkpoint (version 150)
3. After relocation completes, primary checkpoint might revert to an earlier version (version 120)
4. Replica tries to replicate with initial checkpoint version 100
5. Primary returns checkpoint version 120 (appears older than replica's 100)
6. The strict validation added in PR Fix segment replication bug during primary relocation #18944 incorrectly rejects this as "stale"

This is a legitimate scenario during recovery, not an actual stale checkpoint issue.

Root cause:
PR #18944 added strict checkpoint validation to prevent issue #18490, but it doesn't distinguish between:

• Normal replication (where stale checkpoints should be rejected)
• Recovery/initialization (where the replica's persisted checkpoint may legitimately appear newer)

Solution:
Only enforce strict checkpoint validation during normal replication. During recovery (when shard is INITIALIZING or RELOCATING), accept checkpoints that may appear older, as the replica is catching up to the primary's current state.

Related Issues

Resolves #19234
Related to #18490

Check List

• New functionality includes testing

Test Plan

Added two unit tests in SegmentReplicationTargetTests.java:

1. testStaleCheckpointRejected_duringNormalReplication(): Verifies that stale checkpoints are correctly rejected when the shard is active (not recovering)

2. testStaleCheckpointAccepted_duringRecovery(): Verifies that checkpoints are accepted during recovery even if they appear older than the replica's persisted checkpoint

Both tests passed:

> Task :server:test --tests "org.opensearch.indices.replication.SegmentReplicationTargetTests.testStaleCheckpoint*"

SegmentReplicationTargetTests > testStaleCheckpointRejected_duringNormalReplication PASSED
SegmentReplicationTargetTests > testStaleCheckpointAccepted_duringRecovery PASSED

BUILD SUCCESSFUL

Additional Context

This fix is critical for production deployments using segment replication during rolling restarts. Without this fix, shards fail to recover and require manual intervention.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

❌ Gradle check result for 85cb2e0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 4f5ed51: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for bd60de1: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?