[RFC] Reproduce the corruption issue in the segment replication scenario with soft delete enabled.

[guojialiang92]

Background

In the segment copy scenario, to avoid file conflicts after primary promotion, we incremented the segment counter by 100000 when NRTReplicationEngine was closed.

In the earlier discussion #5701, it was evaluated whether to update all generation fields in SegmentCommitInfo. Since the environment where the issue occurred was deleted and it was difficult to reproduce the problem, the issue was ultimately closed.

Currently, the InternalEngine only uses soft delete, so we focus on the segment replication scenario with soft deletion enabled. I have studied the implementation of the current segment replication and believe that in this scenario, there is indeed a possibility of an OpenSearchCorruptionException being thrown.

Reproduce

To reproduce the issue, I constructed an IT test (testPrimaryStopped_ReplicaPromoted_UpdateDoc) based on the latest main branch and submitted it to branch.

The exception stack is as follows.

[2025-12-23T18:05:06,031][ERROR][o.o.i.r.SegmentReplicationTargetService] [node_t1] [shardId [test-idx-1][0]] [replication id 611] Replication failed, timing data: {INIT=0, GET_CHECKPOINT_INFO=0, REPLICATING=0}
org.opensearch.indices.replication.common.ReplicationFailedException: Store corruption during replication
	at org.opensearch.indices.replication.SegmentReplicator$2.onFailure(SegmentReplicator.java:350) [main/:?]
	at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90) [opensearch-core-3.5.0-SNAPSHOT.jar:3.5.0-SNAPSHOT]
	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:84) [opensearch-core-3.5.0-SNAPSHOT.jar:3.5.0-SNAPSHOT]
	at org.opensearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:126) [main/:?]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [main/:?]
	at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:341) [main/:?]
	at org.opensearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:120) [main/:?]
	at org.opensearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:112) [main/:?]
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1604) [?:?]
	at org.opensearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:112) [main/:?]
	at org.opensearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:160) [main/:?]
	at org.opensearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:141) [main/:?]
	at org.opensearch.action.StepListener.innerOnResponse(StepListener.java:79) [main/:?]
	at org.opensearch.core.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:58) [opensearch-core-3.5.0-SNAPSHOT.jar:3.5.0-SNAPSHOT]
	at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:70) [main/:?]
	at org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleResponse(TraceableTransportResponseHandler.java:73) [main/:?]
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1587) [main/:?]
	at org.opensearch.transport.NativeMessageHandler.doHandleResponse(NativeMessageHandler.java:468) [main/:?]
	at org.opensearch.transport.NativeMessageHandler.lambda$handleResponse$3(NativeMessageHandler.java:462) [main/:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916) [main/:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1474) [?:?]
Caused by: org.opensearch.OpenSearchCorruptionException: Shard [test-idx-1][0] has local copies of segments that differ from the primary [name [_0_2_Lucene90_0.dvm], length [160], checksum [12940pr], writtenBy [10.3.2], name [_0_2_Lucene90_0.dvd], length [91], checksum [1aenr37], writtenBy [10.3.2]]
	at org.opensearch.indices.replication.AbstractSegmentReplicationTarget.getFiles(AbstractSegmentReplicationTarget.java:251) ~[main/:?]
	at org.opensearch.indices.replication.AbstractSegmentReplicationTarget.lambda$startReplication$2(AbstractSegmentReplicationTarget.java:180) ~[main/:?]
	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-3.5.0-SNAPSHOT.jar:3.5.0-SNAPSHOT]
	... 20 more

The test process is as follows:

1. Start two data nodes.
2. Create an index contain 1 primary shard and 1 replica shard. Turn off automatic refresh.
3. Write 20 documents.
4. Execute refresh. Generate _0.si.
5. Wait for the segment replication to complete, and both the primary and replica shard contain 20 documents.
6. Mock segment replication process to ensure that segment replication between primary and replica shard cannot be completed.
7. Update the doc with id 5 and execute refresh. Primary shard generate _0_1_Lucene90_0.dvm.
8. Update the doc with id 6 and execute refresh. Primary shard generate _0_2_Lucene90_0.dvm.
9. Restart the node where the primary shard is located.
10. The replica is promoted to a new primary shard, generating _0_1_Lucene90_0.dvm through translog recovery, but the content is different from the previous file with the same name.
11. Update the doc with id 7. The new primary shard generates _0_2_Lucene90_0.dvm, but its content is different from the previous file with the same name.
12. Peer recovery fails due to OpenSearchCorruptionException when performing force segment replication.
13. After throwing OpenSearchCorruptionException, file-based recovery will be executed, and the cluster will eventually turn green.

This test reproduces the situation where OpenSearchCorruptionException occurs, but the test will pass due to the retry of recovery.

Expected behavior

In the segment replication scenario with soft delete enabled, OpenSearchCorruptionException will not occur.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[github-actions]

@guojialiang92 Please reply to this comment with the relevant component to ensure it gets properly triaged:

• Build
• Clients
• Cluster Manager
• Extensions
• Indexing:Performance
• Indexing:Replication
• Indexing
• Libraries
• Other
• Plugins
• Search:Aggregations
• Search:Performance
• Search:Query Capabilities
• Search:Query Insights
• Search:Relevance
• Search:Remote Search
• Search:Resiliency
• Search:Searchable Snapshots
• Search
• Storage:Durability
• Storage:Performance
• Storage:Remote
• Storage:Snapshots
• Storage

Reply with the component name, and we'll label this issue appropriately.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Related PRs

#19647 - Fix [BUG] ClusterMaxMergesAtOnceIT is flaky [merged]
#20150 - [Segment Replication] Avoid data loss in vanilla segment replication [merged]
#20163 - Add support for forward translog reading [merged]
#20270 - Fixed a flaky test in issue 19042 [merged]
#20284 - bug fix for Restore with Index Sort (#20073) [merged]

👤 Suggested Assignees

• laminelam
• guojialiang92
• xuxiong1
• liuguoqingfz
• vishdivs

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[guojialiang92]

Hi, @mch2 @gbbafna @linuxpi @sachinpkale @ashking94 @Bukhtawar

I'd like to hear the community's suggestions. Is this a problem worth fixing in order to enhance the robustness of segment replication?
Since Lucene does not yet support modifying generation fields in SegmentCommitInfo, we can submit a PR in the Lucene community to add this support.

[andrross]

Is this a problem worth fixing in order to enhance the robustness of segment replication?

@guojialiang92 Is this scenario specific to soft deletes? To me it looks like an intrinsic issue where anytime a replica gets promoted and has to recover some data from the translog then it is possible it will generate different segments than what the previous primary had, and then in such a scenario if the previous primary rejoins it will see mismatched files. It also looks like the system ultimately handles this situation by executing file-based recovery. What specific changes do you have in mind here?

/cc @mch2 I'd love to hear your thoughts as well.

[guojialiang92]

Thank you for your attention @andrross.

Is this scenario specific to soft deletes? To me it looks like an intrinsic issue where anytime a replica gets promoted and has to recover some data from the translog then it is possible it will generate different segments than what the previous primary had, and then in such a scenario if the previous primary rejoins it will see mismatched files. It also looks like the system ultimately handles this situation by executing file-based recovery.

Yes, but before recovery, engine fail will be triggered first.

In non-soft-delete scenarios, we avoid engine failure caused by segment number conflicts by incrementing the segment counter.

What specific changes do you have in mind here?

Support modifying generation fields in SegmentCommitInfo.