[BUG][Segment Replication] Stuck Initializing Shard w/ Segment Replication enabled

[mattweber]

Describe the bug
I have recently enabled segment replication on an OpenSearch 2.5.0 cluster and ran into a stuck initializing shard somehow. The only thing different on this cluster is 2.5.0 upgrade + enabling segment replication. I am leaning toward being related to the segment replication since it is experimental.

To Reproduce
I have not been able to reproduce, not sure how it even happened in first place.

Expected behavior
Shards initialize and finish

Additional context
I didn't see anything suspicious in the logs and issue was easy to resolve using the shard reroute api.

[reta]

@mattweber could it be related to this bug [1]? do you have any exceptions in the log to pinpoint the issue?

[1] #5701

[dreamer-89]

Thank you @mattweber for reporting this. I have also seen this error once while working on #5898 where replica shard remained INITIALIZING and never caught up with primary. Performing_refresh and _flush also did not change the state. I did not cut issue as I couldn't repro the issue. One thing to note, this happened on an actual cluster, with 3 data node, 1 master node; with 1 shard, 1 repilca as index settings; for only index I created.

[mattweber]

@reta maybe, I will go dig though the logs again and see what I find. Its a little rough as I have 48 nodes and not shipping logs in this environment.

@dreamer-89 yes I debated even opening this as well and decided just in case someone else might have hit it and have more info. Looks like that was a good call. You can resolve it using the cluster reroute api w/ cancel + allocate_replica commands.

My particular index was 24 shards, 1 replica and only a single replica ran into this issue.

[dreamer-89]

Looking into it.

[dreamer-89]

Not able to reproduce this on main or latest 2.x using following.

1. Integ test. Created integration test to mimic the setup with large number of primaries and node restarts. Code link
2. Manual experiment. Perform node stops during data ingestion. Used nyc_taxis workload from OpenSearch--Benchmark to perform data ingestion. Using 3 cluster-manager nodes, 3 data nodes setup which was used previously.

Index settings

{
    "index_settings":{
        "number_of_replicas": 1,
        "number_of_shards": 24,
        "index":{
           "replication": {"type": "SEGMENT"}
        }
    }
}