[BUG] Evaulate and update all generation fields in SegmentCommitInfo during primary promotion

[dreamer-89]

Coming from #4365 (comment), we need to relook into different SegmentCommitInfo generations and update if needed. As part of this bug, identify use of different generation fields in SegmentCommitInfo and possibly bump them as well.

[Poojita-Raj]

Looking into it

[mhoffm-aiven]

Hey, im not sure if its related but i have been running an experiment using segment replication using an opensearch 2.4.1 cluster with 9 nodes:

OpenSearchException[Segment Replication failed]; nested: IllegalStateException[Shard [opensearch_segment_replication_test_index-2023-01-26][40] has local copies of segments that differ from the primary]

I started facing it after adding 6 nodes to the cluster ( to have 15 in total ) and restarting the opensearch processes on all nodes to add some static configuration. Since then the cluster has been performing relocations and failing due to this error i suppose. All of this was performed under production load.

[dreamer-89]

Hey, im not sure if its related but i have been running an experiment using segment replication using an opensearch 2.4.1 cluster with 9 nodes:

OpenSearchException[Segment Replication failed]; nested: IllegalStateException[Shard [opensearch_segment_replication_test_index-2023-01-26][40] has local copies of segments that differ from the primary]

I started facing it after adding 6 nodes to the cluster ( to have 15 in total ) and restarting the opensearch processes on all nodes to add some static configuration. Since then the cluster has been performing relocations and failing due to this error i suppose. All of this was performed under production load.

Thank you @mhoffm-aiven for sharing this issue. This is really important to make the feature more robust when tested under production workload. So, thank you again!

In order to repro this issue, can you please share the cluster configuration, index settings and type of operations (index/update/delete), workload size.

[mhoffm-aiven]

Hey @dreamer-89,

sorry i already deleted the cluster, from memory though, cluster configuration that was different from our default was

cluster.routing.allocation.cluster_concurrent_rebalance = 16

the indexes were created using this template

PUT /_index_templates/segment_replication_test
{
  "index_patterns": [
    "opensearch_segment_replication_test_index*"
  ],
  "template": {
    "settings": {
      "replication.type": "SEGMENT"
    }
  }
}

using these settings

    "settings": {
        "index": {
            "refresh_interval": "5s",
            "sort.field": "timestamp",
            "sort.order": "asc",
        },
        "number_of_replicas": 1,
        "number_of_shards": 52,
     },

I noticed that the service was relocating shards constantly after adding 6 nodes and restarting the cluster by accident ( i mistakenly configured cluster.routing.allocation.cluster_concurrent_rebalance with static config ). Checking after a while (2 days) why its not done relocating i saw that this bug happened pretty close to messing with the cluster by adding nodes and restarting.

Workload was _bulk with only index operations, and occasionally deletion of old indexes.