[RW Separation] Honor allocation awareness attributes

[mch2]

Today all shards are considered equally by AwarenessAllocationDecider. With search only replicas being primary ineligible, this will leave an imbalance of write or search availability across awareness attributes. To fix this we need to apply the alloc decision to both writer eligible and search only shard groups independently, while also taking into account nodes marked specifically for search with the cluster.routing.allocation.search.replica.dedicated.include setting introduced to allocate search replicas to set of nodes.

[vinaykpud]

Hi @mch2 , Could you assign this issue to me?

[vinaykpud]

Initial commit main...vinaykpud:OpenSearch:rw/search-replica-allocation-awareness

This commit has been added solely for my initial testing. It contains many duplicates that can be optimized. The existing code is highly unreadable.

Changes looks good with the initial shard assignment, works as expected. I went ahead and tested for the zone failure use cases. I found some issues related to the allocation decision(depending on the number of replica and search replica) and shard relocation. I will add the details bellow.

[vinaykpud]

In Addition to this we(Allocator) need to consider allocating replicas and search replicas separately