[META] Cluster Manager Async Shard Fetch Revamp

[amkhar]

Describe the revamp

Original Issue opened/ Project goal : #5098

RFC/Proposal - #5098 (comment)
Please read above issue in detail before starting any discussion, thanks.

Impact of the change

We'll track following as subtasks of this goal. I'll keep adding new issues, if we find more ways to improve this behaviour.

• [Enhancement] Add support for async fetching of metadata for a batch of shards #8956
• [Enhancement] Create new transport action for fetching metadata of multiple primary & replica shards #8957
• [Enhancement] Enhance GatewayAllocator to handle a batch of shards fetching #8958
• [Enhancement] Enhance AllocationService to allocate unassigned shards in batches #8959
• [Enhancement] Add a batch allocator for building allocation decisions for multiple primary & replica shards #8960
• [Bug] Fix cluster health API response  #9020
• [Integ] Add integration test for Async Shard Fetch revamp #9021
• [Deprecate] Remove transport actions TransportNodesListGatewayStartedShards & TransportNodesListShardStoreMetadata #9022
• [Enhancement] Update TransportIndicesShardStoresAction to fetch the shard metadata using batch call #9023
• [Optimization] AsyncShardFetch response contains DiscoveryNode which takes GBs of memory #7861
• [Optimization] Reduce ShardId key size in AsyncShardFetch Revamp strategy #12010
• [Refactoring] Abstract AsyncShardFetch cache to allow restructuring for batch mode #12440
• [Restructure] Change Cache Structure in AsyncShardFetch class #12248

To Reproduce
Steps given in Draft PR #7269

Expected behavior
Cluster manager should be resilient to node restarts in high number.