[Meta] [Zone Decommission] Decommissioning and recommissioning a zone

[gbbafna]

Is your feature request related to a problem? Please describe.

There are use-case where decommissioning a zone/rack might be beneficial

1. In a multi-zone deployment setup it might be good to support zonal deployment rather than a rolling restart per node, which might be too slow for a big cluster and might take sufficiently longer. In such a case zone serves a unit of deployment
2. Andon cord during zonal outages is another handy mechanism which enables graceful traffic shutdown in the impacted zone/rack espl when certain nodes are still operating in a degraded manner

This issue lists the high level tasks . For more info on feature proposal, please refer: #3402 .

Tasks

• [Zone Decommission] APIs to decommission and storing the metadata
• [Zone Decommission] Abdicating the cluster manager if in decommissioned zone
• [Zone Decommission] Graceful decommission
• [Zone Decommission] Removal of nodes in decommissioned zone

[imRishN]

Adding more meta tasks here -

• [Zone Decommission] Add DecommissionService and helper to execute awareness attribute decommissioning #4083
• [Zone Decommission] Mark API as stable once all meta issues completed #4368
• [Zone Decommission] API for deleting decommission #4369
• [Zone Decommission] Service layer changes for deleting decommission #4370
• [Zone Decommission] Controlling Peer Discovery when decommissioned #4371
• [Zone Decommission] Feature Flag for Decommission API #4372
• [Zone Decommission] Cluster Health local API should throw decommissioned exception for local node if decommissioned #4528
• [Zone Decommission] Gracefully handle concurrent zone decommission action #4543
• [Zone Decommission] Execute INIT update and transport voting config exclusion in single state update call  #4571
• [Zone Decommission] Decider to prevent shards moving into nodes of to-be-decommissioned zone #4618
• [Zone Decommission] Add route to get decommission status for particular attribute #4640
• [Zone Decommission] Update peerFinderInterval with backoff policy #4615
• [BUG] [Zone Decommission] GET response has redundant field #4750
• [Zone Decommission] Add validation for minimum awareness attribute values set before executing decommission #4768
• [BUG] [Zone Decommission] MetadataDiff is 0 when publishing cluster state for status update #4799
• [Zone Decommission] Fail weight update when decommission ongoing and fail decommission when attribute not weighed away #4838
• [Zone Decommission] Handle specific awareness attribute clearing during recommission  #4654
• [Zone Decommission] Recommision flow should set the weights back to the value prior to decommission value #4803
• [Zone Decommission]: Have a setting to support minimum connection allowed per node basis for decommission to proceeed #4804
• [Zone Decommission]: Skip the drain timeout if the active request count is 0 #4721
• [Zone Decommission]: Add force flag to skip the timeout for node drain during recommission #4720
• [Zone Decommission] Fix ClusterFormationFailureHelper for decommissioned nodes #5074
• [Zone Decommission] 4xx status code for DecommissioningFailedException #5075

[anasalkouz]

@imRishN @gbbafna I have update this META issue by adding couple of Zone Decommission issues