[META] Reader/Writer Separation

[mch2]

Please describe the end goal of this project

High level goal is to separate indexing and search traffic within remote store enabled clusters to achieve independent scalability and failure/workload isolation.

Supporting References

#15237,
#7258
#14596

Issues

Experimental feature goals - Achieve workload & failure isolation with read scale to zero.

introduce replica type changes, pull based replica, allocation filter and minimal API changes (stats).

• [RW Separation] Introduce new search replica concept and changes to create API #15368
• [RW Separation] Add Allocation filter based on shard type.  #15445
• Use pull based model for Segment Replication #4577
• Add a routing preference to route searches to search replicas #15562

-- Experimental (target 2.17)

GA feature goals: Achieve write scale to zero, ISM/Clients integration, and reasonable (cat/health) API updates.

• [RW Separation] Change search replica recovery flow #15952
• [RW Separation] Updates to restore from snapshot with added search replicas #15532
• Update Segment replication stats APIs to support pull based architecture #15534
• [RW Separation] Disable auto-expand-replica case with search replicas #16912
• Add info on documentation website
• Implement mechanism to fail stale search replicas.
• [Feature Request] Redefine the computation of segment replication metrics in Node Stats #16801
• Update API specs & clients. [FEATURE] API Specification updates for OpenSearch Reader/Writer Separation opensearch-api-specification#884
• [RW Separation] Honor allocation awareness attributes #16987

Follow ups -

• Updates to cluster health API: [RW Separation] [Feature Request] Scale to Zero (Indexing Shards) with Reader/Writer Separation. #16720
• scale to zero writer changes: [RW Separation] [Feature Request] Scale to Zero (Indexing Shards) with Reader/Writer Separation. #16720
• ISM integration to auto scale down writers on index migration

Related component

Search:Performance

[prudhvigodithi]

For scale to zero implementation.

To be able to scale down the primary and replicas and keep only the search replicas for search traffic and ability to bring back the primary and regular replicas for write (index) traffic.

Update Cluster State (Initial)

• Set scale_to_zero flag in index settings.
• Update IndexMetadata to reflect scaled state.
• Trigger cluster state update.

Store Original Configuration

• Save number of original shards and replicas.
• Store original routing table configuration.
• Preserve all allocation details.
• This information will be used during scale-up.

Prepare for Scale Down

• Flush all shards to ensure data persistence.
• Upload latest state to remote store.
• Perform final sync for search replicas.
• Ensure all data is safely stored remotely.

Update Routing Table

• Mark shards as SCALED_TO_ZERO (new status).
• Remove primary and replica assignments.
• Update routing table to reflect scaled state.
• Maintain cluster state consistency.

Remove Shards

• Remove primary shard.
• Remove replica shards.
• Clean up local resources.
• Maintain metadata in cluster state.

Handle Cluster Health

• Implement custom health status for scaled-down indices.
• Update cluster health APIs.
• Provide clear status in cluster state.

Scale Up Process (when flag is removed)

• Detect scale_to_zero flag removal.
• Retrieve original configuration.
• Recover from remote store.
• Restore original routing.
• Re-establish primary and replicas.
• Clear scale_to_zero metadata.

Thanks to @mch2 for the initial setup! Based on the above points, please share any additional thoughts and let me know if required any changes to the implementation order.

Thanks
@getsaurabh02 @vinaykpud