Use pull based model for Segment Replication

[ankitkala]

Pull based model for Segment Replication
We're working on using the existing implementation of Segment replication for cross cluster replication. We've identified 2 major areas in the existing Segment Replication design which doesn't work well for cross cluster replication. Technically we can get around these by overriding those part of implementation just for CCR, but wanted to check if it is possible to maintain a single solution that works for both cases.

Main crux of the issue is that current approach relies on a 2 way communication where primary can talk to replica and the vice versa. This doesn't work well for cross cluster use cases as we establish uni-directional connections where follower polls the leader and get all the data. We did explore whether we can create bi-directional cross cluster connections but decided to go against it.

---

Why not bi-directional connection for CCR?
Pros:

• Can re-use Segment Replication without additional changes.
• Can allow follower to react to changes on leader (instead of polling)(e.g. listen to leader index events like close). But might put additional overhead on leader to communicate with followers (specially with multiple followers).

Cons:

• Breaking change from the existing connection mechanism which is currently used for cross-cluster search as well as replication.
• Bidirectional connection makes version upgrades tricky.
  • With current unidirectional connection, we recommend the users to upgrade follower first to ensure that CCR doesn't break during upgrade. With introduction of bi-directional connections, both clusters can be leader for different indices which will result in cyclic dependencies. Thus user can't upgrade the clusters without halting the replication.
• From CCR long term perspective, it doesn't add substantial value for us to justify a major overhaul of the current design. IMHO, it'd be much easier to modify the Segment Replication design unless there is a strong reason not to do so.

---

These are the 2 changes that would align the existing Segment Replication implementation with CCR. Kudos to the folks working on the original design who ensured such implementation swap can be done easily without modifying other other sub-components.

1. Use a pull mechanism for downloading the segments (instead of push)
Current implementation for Segment Replication relies on the peer recovery constructs where it pushes the segments from primary's node to replica's node using the MultiChunkTransfer by invoking write chunk on the replica's node. We can change the direction here where replica requests each chunk from leader and stores locally. This might require some additional book-keeping on the replica side but should be do-able. We're already doing something similar for CCR where we fetch the segment from leader cluster by treating it as a snapshot repository here
Also, this align with the remote store integration where replica would need to pull the segments from primary's remote store.

2. Use a polling mechanism on replica instead of listeners on primary refresh
Instead of listening to refreshes on Primary, we can model it as a periodic polling where each replica polls for the latest checkpoint and trigger the replication event. The change in itself should be simple to do but we might want to do a benchmark comparison to ensure that there is no regression.

[mch2]

@ankitkala Thanks for raising this. Imo the right idea long-term is to let users configure the behavior based on need & add support for a pull model. However, with segrep having two sources of copy in the near future (primary & remote store), is it worth adding a pull model with the primary only implementation of CCR now or should we start with CCR+segrep support only with a remote store?

[Bukhtawar]

Thanks @mch2 I think we should also think about how the interactions would look like if we just had remote store integration with primary(even outside CCR use case). Wouldn't that inherently be a pull based mechanism, where replicas poll for checkpoints and download segments diffs. So I think pull based mechanism will inherently be used even in local cluster with remote store. Now the only place we would be doing a push is in-cluster segrep with a local store.
That's something we need to analyse if there are any known implications of changing that to pull. Let me know if we've done a similar analysis to conclude one way or the other

[gbbafna]

How are we envisioning OpenSearch users to use the SegRep feature :

1. in-cluster segrep with a local store.
2. in-cluster segrep with remote store

If the firstsecond one is the way forward for most of them , we might not need to solve for problems like #4245

[ankitkala]

is it worth adding a pull model with the primary only implementation of CCR now or should we start with CCR+segrep support only with a remote store

@mch2 For CCR, SegRep with remote store is definitely the first choice for us. Whether we want to support only one combination or both needs to be decided(will bring this up during feature brief). But if we want to keep the option to support segrep without remote store open, it might be worthwhile to do these refactors early on.

For local SegRep, are there any pros of not relying on remote store? The only reason i can think of is the additional effort required to setup the repository which might affect the adoption.

[ankitkala]

How are we envisioning OpenSearch users to use the SegRep feature :

1. in-cluster segrep with a local store.
2. in-cluster segrep with remote store

If the first one is the way forward for most of them , we might not need to solve for problems like #4245

Isn't it other way round? With remote store, we won't have to solve for the network bandwidth.

[gbbafna]

@ankitkala : Thanks , corrected my comment.

For local SegRep, are there any pros of not relying on remote store? The only reason i can think of is the additional effort required to setup the repository which might affect the adoption.

Most of the users already use snapshot for durability. So setting up repository shouldn't be a hindrance IMO. Cost might be a reason not to rely on remote store . @mch2 @Bukhtawar would know better on this .

I prefer that there are few modes for OpenSearch to operate for ease of operations and maintenance.