[FEATURE] Documentation confirming version compatibility of cross cluster replication during leader/follower versions

[arjunrajnetapp]

Is your feature request related to a problem?
A clear and concise description of what the problem is, e.g. I'm always frustrated when [...]

I could not find any documentation under https://docs.opensearch.org/latest/tuning-your-cluster/replication-plugin/index/ or https://docs.opensearch.org/latest/migrate-or-upgrade/rolling-upgrade/ about version compatibility for cross cluster replication between Leaders and Followers. I can see from https://docs.aws.amazon.com/opensearch-service/latest/developerguide/replication.html there is some mention of upgrading Follower clusters first, but based on my observation -

1. Creating new replications returns a validation error when the Leader cluster is higher than the Follower, for example - Leader index[leader_index2] is on higher version [3.2.0] than follower [2.19.3]
2. Existing replication work fine regardless of the order of upgrading the Leader/Follower clusters. I even tried upgrading the Leader cluster first to 3.2.0 and the replication continued to work without a hitch. So that point on the AWS docs don't seem to be entirely accurate based on how CCR currently works. I assume the reason this works is because the lucene version for the leader and follower indices are the same.

What solution would you like?
A clear and concise description of what you want to happen.

It would be good to get confirmation that upgrading Leader/Follower clusters, regardless of the order, does not interrupt existing replications by design.

It would also be good to document the expected compatibility and reliability information on https://docs.opensearch.org/latest/tuning-your-cluster/replication-plugin/index/ and/or https://docs.opensearch.org/latest/migrate-or-upgrade/rolling-upgrade/ as it would provide essential information for those using cross cluster replication on how to plan their upgrades.

What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.

None.

Do you have any additional context?
Add any other context or screenshots about the feature request here.

None.

[andrross]

Catch All Triage - 1 2

@ankitkala Can a CCR maintainer follow up here?

[ankitkala]

I think we need to update the documentation with the version compatibility.

As called out in aws documentation, follower cluster should be upgraded first. @arjunrajnetapp Its likely that the existing replication might will continue to work even if leader is upgraded first, specially when there is no breaking change (e.g. major version bump for Lucene). But the general guidance is to upgrade follower first, this ensure that replication flows are always compatible.

[arjunrajnetapp]

Thanks @ankitkala , it would be good to document the expected behaviour when a leader is indeed upgraded first.

The reason being that we have use cases where CCR is used in both directions between a pair of clusters. Based on what we observed - the impact of such an architecture is that new replications can't be created until both clusters are on the same version, but existing replications remain unaffected.

It would be helpful to document this behaviour down for both informational purposes for others who follow this sort of 2 way replication architecture, and as acknowledgement that it is the intended behaviour.

[arjunrajnetapp]

@andrross @ankitkala @krisfreedain any updates on whether the suggestion I've provided above makes sense?

[farman-76]

I am exploring the possibility of using CCR for upgrading from one OpenSearch version to another. In this regard it will be very beneficial to have the compatibility matrix between different OpenSearch versions where CCR is possible.
I have also read that CCR will work if the source and target clusters are in the same major version or if target cluster is in the immediate next version as compared to the source cluster (e.g. source in 2.X and target in 3.X). Is it correct?

[ankitkala]

@farman-76 that is correct. data written with newer major version codec is not guaranteed to be readable by older version.

---

the impact of such an architecture is that new replications can't be created until both clusters are on the same version, but existing replications remain unaffected.

@arjunrajnetapp feel free to raise PR for the documentation update. The only point i'd like to call out is that, though the replication might work as usual, its not guaranteed that existing replication will remain unaffected. Specially if there is a breaking change from lucene where same operation applied on a newer version leader might not be replayable on follower cluster. That's why the general guidance is to upgrade follower first. This gets tricky with bidirectional cases, so safe recommendation is to stop replication in one direction and the upgrade both clusters.