Add transport action for primary term validation for remote-backed indices

[ashking94]

Signed-off-by: Ashish Singh ssashish@amazon.com

Description

Solves #5464

Issues Resolved

Solves #5464

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/8341/
• CommitID: 8743830

[codecov-commenter]

Codecov Report

Merging #5616 (5d546b2) into main (306b199) will increase coverage by 0.09%.
The diff coverage is 70.43%.

@@             Coverage Diff              @@
##               main    #5616      +/-   ##
============================================
+ Coverage     70.84%   70.93%   +0.09%     
- Complexity    58296    58339      +43     
============================================
  Files          4741     4740       -1     
  Lines        278541   278635      +94     
  Branches      40268    40273       +5     
============================================
+ Hits         197322   197654     +332     
+ Misses        65068    64839     -229     
+ Partials      16151    16142       -9     

Impacted Files	Coverage Δ
...eplication/checkpoint/PublishCheckpointAction.java	27.58% <ø> (+5.17%)	⬆️
...ation/OpenSearchIndexLevelReplicationTestCase.java	89.81% <ø> (ø)
...ensearch/action/bulk/TransportShardBulkAction.java	76.71% <63.01%> (-3.14%)	⬇️
...upport/replication/TransportReplicationAction.java	76.56% <66.66%> (+0.39%)	⬆️
...support/replication/ReplicationModeAwareProxy.java	81.25% <70.00%> (-8.75%)	⬇️
...on/support/replication/FanoutReplicationProxy.java	85.71% <83.33%> (-14.29%)	⬇️
...tion/support/replication/ReplicationOperation.java	73.29% <100.00%> (-0.33%)	⬇️
...h/action/support/replication/ReplicationProxy.java	100.00% <100.00%> (+20.00%)	⬆️
...n/support/replication/ReplicationProxyRequest.java	100.00% <100.00%> (ø)
...tion/support/replication/TransportWriteAction.java	81.34% <100.00%> (ø)
... and 487 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/8347/
• CommitID: 3e56d19

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/8454/
• CommitID: 7a6543b

[sachinpkale]

Don't we need a changelog entry for this change?

No, while working for a bigger feature, we dont require a changelog. Check this - https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#which-changes-require-a-changelog-entry

But no-op replication can be considered as a big feature, right?

[ashking94]

Don't we need a changelog entry for this change?

No, while working for a bigger feature, we dont require a changelog. Check this - https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#which-changes-require-a-changelog-entry

But no-op replication can be considered as a big feature, right?

No-op itself is not a feature, but a part of "Request level durability for remote backed indices". Refer this comment - #4954 (comment)

[Bukhtawar]

Should we need to validate if clusterService.state().getNodes().getMinNodeVersion() of all nodes is greater than this version to ensure remote translog works. Since this can break during mixed versions rolling upgrades

[ashking94]

Should we need to validate if clusterService.state().getNodes().getMinNodeVersion() of all nodes is greater than this version to ensure remote translog works. Since this can break during mixed versions rolling upgrades

Had discussed this offline with @gbbafna and @sachinpkale. Major point around this was - If there is a mixed cluster with let's say older version and 2.5, then a primary on 2.5 will send requests to replicas on older version which can fail to understand this. However, since this is an experimental feature, We can handle this when removing the experimental feature flag. This also seems to be the status quo as the segment replication feature which is also behind the experimental feature flag has PublishCheckpointAction which older version replicas would not understand when a index is created in a mixed cluster setup.

[Bukhtawar]

Should we need to validate if clusterService.state().getNodes().getMinNodeVersion() of all nodes is greater than this version to ensure remote translog works. Since this can break during mixed versions rolling upgrades

Had discussed this offline with @gbbafna and @sachinpkale. Major point around this was - If there is a mixed cluster with let's say older version and 2.5, then a primary on 2.5 will send requests to replicas on older version which can fail to understand this. However, since this is an experimental feature, We can handle this when removing the experimental feature flag. This also seems to be the status quo as the segment replication feature which is also behind the experimental feature flag has PublishCheckpointAction which older version replicas would not understand when a index is created in a mixed cluster setup.

Can you please open up a tracking issue for the same?

[ashking94]

Can you please open up a tracking issue for the same?

Created #5635 for the same.

[Bukhtawar]

Can we add tests to ensure that primary term and no replication modes are used correctly across TransportActions

[ashking94]

Can we add tests to ensure that primary term and no replication modes are used correctly across TransportActions

Added

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/8465/
• CommitID: 6c2804b
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/8467/
• CommitID: 5d546b2

[Bukhtawar]

Thanks for the UTs can we open a follow up for an Integ test for the transport calls and validating primary term failure cases during isolated writes

[ashking94]

Tracking issue - #5636