[Remote translog] Add integration tests for primary term validation

[ashking94]

Description

Add integration test for primary term validation for remote translog enabled indexes.

Related Issues

Resolves #5636

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication
      1 org.opensearch.indices.replication.SegmentReplicationIT.testCancellation

• URL: https://build.ci.opensearch.org/job/gradle-check/18953/
• CommitID: 5086c7a
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication

• URL: https://build.ci.opensearch.org/job/gradle-check/18956/
• CommitID: 2849b18
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8406 (276098d) into main (240193a) will decrease coverage by 0.14%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #8406      +/-   ##
============================================
- Coverage     70.95%   70.81%   -0.14%     
+ Complexity    56994    56886     -108     
============================================
  Files          4757     4757              
  Lines        269125   269125              
  Branches      39403    39403              
============================================
- Hits         190957   190585     -372     
- Misses        62105    62471     +366     
- Partials      16063    16069       +6     

see 445 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication

• URL: https://build.ci.opensearch.org/job/gradle-check/18959/
• CommitID: 919b5fc
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication
      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocation

• URL: https://build.ci.opensearch.org/job/gradle-check/18979/
• CommitID: 276098d
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.