Use CloseableRetryableRefreshListener to drain ongoing after refresh tasks on relocation

[ashking94]

Description

RefreshListeners are async in nature and are triggered after segments are refreshed. Today, during relocation handoff, it is possible that the remote segments upload are happening from the older primary while the relocation has happened. Now, we are introducing a CloseableRetryableRefreshListener which will be extended by RemoteStoreRefreshListener and CheckpointRefreshListener. The CloseableRetryableRefreshListener has capabilities to be closed which guarantees that refreshes would not trigger any after refresh operations on these listeners once closed.

In summary, the PR does the following -

1. Introduces CloseableRetryableRefreshListener which has capabilities to be closed. It achieves the same by acquiring all available permits during close and leading to no further invocation of void afterRefresh(boolean didRefresh) method.
2. CloseableRetryableRefreshListener invokes the performAfterRefresh(boolean didRefresh, boolean isRetry) synchronously on the same calling thread. The performAfterRefresh method returns true if the invocation was successful and otherwise false.
3. CloseableRetryableRefreshListener provides capabilities to schedule retry if the original performAfterRefresh returns false. It would retry the same performAfterRefresh after an interval returned by the implementor of the CloseableRetryableRefreshListener abstract class.
4. CloseableRetryableRefreshListener also has constructs present internally to ensure that there are at max at a time no more than 1 retry scheduled for a future time. It also ensures that if the performAfterRefresh and retry runs do not overlap by using semaphore permits.
5. In IndexShard, the relocation method has been updated to ensure that the refresh listeners are closed before the handoff which ensures consistency of the uploaded segments data.
6. REMOTE_REFRESH threadpool has been renamed to REMOTE_REFRESH_RETRY.
7. The afterRefresh method invocation in the RemoteStoreRefreshListener is synchronous as before but also happens on the original REFRESH threadpool.

Related Issues

Resolves #8345

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20130/
• CommitID: 70ba2b2
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20132/
• CommitID: 3ee9ac3
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testStaleCommitDeletionWithoutInvokeFlush
      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

• URL: https://build.ci.opensearch.org/job/gradle-check/20141/
• CommitID: 2ee3f18
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8683 (fabb98b) into main (064f265) will increase coverage by 0.07%.
The diff coverage is 81.08%.

@@             Coverage Diff              @@
##               main    #8683      +/-   ##
============================================
+ Coverage     70.87%   70.95%   +0.07%     
- Complexity    57201    57252      +51     
============================================
  Files          4771     4772       +1     
  Lines        270312   270352      +40     
  Branches      39505    39513       +8     
============================================
+ Hits         191590   191823     +233     
+ Misses        62619    62414     -205     
- Partials      16103    16115      +12     

Impacted Files	Coverage Δ
...nsearch/index/shard/CheckpointRefreshListener.java	84.61% <50.00%> (+1.28%)	⬆️
...index/shard/CloseableRetryableRefreshListener.java	73.33% <73.33%> (ø)
...search/index/shard/RemoteStoreRefreshListener.java	82.82% <94.73%> (+1.96%)	⬆️
...in/java/org/opensearch/index/shard/IndexShard.java	70.15% <100.00%> (+0.13%)	⬆️
...ain/java/org/opensearch/threadpool/ThreadPool.java	82.52% <100.00%> (+0.57%)	⬆️

... and 443 files with indirect coverage changes

[Bukhtawar]

Where are we closing the listeners?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/20178/
• CommitID: 0fc88a3
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[ashking94]

Where are we closing the listeners?

Have added it now.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication
      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocation

• URL: https://build.ci.opensearch.org/job/gradle-check/20179/
• CommitID: 6bc8eff
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testScrollCreatedOnReplica

• URL: https://build.ci.opensearch.org/job/gradle-check/20273/
• CommitID: 22285b9
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocationWithSegRepFailure

• URL: https://build.ci.opensearch.org/job/gradle-check/20283/
• CommitID: 28dc90b
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testStaleCommitDeletionWithInvokeFlush
      1 org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush

• URL: https://build.ci.opensearch.org/job/gradle-check/20315/
• CommitID: fabb98b
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8683-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 2ba1157947c84418234386ad5671719a99f4b889
# Push it to GitHub
git push --set-upstream origin backport/backport-8683-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8683-to-2.x.