Clear Stale Persistent Tasks in Stop/Pause API

[mohit10011999]

Description

When CCR is stopped or paused, all the index and shard replication tasks should be stopped. But if the stop/ pause is not completely successful, some of the replication tasks might stay running. This can cause conflict when we restart/resume the replication.

1. StaleTaskUtils provides shared detection/removal of unassigned replication tasks by index name. Both Start and Resume APIs call StaleTaskUtils.removeStaleTasksForIndex before creating new tasks. Detection uses precise regex matching for both replication:index:{name} and replication:[{name}][{shard}] formats. ResourceNotFoundException is handled gracefully for idempotency.
2. The stop action wraps every step in try/catch so failures are non-fatal and logged. The flow: remove block → close index → retention leases → cluster state update → reopen → remove stale tasks → delete metadata. All operations are idempotent — calling stop multiple times or on a non-replicated index succeeds.
3. validateNoActiveMetadata in the Start action checks cluster state for existing replication metadata. Returns actionable errors for RUNNING/PAUSED states ("use resume API") and STOPPED/FAILED states ("run Stop API to clean up"). Fires after the existing hasIndex check to preserve the original error message ordering.
4. Idempotent stop and stale task removal are implemented. Two integration tests updated to reflect idempotent stop behavior. Concurrency safety relies on OpenSearch's built-in cluster state atomic updates.
5. The SecurityCustomRolesIT.test for FOLLOWER that STOP replication works for user with valid permissions test was expecting stopReplication("follower-index1") to throw a ResponseException with "No replication in progress", but the idempotent stop changes made the stop API succeed silently when there's no active replication. Updated the test to simply call stopReplication and expect success, matching the same pattern already applied in StopReplicationIT. Also fixed the same issue in SingleClusterSanityIT

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ankitkala]

Two major feedback:

• The PR has lot of additional code changes which doesn't seems to be related to the actual change. Can you remove all the unnecessary changes so its easier to review
• Stale replication tasks are problem when you're trying to create the task again(start or resume). I think we should be able to simplify by just handling this during task creation here (need to verify though, with any stacktrace from last few occurence of the issue)

[soosinha]

One of the issues currently is not assigned tasks are not getting cleared. We need to clear the assigned tasks as well.