[BUG] cluster.routing.PrimaryAllocationIT.testPrimaryReplicaResyncFailed intermittently fails

[dblock]

Describe the bug

Tests with failures:
 - org.opensearch.cluster.routing.PrimaryAllocationIT.testPrimaryReplicaResyncFailed

> Task :server:internalClusterTest

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.PrimaryAllocationIT.testPrimaryReplicaResyncFailed" -Dtests.seed=6D4D4C998D58C02E -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m -Djava.security.manager=allow" -Dtests.locale=es-CR -Dtests.timezone=Africa/Johannesburg -Druntime.java=17

org.opensearch.cluster.routing.PrimaryAllocationIT > testPrimaryReplicaResyncFailed FAILED
    java.lang.AssertionError: 
    Expected: a collection with size <4>
         but: collection size was <3>
        at __randomizedtesting.SeedInfo.seed([6D4D4C998D58C02E:A2886BE2A57855BE]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.opensearch.cluster.routing.PrimaryAllocationIT.lambda$testPrimaryReplicaResyncFailed$14(PrimaryAllocationIT.java:727)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1048)
        at org.opensearch.cluster.routing.PrimaryAllocationIT.testPrimaryReplicaResyncFailed(PrimaryAllocationIT.java:724)

#1721

https://ci.opensearch.org/logs/ci/workflow/OpenSearch_CI/PR_Checks/Gradle_Check/gradle_check_1478.log

[kotwanikunal]

Taking a look at this issue.

[anasalkouz]

@kotwanikunal are you actively working on this issue?

[kotwanikunal]

@kotwanikunal are you actively working on this issue?

This is currently on the backlog and I am not actively looking into it.

[adnapibar]

This seems out of data as it hasn't been reported again in the CI. Not reproducible locally.