[BUG] org.opensearch.index.shard.RemoteIndexShardTests.testNRTReplicaWithRemoteStorePromotedAsPrimaryCommitRefresh is flaky

[reta]

Describe the bug
The org.opensearch.index.shard.RemoteIndexShardTests.testNRTReplicaWithRemoteStorePromotedAsPrimaryCommitRefresh is flaky, introduced in #8767:

org.opensearch.index.shard.RemoteIndexShardTests.testNRTReplicaWithRemoteStorePromotedAsPrimaryCommitRefresh

java.lang.AssertionError: expected:<4> but was:<3>
	at __randomizedtesting.SeedInfo.seed([EE7A1A0EDFCB8BA9:911967B0DCD2FCB7]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.opensearch.index.shard.RemoteIndexShardTests.testNRTReplicaWithRemoteStorePromotedAsPrimary(RemoteIndexShardTests.java:139)
	at org.opensearch.index.shard.RemoteIndexShardTests.testNRTReplicaWithRemoteStorePromotedAsPrimaryCommitRefresh(RemoteIndexShardTests.java:75)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1

To Reproduce

 ./gradlew ':server:test' --tests "org.opensearch.index.shard.RemoteIndexShardTests.testNRTReplicaWithRemoteStorePromotedAsPrimaryCommitRefresh" -Dtests.seed=EE7A1A0EDFCB8BA9 

Expected behavior
Test must always pass

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• CI

Additional context
https://build.ci.opensearch.org/job/gradle-check/23624/testReport/junit/org.opensearch.index.shard/RemoteIndexShardTests/testNRTReplicaWithRemoteStorePromotedAsPrimaryCommitRefresh/

[reta]

@dreamer-89 seems like introduced as part of your changes, could you please take a look? thank you!

[sachinpkale]

Not able to reproduce this on local even with 300 iterations. Closing.