[BUG] org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication is flaky

[psychbot]

Describe the bug
org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication is flaky

https://build.ci.opensearch.org/job/gradle-check/17543/testReport/junit/org.opensearch.remotestore/SegmentReplicationUsingRemoteStoreIT/testDropPrimaryDuringReplication/

#8057 (comment)

Assertion Failure

java.lang.AssertionError: Expected search hits on node: node_t3 to be at least 111 but was: 110
	at __randomizedtesting.SeedInfo.seed([5E122859418B05B6:F3568B11F61BBC1C]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:122)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1084)
	at org.opensearch.indices.replication.SegmentReplicationBaseIT.waitForSearchableDocs(SegmentReplicationBaseIT.java:117)
	at org.opensearch.indices.replication.SegmentReplicationBaseIT.waitForSearchableDocs(SegmentReplicationBaseIT.java:112)
	at org.opensearch.indices.replication.SegmentReplicationIT.testDropPrimaryDuringReplication(SegmentReplicationIT.java:739)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)

To Reproduce

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication" -Dtests.seed=5E122859418B05B6 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-CO -Dtests.timezone=Asia/Bishkek -Druntime.java=20

[psychbot]

@ankitkala
Tagging IT author to take an initial look.

[ankitkala]

I triggered more than 100 runs locally for this test and i was not able to reproduce this again. Let's revisit this again if we see another such occurrence.

I also went through the gradle check logs shared above couldn't find anything useful.

[ankitkala]

• https://build.ci.opensearch.org/job/gradle-check/17793/
• https://build.ci.opensearch.org/job/gradle-check/17803/
• https://build.ci.opensearch.org/job/gradle-check/17927/
• https://build.ci.opensearch.org/job/gradle-check/18090/