[BUG] RemoteStoreIT.testRemoteTranslogRestore flaky test failure

[andrross]

The following failed with a timeout, but is not reproducible:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.RemoteStoreIT.testRemoteTranslogRestore" -Dtests.seed=DF38037B9405A5AE -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=de-CH -Dtests.timezone=America/North_Dakota/Beulah -Druntime.java=17

org.opensearch.remotestore.RemoteStoreIT > testRemoteTranslogRestore FAILED
    java.lang.AssertionError: timed out waiting for yellow state
        at __randomizedtesting.SeedInfo.seed([DF38037B9405A5AE:C4871B4D04A1356B]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureColor(OpenSearchIntegTestCase.java:1007)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureYellowAndNoInitializingShards(OpenSearchIntegTestCase.java:960)
        at org.opensearch.remotestore.RemoteStoreIT.verifyRestoredData(RemoteStoreIT.java:143)
        at org.opensearch.remotestore.RemoteStoreIT.testRemoteTranslogRestore(RemoteStoreIT.java:181)

(#6182) https://build.ci.opensearch.org/job/gradle-check/10624/consoleFull

/cc @sachinpkale

[sachinpkale]

Thanks @andrross , I will take a look and fix.

[sachinpkale]

The changes in #6086 should fix this flakiness of the test, will check further.

[sachinpkale]

The backport of #6086 is still in flight. Once the changes are backported, this test should not fail.

[ashking94]

PR build is in progress for the backport PR. #6170 - we can merge it once the build is successful.

[ashking94]

The above PR has been merged. Closing it.

[reta]

Still happening on 2.x:

Failed
org.opensearch.remotestore.RemoteStoreIT.testRemoteTranslogRestore

java.lang.AssertionError: timed out waiting for yellow state
	at __randomizedtesting.SeedInfo.seed([7E138B60D5A3FE42:65AC935645076E87]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.opensearch.test.OpenSearchIntegTestCase.ensureColor(OpenSearchIntegTestCase.java:1007)
	at org.opensearch.test.OpenSearchIntegTestCase.ensureYellowAndNoInitializingShards(OpenSearchIntegTestCase.java:960)
	at org.opensearch.remotestore.RemoteStoreIT.verifyRestoredData(RemoteStoreIT.java:143)

[1] https://build.ci.opensearch.org/job/gradle-check/10805/