DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock is flaky

[andrross]

See https://build.ci.opensearch.org/job/gradle-check/9659/

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock" -Dtests.seed=47FA74E827DEE212 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=th-TH -Dtests.timezone=Africa/Djibouti -Druntime.java=19

org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT > testIndexCreateBlockWithAReadOnlyBlock FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([47FA74E827DEE212:57BCB87090067887]:0)
        at org.junit.Assert.fail(Assert.java:87)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertTrue(Assert.java:53)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.lambda$testIndexCreateBlockWithAReadOnlyBlock$11(DiskThresholdDeciderIT.java:297)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1049)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock(DiskThresholdDeciderIT.java:295)

This was introduced in #5852. Maybe this is what #5952 is trying to fix? @RS146BIJAY

[RS146BIJAY]

Yes. It seems like the test cases are flaky even now. I am trying to fix those flaky test cases as a part of #5952 .

[sachinpkale]

Build failure due to these tests - https://build.ci.opensearch.org/job/gradle-check/9749/

[andrross]

Seems like this isn't completely fixed: https://build.ci.opensearch.org/job/gradle-check/10405/

/cc @RS146BIJAY

[RS146BIJAY]

Unable to reproduce this failure on local. Validating what else may be causing race condition scenario which is causing test cases to fail.