[BUG] org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testRequestStats is flaky

[shiv0408]

Describe the bug
The org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testRequestStats is flaky

java.lang.AssertionError: Only index blobs should remain in repository but found [/container/tests-1Gw9amxARjeifC0te8CG_A/master.dat]
    Expected: a collection with size <0>
         but: collection size was <1>
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.opensearch.repositories.blobstore.OpenSearchMockAPIBasedRepositoryIntegTestCase.tearDownHttpServer(OpenSearchMockAPIBasedRepositoryIntegTestCase.java:145)
        at java.****/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.****/java.lang.reflect.Method.invoke(Method.java:578)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)

To Reproduce

./gradlew ':plugins:repository-azure:internalClusterTest' --tests "org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testRequestStats" -Dtests.seed=D1E7629061A3A0D2 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-OM -Dtests.timezone=Europe/Guernsey -Druntime.java=20

Expected behavior
Test must always pass

Plugins
Standard

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
https://build.ci.opensearch.org/job/gradle-check/24788/testReport/org.opensearch.repositories.azure/AzureBlobStoreRepositoryTests/testRequestStats/

[saratvemulapalli]

This was closed with an unrelated PR.
Anyway I spent sometime on this test, I tried running this over 6k times and it passes.

BUILD SUCCESSFUL in 35s
65 actionable tasks: 6 executed, 59 up-to-date
Run #No (7269)

testRequestStats is run by multiple repositories implementations: https://github.com/opensearch-project/OpenSearch/blob/main/test/framework/src/main/java/org/opensearch/repositories/blobstore/OpenSearchMockAPIBasedRepositoryIntegTestCase.java#L190

S3 Repo failure: https://build.ci.opensearch.org/job/gradle-check/28369/testReport/junit/org.opensearch.repositories.s3/S3BlobStoreRepositoryTests/testRequestStats/

Azure Repo Failure:

1. https://build.ci.opensearch.org/job/gradle-check/27280/consoleFull

org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests > testRequestStats FAILED
    java.lang.AssertionError: SDK sent [{PutBlock=4, GetBlobProperties=3, PutBlockList=4, GetBlob=34, PutBlob=35, ListBlobs=6}] calls and handler measured [{PutBlob=35, ListBlobs=7, GetBlob=34, GetBlobProperties=3, PutBlockList=4, PutBlock=4}] calls expected:<{PutBlob=35, ListBlobs=7, GetBlob=34, GetBlobProperties=3, PutBlockList=4, PutBlock=4}> but was:<{PutBlock=4, GetBlobProperties=3, PutBlockList=4, GetBlob=34, PutBlob=35, ListBlobs=6}>

2. https://build.ci.opensearch.org/job/gradle-check/24788/testReport/org.opensearch.repositories.azure/AzureBlobStoreRepositoryTests/testRequestStats/

RepositoryVerificationException[[y] path  is not accessible on cluster-manager node]; nested: FileAlreadyExistsException[tests-1Gw9amxARjeifC0te8CG_A/master.dat: Status code 409, "<?xml version="1.0" encoding="UTF-8"?><Error><Code>BlobAlreadyExists</Code><Message>CONFLICT</Message></Error>"];
	at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1856)
	at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:356)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1623)
Caused by: java.nio.file.FileAlreadyExistsException: tests-1Gw9amxARjeifC0te8CG_A/master.dat: Status code 409, "<?xml version="1.0" encoding="UTF-8"?><Error><Code>BlobAlreadyExists</Code><Message>CONFLICT</Message></Error>"
	at org.opensearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:379)
	at org.opensearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:133)
	at org.opensearch.repositories.azure.AzureBlobContainer.writeBlobAtomic(AzureBlobContainer.java:141)
	at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1848)
	... 6 more

3. https://build.ci.opensearch.org/job/gradle-check/31603/consoleFull

org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests > testRequestStats FAILED
    java.lang.AssertionError: SDK sent [{PutBlock=4, PutBlob=37, ListBlobs=7, GetBlob=35, GetBlobProperties=3, PutBlockList=4}] calls and handler measured [{PutBlock=4, ListBlobs=7, PutBlob=38, GetBlob=35, GetBlobProperties=3, PutBlockList=4}] calls expected:<{PutBlock=4, ListBlobs=7, PutBlob=38, GetBlob=35, GetBlobProperties=3, PutBlockList=4}> but was:<{PutBlock=4, PutBlob=37, ListBlobs=7, GetBlob=35, GetBlobProperties=3, PutBlockList=4}>

[saratvemulapalli]

Opened a PR #12079 to add additional logging incase we see the failure again.
Feel free to re-open again.

[shwetathareja]

Failed again - #12773 (comment)