[BUG] org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessAfterFailureInFirstAttemptAfterSnapshotAndMetadataUpload is flaky

[reta]

Describe the bug
The org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessAfterFailureInFirstAttemptAfterSnapshotAndMetadataUpload test is flaky :

java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still 1 open files: {metadata__11__3__k7CAyYgBoomrakrqpTVg=1}
	at __randomizedtesting.SeedInfo.seed([245CC13AEFCE68E1:4A2CAE292B510978]:0)
	at org.apache.lucene.tests.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:876)
	at org.apache.lucene.store.FilterDirectory.close(FilterDirectory.java:111)
	at org.apache.lucene.store.FilterDirectory.close(FilterDirectory.java:111)
	at org.opensearch.index.store.Store$StoreDirectory.innerClose(Store.java:990)
	at org.opensearch.index.store.Store.closeInternal(Store.java:554)
	at org.opensearch.index.store.Store$1.closeInternal(Store.java:194)
	at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
	at org.opensearch.index.store.Store.decRef(Store.java:529)
	at org.opensearch.index.store.Store.close(Store.java:536)
	at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
	at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
	at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:81)

To Reproduce

./gradlew ':server:test' --tests "org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessAfterFailureInFirstAttemptAfterSnapshotAndMetadataUpload" -Dtests.seed=245CC13AEFCE68E1 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=mt-MT -Dtests.timezone=MST7MDT -Druntime.java=17

Expected behavior
Test must always pass

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• CI

Additional context
https://build.ci.opensearch.org/job/gradle-check/17846/

[sachinpkale]

@linuxpi Can you please look into this?

[linuxpi]

Sure, will take a look

[linuxpi]

Failure not reproduce-able even with 2000 iterations

./gradlew ':server:test' --tests "org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessAfterFailureInFirstAttemptAfterSnapshotAndMetadataUpload" -Dtests.seed=245CC13AEFCE68E1 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=mt-MT -Dtests.timezone=MST7MDT -Druntime.java=17 -Dtests.iters=2000

[reta]

@linuxpi just from today's CI builds [1]:

org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessAfterFailureInFirstAttemptAfterSnapshotAndMetadataUpload

java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still 1 open files: {metadata__9223372036854775788__9223372036854775804__9223372036854775805__9223372036854775804__9223370347789126145__1=1}
	at __randomizedtesting.SeedInfo.seed([42912DBBDDFA3EF5:2CE142A819655F6C]:0)
	at org.apache.lucene.tests.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:876)
	at org.apache.lucene.store.FilterDirectory.close(FilterDirectory.java:111)
	at org.apache.lucene.store.FilterDirectory.close(FilterDirectory.java:111)
	at org.opensearch.index.store.Store$StoreDirectory.innerClose(Store.java:1001)
	at org.opensearch.index.store.Store.closeInternal(Store.java:554)
	at org.opensearch.index.store.Store$1.closeInternal(Store.java:194)
	at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
	at org.opensearch.index.store.Store.decRef(Store.java:529)</pre>org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessAfterFailureInFirstAttemptAfterSnapshotAndMetadataUpload
Failing for the past 1 build (Since
[#19825](https://build.ci.opensearch.org/job/gradle-check/19825/) )
[Took 1.6 sec.](https://build.ci.opensearch.org/job/gradle-check/19825/testReport/junit/org.opensearch.index.shard/RemoteStoreRefreshListenerTests/testRefreshSuccessAfterFailureInFirstAttemptAfterSnapshotAndMetadataUpload/history)
Error Message

java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still 1 open files: {metadata__9223372036854775788__9223372036854775804__9223372036854775805__9223372036854775804__9223370347789126145__1=1}

Stacktrace

java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still 1 open files: {metadata__9223372036854775788__9223372036854775804__9223372036854775805__9223372036854775804__9223370347789126145__1=1}
	at __randomizedtesting.SeedInfo.seed([42912DBBDDFA3EF5:2CE142A819655F6C]:0)
	at org.apache.lucene.tests.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:876)
	at org.apache.lucene.store.FilterDirectory.close(FilterDirectory.java:111)
	at org.apache.lucene.store.FilterDirectory.close(FilterDirectory.java:111)
	at org.opensearch.index.store.Store$StoreDirectory.innerClose(Store.java:1001)
	at org.opensearch.index.store.Store.closeInternal(Store.java:554)
	at org.opensearch.index.store.Store$1.closeInternal(Store.java:194)
	at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
	at org.opensearch.index.store.Store.decRef(Store.java:529)

[1] https://build.ci.opensearch.org/job/gradle-check/19825/testReport/junit/org.opensearch.index.shard/RemoteStoreRefreshListenerTests/testRefreshSuccessAfterFailureInFirstAttemptAfterSnapshotAndMetadataUpload/

[sachinpkale]

Taking a look.