[CI] o.o.gateway.RecoveryFromGatewayIT.testReuseInFileBasedPeerRecovery

[nknize]

Failed on unrelated PR #1742. Not reproducible locally. Opening to track if this continues to fail.

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.gateway.RecoveryFromGatewayIT.testReuseInFileBasedPeerRecovery" -Dtests.seed=1183E5842BAA4635 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m -Djava.security.manager=allow" -Dtests.locale=ja-JP -Dtests.timezone=Pacific/Kwajalein -Druntime.java=17

org.opensearch.gateway.RecoveryFromGatewayIT > testReuseInFileBasedPeerRecovery FAILED
    java.lang.AssertionError: shard [test][0] on node [node_t1] has pending operations:
     --> RetentionLeaseBackgroundSyncAction.Request{retentionLeases=RetentionLeases{primaryTerm=1, version=1468, leases={peer_recovery/_23_6236SQekFQ4X2S5HWQ=RetentionLease{id='peer_recovery/_23_6236SQekFQ4X2S5HWQ', retainingSequenceNumber=1333, timestamp=1639665151286, source='peer recovery'}, peer_recovery/txItgQoGQoyJSR_SvgGAIQ=RetentionLease{id='peer_recovery/txItgQoGQoyJSR_SvgGAIQ', retainingSequenceNumber=1333, timestamp=1639665151286, source='peer recovery'}}}, shardId=[test][0], timeout=1m, index='test', waitForActiveShards=0}
    	at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:248)
    	at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:3231)
    	at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1117)
    	at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:434)
    	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50)
    	at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:378)
    	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:91)
    	at org.opensearch.transport.TransportService$8.doRun(TransportService.java:944)
    	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792)
    	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50)
    	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    	at java.base/java.lang.Thread.run(Thread.java:833)
        at __randomizedtesting.SeedInfo.seed([1183E5842BAA4635:471B0010ABF29B6E]:0)
        at org.opensearch.test.InternalTestCluster.lambda$assertNoPendingIndexOperations$12(InternalTestCluster.java:1434)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1060)
        at org.opensearch.test.InternalTestCluster.assertNoPendingIndexOperations(InternalTestCluster.java:1421)
        at org.opensearch.test.InternalTestCluster.beforeIndexDeletion(InternalTestCluster.java:1349)
        at org.opensearch.test.OpenSearchIntegTestCase.beforeIndexDeletion(OpenSearchIntegTestCase.java:636)

[dreamer-89]

Another occurrence : PR 2026

Gradle log

[dblock]

#2069 (comment)