[BUG] org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreShallowSnapshotIndexAfterSnapshot is flaky

[reta]

Describe the bug
The test case org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreShallowSnapshotIndexAfterSnapshot is flaky:

org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreShallowSnapshotIndexAfterSnapshot

java.lang.AssertionError: max seq. no. [28] does not match [27]
	at __randomizedtesting.SeedInfo.seed([BD50E4879DC8A493]:0)
	at org.opensearch.index.engine.ReadOnlyEngine.assertMaxSeqNoEqualsToGlobalCheckpoint(ReadOnlyEngine.java:209)
	at org.opensearch.index.engine.ReadOnlyEngine.ensureMaxSeqNoEqualsToGlobalCheckpoint(ReadOnlyEngine.java:195)
	at org.opensearch.index.engine.ReadOnlyEngine.<init>(ReadOnlyEngine.java:143)
	at org.opensearch.index.engine.NoOpEngine.<init>(NoOpEngine.java:77)
	at org.opensearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2463)
	at org.opensearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2405)
	at org.opensearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:2375)
	at org.opensearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:660)
	at org.opensearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:121)
	at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344)
	at org.opensearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:119)
	at org.opensearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2748)
	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreShallowSnapshotIndexAfterSnapshot" -Dtests.seed=BD50E4879DC8A493

Expected behavior
The test must always pass

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• CI

Additional context

• https://build.ci.opensearch.org/job/gradle-check/29792/testReport/junit/org.opensearch.remotestore/RemoteRestoreSnapshotIT/testRestoreShallowSnapshotIndexAfterSnapshot/

[astute-decipher]

Taking a look