[BUG] org.opensearch.upgrades.IndexingIT.testIndexingWithSegRep flaky

[peternied]

Describe the bug

org.opensearch.upgrades.IndexingIT.testIndexingWithSegRep is flaky, jenkins logs.

Assertion is from this codeblock:

OpenSearch/qa/rolling-upgrade/src/test/java/org/opensearch/upgrades/IndexingIT.java

Lines 119 to 121 in f73c82a

	// Ensure there is result for replica copies before processing the result. This results in retry when there
	// are not enough number of rows vs failing with IndexOutOfBoundsException
	assertEquals(0, filteredList.size() % (replicaCount + 1));

Suppressed: java.lang.AssertionError: expected:<0> but was:<1>
  at org.junit.Assert.fail(Assert.java:89)
  at org.junit.Assert.failNotEquals(Assert.java:835)
  at org.junit.Assert.assertEquals(Assert.java:647)
  at org.junit.Assert.assertEquals(Assert.java:633)
  at org.opensearch.upgrades.IndexingIT.lambda$waitForSearchableDocs$0(IndexingIT.java:121)
  at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
  ... 40 more

Related component

Indexing

To Reproduce

REPRODUCE WITH: ./gradlew ':qa:rolling-upgrade:v2.13.0#upgradedClusterTest' --tests "org.opensearch.upgrades.IndexingIT.testIndexingWithSegRep" -Dtests.seed=326EEB751FF1EA22 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ru-RU -Dtests.timezone=Africa/Timbuktu -Druntime.java=21

Expected behavior

Test pass consistently

Additional Details

• Might be related to [BUG] org.opensearch.upgrades.IndexingIT.testIndexingWithSegRep test failure #7679

[peternied]

@andrross @dreamer-89 @sarthakaggarwal97 @mch2 Adding you folks in since you seem to have context on the previous fix - this issue might be from a different source since the callstack don't seem to align with #7679

[peternied]

[Triage - attendees 1 2 3 4 5]
@peternied thanks for filing, consider a pull request to resolve