[AUTOCUT] Gradle Check Flaky Test Report for WarmIndexSegmentReplicationIT

[opensearch-ci-bot]

Flaky Test Report for WarmIndexSegmentReplicationIT

Noticed the WarmIndexSegmentReplicationIT has some flaky, failing tests that failed during post-merge actions and timer trigerred run on main branch.

Details

Git Reference	Merged Pull Request	Build Details	Test Name
main	null	66915 67982 57203 60336 60346 60883 61065 61090 62120 62599 63244 63374 63786 64498 65023 66136 66617 66899 66989 67400 67939 68254 69312 69319 69344 69395 69498 69671 69725 69763 69841 69881 69939 69943 70009 70013 70081 70347 70473 70475 70694 70810 70846 71342 71359 71567 72259 72279 72466 72567 72997 73194 73321	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testIndexWhileRecoveringReplica org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testReplicationAfterForceMergeOnPrimaryShardsOnly org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testReplicationAfterPrimaryRefreshAndFlush org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testDropPrimaryDuringReplication org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testIndexReopenClose org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryStopped_ReplicaPromoted org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testReplicaHasDiffFilesThanPrimary org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testReplicationAfterForceMerge org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testScrollCreatedOnReplica org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.classMethod org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testCancelPrimaryAllocation org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testCancellation org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testNodeDropWithOngoingReplication org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryReceivesDocsDuringReplicaRecovery org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testShardPathDeletionWhenWarmIndexRelocate org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testDeleteOperations org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRealtimeGetRequestsSuccessful org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRealtimeTermVectorRequestsSuccessful org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRestartPrimary org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRestartPrimaryAndReplicaWithDocuments org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRestartPrimary_NoReplicas org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testScrollWithConcurrentIndexAndSearch org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testUpdateOperations
0e2783c	20842	72436	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testIndexWhileRecoveringReplica
76c7b66	20695	71615	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRestartPrimary_NoReplicas
5041d43	20361	70392	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryStopped_ReplicaPromoted
a7c85f2	20427	70308	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testScrollCreatedOnReplica
9e586e5	20331	69543	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testCancelPrimaryAllocation
ce178f6	20184	68585	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testNodeDropWithOngoingReplication
859f7c3	20132	68098	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRealtimeTermVectorRequestsSuccessful
99f3189	19495	64883	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRestartPrimary
4b9aa27	18550	63886	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testReplicationAfterPrimaryRefreshAndFlush
c56da68	19296	63885	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testShardPathDeletionWhenWarmIndexRelocate
2cec4b5	18646	59878	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testScrollWithConcurrentIndexAndSearch
1adc8cd	18585	59573	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testShardPathDeletionWhenWarmIndexRelocate
cb30055	18060	57402	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryStopped_ReplicaPromoted
0795bb2	18099	57396	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryStopped_ReplicaPromoted

The other pull requests, besides those involved in post-merge actions, that contain failing tests with the WarmIndexSegmentReplicationIT class are:

• 17617
• 18147
• 18333
• 17507
• 17902
• 19003
• 18467
• 18565
• 18599
• 19703
• 20160
• 20192
• 20503
• 17556
• 17580
• 17598
• 17606
• 17792
• 17890
• 18069
• 18070
• 18116
• 18166
• 18195
• 18424
• 18573
• 18593
• 18610
• 18621
• 18631
• 18656
• 18664
• 18702
• 18720
• 18749
• 18800
• 18824
• 18826
• 18842
• 18887
• 18890
• 18929
• 18944
• 18951
• 18973
• 19001
• 19007
• 19054
• 19055
• 19132
• 19152
• 19165
• 19226
• 19278
• 19294
• 19307
• 19387
• 19508
• 19555
• 19576
• 19619
• 19648
• 19684
• 19711
• 19744
• 19782
• 19798
• 19857
• 19868
• 19933
• 19935
• 19966
• 20017
• 20057
• 20065
• 20080
• 20082
• 20123
• 20145
• 20204
• 20251
• 20280
• 20345
• 20351
• 20371
• 20377
• 20434
• 20478
• 20511
• 20514
• 20585
• 20623
• 20672
• 20683
• 20691
• 20702
• 20802
• 20810
• 20840
• 20854
• 20869
• 20924
• 20929
• 20978

For more details on the failed tests refer to OpenSearch Gradle Check Metrics dashboard.

[andrross]

FYI @skumawat2025

[andrross]

https://build.ci.opensearch.org/job/gradle-check/57471/consoleText:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryReceivesDocsDuringReplicaRecovery" -Dtests.seed=C5CF2E73AD7D00AF -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-PR -Dtests.timezone=Europe/Busingen -Druntime.java=21

WarmIndexSegmentReplicationIT > testPrimaryReceivesDocsDuringReplicaRecovery FAILED
    java.lang.AssertionError: Expected search hits on node: node_t1 to be at least 2 but was: 1
        at __randomizedtesting.SeedInfo.seed([C5CF2E73AD7D00AF:68976B8B8CAA7ED1]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:122)
        at org.opensearch.test.OpenSearchTestCase.assertBusyWithFixedSleepTime(OpenSearchTestCase.java:1214)
        at org.opensearch.indices.replication.SegmentReplicationBaseIT.waitForSearchableDocs(SegmentReplicationBaseIT.java:117)
        at org.opensearch.indices.replication.SegmentReplicationBaseIT.waitForSearchableDocs(SegmentReplicationBaseIT.java:112)
        at org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryReceivesDocsDuringReplicaRecovery(WarmIndexSegmentReplicationIT.java:1211)

https://build.ci.opensearch.org/job/gradle-check/57472/consoleText:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testDeleteOperations" -Dtests.seed=3D049AF853D4BA75 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=el-CY -Dtests.timezone=Antarctica/Troll -Druntime.java=21

WarmIndexSegmentReplicationIT > testDeleteOperations FAILED
    java.lang.AssertionError: 
    Expected: a value equal to or greater than <2L>
         but: <0L> was less than <2L>
        at __randomizedtesting.SeedInfo.seed([3D049AF853D4BA75:8E50F0725EFA66E3]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.junit.Assert.assertThat(Assert.java:930)
        at org.opensearch.test.OpenSearchIntegTestCase.lambda$waitForDocs$3(OpenSearchIntegTestCase.java:1078)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1182)
        at org.opensearch.test.OpenSearchIntegTestCase.waitForDocs(OpenSearchIntegTestCase.java:1045)
        at org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testDeleteOperations(WarmIndexSegmentReplicationIT.java:794)

[skumawat2025]

Looks like some of the tests are still being flaky - Will take a look and get these fixed.