[AUTOCUT] Gradle Check Flaky Test Report for WorkloadManagementIT

[opensearch-ci-bot]

Flaky Test Report for WorkloadManagementIT

Noticed the WorkloadManagementIT has some flaky, failing tests that failed during post-merge actions and timer trigerred run on main branch.

Details

Git Reference	Merged Pull Request	Build Details	Test Name
a60cecd	20900	72965	org.opensearch.wlm.WorkloadManagementIT.testHighCPUInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"false"}}
main	null	66605 60945 61073 61110 61276 62121 62127 62442 62448 62785 63339 63402 63515 63635 64781 65655 66129 66616 67565 67608 69312 71503	org.opensearch.wlm.WorkloadManagementIT.testHighCPUInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.wlm.WorkloadManagementIT.testHighCPUInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"true"}} org.opensearch.wlm.WorkloadManagementIT.testHighMemoryInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.wlm.WorkloadManagementIT.testHighMemoryInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"true"}} org.opensearch.wlm.WorkloadManagementIT.testHighCPUInMonitorMode {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.wlm.WorkloadManagementIT.testHighCPUInMonitorMode {p0={"search.concurrent_segment_search.enabled":"true"}} org.opensearch.wlm.WorkloadManagementIT.testHighMemoryInMonitorMode {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.wlm.WorkloadManagementIT.testNoCancellation {p0={"search.concurrent_segment_search.enabled":"false"}}
47d3369	20026	67513	org.opensearch.wlm.WorkloadManagementIT.testHighMemoryInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"true"}}
49f0b89	19789	66317	org.opensearch.wlm.WorkloadManagementIT.testHighCPUInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"true"}}
b286d2d	19476	65458	org.opensearch.wlm.WorkloadManagementIT.testHighMemoryInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"true"}}
d9d178d	19604	65427	org.opensearch.wlm.WorkloadManagementIT.testHighCPUInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"true"}}
5c8036d	19031	62394	org.opensearch.wlm.WorkloadManagementIT.testHighMemoryInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"true"}}
8fd9aeb	18631	59881	org.opensearch.wlm.WorkloadManagementIT.testHighCPUInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"true"}}
bf8696e	18548	59389	org.opensearch.wlm.WorkloadManagementIT.testHighCPUInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"true"}}
bc04657	17276	53165	org.opensearch.wlm.WorkloadManagementIT.testHighMemoryInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"false"}}
4a48b4b	17147	52716	org.opensearch.wlm.WorkloadManagementIT.testHighCPUInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"false"}}

The other pull requests, besides those involved in post-merge actions, that contain failing tests with the WorkloadManagementIT class are:

• 16818
• 18957
• 20536
• 17591
• 17043
• 17137
• 17190
• 17299
• 17390
• 17446
• 17555
• 17575
• 17631
• 17635
• 17714
• 17777
• 17817
• 18054
• 18215
• 18470
• 18491
• 18497
• 18568
• 18573
• 18639
• 18685
• 18722
• 18743
• 18771
• 18922
• 18929
• 19005
• 19033
• 19116
• 19178
• 19184
• 19209
• 19278
• 19296
• 19331
• 19357
• 19373
• 19472
• 19565
• 19715
• 19798
• 19835
• 19966
• 20150
• 20887

For more details on the failed tests refer to OpenSearch Gradle Check Metrics dashboard.

[kaushalmahi12]

@ruai0511 Can you look into it ?

[krisfreedain]

Catch All Triage - 1

[andrross]

@ruai0511 @kaushalmahi12 If I'm reading the code correctly, it looks like there is a race condition in the "testHighMemory" case here:

OpenSearch/server/src/internalClusterTest/java/org/opensearch/wlm/WorkloadManagementIT.java

Lines 414 to 418 in df3e645

	case "MEMORY":
	int bytesToAllocate = (int) (Runtime.getRuntime().totalMemory() * 0.01);
	Byte[] bytes = new Byte[bytesToAllocate];
	int[] ints = new int[bytesToAllocate];
	break;

The code will continuously loop while allocating new on heap arrays, but those arrays immediately go out of scope and are eligible for garbage collection. It seems possible to observe the resources when the doWork() loop is holding no references and the GC has freed everything, in which case the memory threshold might not be breached. It seems like it would be better to allocate the memory and hold on to the references until the task is canceled to guarantee the resource utilization.