[BUG] org.opensearch.recovery.RecoveryWhileUnderLoadIT.testRecoverWhileUnderLoadAllocateReplicasRelocatePrimariesTest {p0={"cluster.indices.replication.strategy":"DOCUMENT"}} is flaky #13820
Description
Describe the bug
The test case org.opensearch.recovery.RecoveryWhileUnderLoadIT.testRecoverWhileUnderLoadAllocateReplicasRelocatePrimariesTest {p0={"cluster.indices.replication.strategy":"DOCUMENT"}} is flaky:
java.lang.AssertionError: All incoming requests on node [node_s4] should have finished. Expected 0 but got 368; pending tasks [[{
"node" : "node_s4",
"id" : 76,
"type" : "direct",
"action" : "indices:admin/seq_no/retention_lease_sync[p]",
"status" : {
"phase" : "primary"
},
"description" : "retention_lease_sync shardId=[test][3]",
"start_time" : "2024-05-24T17:25:25.617Z",
"start_time_in_millis" : 1716571525617,
"running_time" : "1m",
"running_time_in_nanos" : 65752135745,
"cancellable" : false,
"cancelled" : false,
"parent_task_id" : "MWBqDHQmSkefpKFbNwRdHg:75",
"headers" : { },
"resource_stats" : {
"average" : {
"cpu_time_in_nanos" : 0,
"memory_in_bytes" : 0
},
"total" : {
"cpu_time_in_nanos" : 0,
"memory_in_bytes" : 0
},
"min" : {
"cpu_time_in_nanos" : 0,
"memory_in_bytes" : 0
},
"max" : {
"cpu_time_in_nanos" : 0,
"memory_in_bytes" : 0
},
"thread_info" : {
"thread_executions" : 0,
"active_threads" : 0
}
}
},{
"node" : "node_s4",
"id" : 57,
"type" : "transport",
"action" : "internal:index/shard/recovery/start_recovery",
"description" : "",
"start_time" : "2024-05-24T17:25:25.549Z",
"start_time_in_millis" : 1716571525549,
"running_time" : "1m",
"running_time_in_nanos" : 65820120649,
"cancellable" : false,
"cancelled" : false,
"headers" : { },
"resource_stats" : {
"average" : {
"cpu_time_in_nanos" : 0,
"memory_in_bytes" : 0
},
"total" : {
"cpu_time_in_nanos" : 0,
"memory_in_bytes" : 0
},
"min" : {
"cpu_time_in_nanos" : 0,
"memory_in_bytes" : 0
},
"max" : {
"cpu_time_in_nanos" : 0,
"memory_in_bytes" : 0
},
"thread_info" : {
"thread_executions" : 0,
"active_threads" : 0
}
}
},{
"node" : "node_s4",
"id" : 75,
"type" : "transport",
"action" : "retention_lease_sync",
"status" : {
"phase" : "starting"
},
"description" : "retention_lease_sync shardId=[test][3]",
"start_time" : "2024-05-24T17:25:25.617Z",
"start_time_in_millis" : 1716571525617,
"running_time" : "1m",
"running_time_in_nanos" : 65752504066,
"cancellable" : false,
"cancelled" : false,
"headers" : { },
"resource_stats" : {
"average" : {
"cpu_time_in_nanos" : 0,
"memory_in_bytes" : 0
},
"total" : {
"cpu_time_in_nanos" : 0,
"memory_in_bytes" : 0
},
"min" : {
"cpu_time_in_nanos" : 0,
"memory_in_bytes" : 0
},
"max" : {
"cpu_time_in_nanos" : 0,
"memory_in_bytes" : 0
},
"thread_info" : {
"thread_executions" : 0,
"active_threads" : 0
}
}
}]]
Related component
Storage
To Reproduce
./gradlew ':server:internalClusterTest' --tests "org.opensearch.recovery.RecoveryWhileUnderLoadIT" -Dtests.method="testRecoverWhileUnderLoadAllocateReplicasRelocatePrimariesTest {p0={"cluster.indices.replication.strategy":"DOCUMENT"}}" -Dtests.seed=FE4B39551EA78350
Expected behavior
The test must always pass
Additional Details
Plugins
Please list all plugins currently enabled.
Screenshots
Standard
Host/Environment (please complete the following information):
- CI
Additional context