[BUG]org.opensearch.indices.replication.SegmentReplicationAllocationIT.testSingleIndexShardAllocation is flaky

[gaobinlong]

Describe the bug

See the test failure: https://build.ci.opensearch.org/job/gradle-check/28158/testReport/junit/org.opensearch.indices.replication/SegmentReplicationAllocationIT/testSingleIndexShardAllocation/

To Reproduce
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.SegmentReplicationAllocationIT.testSingleIndexShardAllocation" -Dtests.seed=7F70CFAC7DF5CE33 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=tr -Dtests.timezone=Asia/Oral -Druntime.java=17

Expected behavior
Fix the flaky test.

[dreamer-89]

The test creates 50 shard 1 replica for single index on a 5 data node cluster with one dedicated cluster manager node. The assertion to ensure green cluster trips post stopping one data node. The request to obtain cluster health times out even though all shards are in assigned state. I don't understand the reason for request time out here. Tagging folks who may know and can help @shwetathareja @soosinha.

1> [2023-11-15T00:46:57,384][INFO ][o.o.i.r.SegmentReplicationAllocationIT] [testSingleIndexShardAllocation] ensureGreen timed out, cluster state:
  1> cluster uuid: orO7R8vsRYmor5yeWwLSVw [committed: true]
  1> version: 76
  1> state uuid: vN8ySEMoTPyOw54N0g3mtg
  1> from_diff: false
  1> meta data version: 54
  1>    coordination_metadata:
  1>       term: 1
  1>       last_committed_config: VotingConfiguration{hVmxaxMrQnqidcz_30lqBA,4gBVx4fvSQmkrUvpb9YwPA,ItVjyrpyQRK4BPoNPVzKMw,VQ5hwxBzR1u_0saX5eJzKg,jtKLIchnSLKJA_TLr7-VtA}
  1>       last_accepted_config: VotingConfiguration{hVmxaxMrQnqidcz_30lqBA,4gBVx4fvSQmkrUvpb9YwPA,ItVjyrpyQRK4BPoNPVzKMw,VQ5hwxBzR1u_0saX5eJzKg,jtKLIchnSLKJA_TLr7-VtA}
  1>       voting tombstones: []
  1>    [test/1odmZmTERqKM3BcOCvW2Sg]: v[48], mv[1], sv[1], av[1]
  1>       0: p_term [2], isa_ids [deGaVB2PQdGMyU1OaNXcDQ, hjXVyVkBRSqwCdxENtVpHQ]
  1>       1: p_term [1], isa_ids [7VYxsFXpRwGkCqqbjdvMgg, b19Gr5OiQiSVDEQ9uJnWeg]
  1>       2: p_term [1], isa_ids [4ujnJ1jyQfShoXE10tP0UA, Ci5I848fT7Wf-NQbSojTZg]
  1>       3: p_term [1], isa_ids [LFeh8eUFRNKxsou_EZo7XQ, LyfDArqURXuOJ7C65p8G_Q]
  1>       4: p_term [1], isa_ids [bkKknGCuTZq6hgHGw1DA7A, nJ-nZyWDSj6MEHDYKOeEcw]
  1>       5: p_term [2], isa_ids [DxjASXPoRf27u083gp3sPw, Z-iPAWRFTFGZoZK0JGlrCA]
  1>       6: p_term [1], isa_ids [AFHX1UUHQA-xmXMBdpvdMw, 8auYrWZITsaIGDQIPlMpCg]
  1>       7: p_term [1], isa_ids [-KGuOnHHSCWiHpwK1L9EyA, 81nP9Yf1QfiFlSU_zC1iaQ]
  1>       8: p_term [1], isa_ids [tmRh-2aAT-eDSDWO2eZ4SA, F2mur4pEQXe7wJEn9zXD7Q]
  1>       9: p_term [1], isa_ids [wOFcXY3_TNuye_bIpDN3pg, e4-NSG9iQ82qle6XQjmzLw]
  1>       10: p_term [2], isa_ids [QyBP1RilSaaY1SSO3uTuzQ, 3S-lferyQmu1x-If-IJLmg]
  1>       11: p_term [1], isa_ids [nvxUjNyuS9OScxj-eZ6EFg, WlebT756TiSScAgeDaaKMw]
  1>       12: p_term [1], isa_ids [FB85wH_6Q8aJve1DVJjJDg, h8q-yn2QSUGArxLqTtCx1A]
  1>       13: p_term [1], isa_ids [ACZtruzESOGVP-psdFpMDw, rPtuLHnAQlGDqEsfmHQq3A]
  1>       14: p_term [1], isa_ids [5lHnCMmuRUm5R_MWZGjSKQ, Adc2olK2TTqs9QcyarsFWw]
  1>       15: p_term [2], isa_ids [qAMXN76CR0OPmJcxqHfKCA, zx-Zkzo-T7yBrPX0oK9t6A]
  1>       16: p_term [1], isa_ids [y3SZY72GTXeYmXRRIGHm3g, yt8eDezxQhqSSWrXxB5Y7Q]
  1>       17: p_term [1], isa_ids [JtapsSksS4iKHyCgzVR4Dg, ZmjHitbCRQaZEmTlsMghVA]
  1>       18: p_term [1], isa_ids [ehLiaBGqRAmWavoVzvbx1A, T8yV7xSiT3G_lcKODs0_gg]
  1>       19: p_term [1], isa_ids [Q5DJ1pvfTRef-tw-ySzQaA, y1aksXU2SpSOUslswcGt5Q]
  1>       20: p_term [2], isa_ids [dr8VbJmUQzyje4CFGDsr1A, EzKxlA9TTPO9mK8No6q__g]
  1>       21: p_term [1], isa_ids [vhDE4A0iTDue0PD1hLgoEQ, b3oTcdBJQc2pvFyBTxPhaw]
  1>       22: p_term [1], isa_ids [MFq0Enm0SdmpENs9AnnoNA, NbojWS97SPyQjRQG1qFndA]
  1>       23: p_term [1], isa_ids [2AV73Y_NTR22R3aSl6BhSQ, uNCk4y5pS6-YE6aNlinjnQ]
  1>       24: p_term [1], isa_ids [Xn-CFyG-RtyBFn8EZiJCCg, FaLFQdrqQU2YaJlpSdjibg]
  1>       25: p_term [2], isa_ids [phTMobimSgCo8UlSi-8zWQ, yMRiyBQOTZmh5j8rLREEog]
  1>       26: p_term [1], isa_ids [dxvqXVg-S1qMMWzgh0Gr0w, Nzk5wSXWS4G_PNfpXqF6CA]
  1>       27: p_term [1], isa_ids [-2rKc6LRR-iP9NyQec1z7w, -T57QsNVQdO2LuP0Wf_XPA]
  1>       28: p_term [1], isa_ids [m-NLbQHGSbal29P-IlMMDA, _MPqHzmMTomzAty3NVQfkg]
  1>       29: p_term [1], isa_ids [bMQikTTpQFCdtyy1T6LURg, IdG6PeHZTEqkFc5D7g4y0Q]
  1>       30: p_term [2], isa_ids [wuC_77srRDWWc9vnLYfwBg, fb_ykPCSQniCIFslTAtL3g]
  1>       31: p_term [1], isa_ids [oGz94lSOTU2NBS5mUrxNBg, zHingh4sRUiM5S1uSR8ITQ]
  1>       32: p_term [1], isa_ids [ZnglMc7-SRyEKz6IKpmLmg, ZbLdfcnPSZWUGTaTTpY22g]
  1>       33: p_term [1], isa_ids [u2nCeo7HSAye0RWPeFPK9A, DG898wP9SCOESTgdYQnS1Q]
  1>       34: p_term [1], isa_ids [JxZEiSDES0mrHbQVM0MEXQ, zTtBO_j2TBCIkJ9SRABrZg]
  1>       35: p_term [2], isa_ids [LLKBA2pUT4yc0x7T_fuKJw, LFXtfalaQv--QPM3WjnOzg]
  1>       36: p_term [1], isa_ids [7f9idJ2hSgedehHsEsDkdA, J37QgnpbRB2af3WxUPtdPA]
  1>       37: p_term [1], isa_ids [CPa6gTmsQlu1n_zWh3eOKQ, UUSSq4AiTjuUXf37MZ56VQ]
  1>       38: p_term [1], isa_ids [1b6ABJHpRvSHxothbhnaow, kqcMgBG6QTKjMakskQ50BA]
  1>       39: p_term [1], isa_ids [Ryc5wl0MRPuHoU_IC2STUA, wJfNJOmiQ-6qFaIdNn37NQ]
  1>       40: p_term [2], isa_ids [8A7ecngXRE-HQUY2vew-DA, lHprH2HfQVaHeKCubaOFoQ]
  1>       41: p_term [1], isa_ids [8n5425nPQ6WyF00LmdLzxg, gIEM8cD4TT2UPuaa0u5WOw]
  1>       42: p_term [1], isa_ids [lnWnS8nCSfa8DJlxLpV0gA, rQljkQ9eTnahn26TiT1Cbg]
  1>       43: p_term [1], isa_ids [RqlSoxI7Qm26FF-h77uSlA, Ai5QEcCzTmeJ2QK3Y4oUAQ]
  1>       44: p_term [1], isa_ids [FYgspJVcTQCRDcjxNR8AVQ, azwdytVWSYa5RczROskYsg]
  1>       45: p_term [2], isa_ids [J-jLCzsbTvOsNneB-mKcKg, il5FyZqnTtyTXWXtT1pdZw]
  1>       46: p_term [1], isa_ids [542okCcdRvCA-XAb-Tl56A, SE6y9IskRaK6RC1qEIVfXw]
  1>       47: p_term [1], isa_ids [9cMvLNZ6Sk-_z60phHlnTg, b9J0KHAVT0KUMDf6ONVOzQ]
  1>       48: p_term [1], isa_ids [jTkiV3VkR4qDLLAex9nqvw, qGmwJAmRSfOC6CtuOxAFyQ]
  1>       49: p_term [1], isa_ids [Ur2fESeFTl6rQWdTUVNg1Q, 0LAzPGBWQqyXJIW5biyWIQ]
  1> metadata customs:
  1>    index-graveyard: IndexGraveyard[[]]
  1> nodes: 
  1>    {node_t3}{hVmxaxMrQnqidcz_30lqBA}{E5FK75_sQtOhMebdY5fMBA}{127.0.0.1}{127.0.0.1:34173}{dimr}{shard_indexing_pressure_enabled=true}
  1>    {node_t2}{4gBVx4fvSQmkrUvpb9YwPA}{CWueQ7yLTcCX0iCgrV35Sw}{127.0.0.1}{127.0.0.1:33193}{dimr}{shard_indexing_pressure_enabled=true}
  1>    {node_t5}{VQ5hwxBzR1u_0saX5eJzKg}{_SsnYimkRC-gvhzDb3iDvQ}{127.0.0.1}{127.0.0.1:38423}{dimr}{shard_indexing_pressure_enabled=true}
  1>    {node_t0}{jtKLIchnSLKJA_TLr7-VtA}{Uo2ZyVO6SZ6ytmYM033rKw}{127.0.0.1}{127.0.0.1:37789}{m}{shard_indexing_pressure_enabled=true}, cluster-manager
  1>    {node_t4}{ItVjyrpyQRK4BPoNPVzKMw}{4tshpCS9TymLF93SAiZR4g}{127.0.0.1}{127.0.0.1:43149}{dimr}{shard_indexing_pressure_enabled=true}
  1> routing_table (version 64):
  1> -- index [[test/1odmZmTERqKM3BcOCvW2Sg]]
  1> ----shard_id [test][0]
  1> --------[test][0], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=deGaVB2PQdGMyU1OaNXcDQ]
  1> --------[test][0], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=hjXVyVkBRSqwCdxENtVpHQ]
  1> ----shard_id [test][1]
  1> --------[test][1], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=7VYxsFXpRwGkCqqbjdvMgg]
  1> --------[test][1], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=b19Gr5OiQiSVDEQ9uJnWeg]
  1> ----shard_id [test][2]
  1> --------[test][2], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=Ci5I848fT7Wf-NQbSojTZg]
  1> --------[test][2], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=4ujnJ1jyQfShoXE10tP0UA]
  1> ----shard_id [test][3]
  1> --------[test][3], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=LFeh8eUFRNKxsou_EZo7XQ]
  1> --------[test][3], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=LyfDArqURXuOJ7C65p8G_Q]
  1> ----shard_id [test][4]
  1> --------[test][4], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=nJ-nZyWDSj6MEHDYKOeEcw]
  1> --------[test][4], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=bkKknGCuTZq6hgHGw1DA7A]
  1> ----shard_id [test][5]
  1> --------[test][5], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=DxjASXPoRf27u083gp3sPw]
  1> --------[test][5], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Z-iPAWRFTFGZoZK0JGlrCA]
  1> ----shard_id [test][6]
  1> --------[test][6], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=8auYrWZITsaIGDQIPlMpCg]
  1> --------[test][6], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=AFHX1UUHQA-xmXMBdpvdMw]
  1> ----shard_id [test][7]
  1> --------[test][7], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=-KGuOnHHSCWiHpwK1L9EyA]
  1> --------[test][7], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=81nP9Yf1QfiFlSU_zC1iaQ]
  1> ----shard_id [test][8]
  1> --------[test][8], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=F2mur4pEQXe7wJEn9zXD7Q]
  1> --------[test][8], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=tmRh-2aAT-eDSDWO2eZ4SA]
  1> ----shard_id [test][9]
  1> --------[test][9], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=wOFcXY3_TNuye_bIpDN3pg]
  1> --------[test][9], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=e4-NSG9iQ82qle6XQjmzLw]
  1> ----shard_id [test][10]
  1> --------[test][10], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=3S-lferyQmu1x-If-IJLmg]
  1> --------[test][10], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=QyBP1RilSaaY1SSO3uTuzQ]
  1> ----shard_id [test][11]
  1> --------[test][11], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=WlebT756TiSScAgeDaaKMw]
  1> --------[test][11], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=nvxUjNyuS9OScxj-eZ6EFg]
  1> ----shard_id [test][12]
  1> --------[test][12], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=h8q-yn2QSUGArxLqTtCx1A]
  1> --------[test]  1> [12], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=FB85wH_6Q8aJve1DVJjJDg]
  1> ----shard_id [test][13]
  1> --------[test][13], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=ACZtruzESOGVP-psdFpMDw]
  1> --------[test][13], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=rPtuLHnAQlGDqEsfmHQq3A]
  1> ----shard_id [test][14]
  1> --------[test][14], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=Adc2olK2TTqs9QcyarsFWw]
  1> --------[test][14], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=5lHnCMmuRUm5R_MWZGjSKQ]
  1> ----shard_id [test][15]
  1> --------[test][15], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=qAMXN76CR0OPmJcxqHfKCA]
  1> --------[test][15], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=zx-Zkzo-T7yBrPX0oK9t6A]
  1> ----shard_id [test][16]
  1> --------[test][16], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=yt8eDezxQhqSSWrXxB5Y7Q]
  1> --------[test][16], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=y3SZY72GTXeYmXRRIGHm3g]
  1> ----shard_id [test][17]
  1> --------[test][17], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=JtapsSksS4iKHyCgzVR4Dg]
  1> --------[test][17], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=ZmjHitbCRQaZEmTlsMghVA]
  1> ----shard_id [test][18]
  1> --------[test][18], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=ehLiaBGqRAmWavoVzvbx1A]
  1> --------[test][18], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=T8yV7xSiT3G_lcKODs0_gg]
  1> ----shard_id [test][19]
  1> --------[test][19], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=y1aksXU2SpSOUslswcGt5Q]
  1> --------[test][19], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Q5DJ1pvfTRef-tw-ySzQaA]
  1> ----shard_id [test][20]
  1> --------[test][20], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=EzKxlA9TTPO9mK8No6q__g]
  1> --------[test][20], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=dr8VbJmUQzyje4CFGDsr1A]
  1> ----shard_id [test][21]
  1> --------[test][21], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=b3oTcdBJQc2pvFyBTxPhaw]
  1> --------[test][21], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=vhDE4A0iTDue0PD1hLgoEQ]
  1> ----shard_id [test][22]
  1> --------[test][22], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=MFq0Enm0SdmpENs9AnnoNA]
  1> --------[test][22], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=NbojWS97SPyQjRQG1qFndA]
  1> ----shard_id [test][23]
  1> --------[test][23], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=2AV73Y_NTR22R3aSl6BhSQ]
  1> --------[test][23], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=uNCk4y5pS6-YE6aNlinjnQ]
  1> ----shard_id [test][24]
  1> --------[test][24], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=FaLFQdrqQU2YaJlpSdjibg]
  1> --------[test][24], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Xn-CFyG-RtyBFn8EZiJCCg]
  1> ----shard_id [test][25]
  1> --------[test][25], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=phTMobimSgCo8UlSi-8zWQ]
  1> --------[test][25], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=yMRiyBQOTZmh5j8rLREEog]
  1> ----shard_id [test][26]
  1> --------[test][26], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=Nzk5wSXWS4G_PNfpXqF6CA]
  1> --------[test][26], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=dxvqXVg-S1qMMWzgh0Gr0w]
  1> ----shard_id [test][27]
  1> --------[test][27], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=-2rKc6LRR-iP9NyQec1z7w]
  1> --------[test][27], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=-T57QsNVQdO2LuP0Wf_XPA]
  1> ----shard_id [test][28]
  1> --------[test][28], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=m-NLbQHGSbal29P-IlMMDA]
  1> --------[test][28], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=_MPqHzmMTomzAty3NVQfkg]
  1> ----shard_id [test][29]
  1> --------[test][29], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=bMQikTTpQFCdtyy1T6LURg]
  1> --------[test][29], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=IdG6PeHZTEqkFc5D7g4y0Q]
  1> ----shard_id [test][30]
  1> --------[test][30], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=fb_ykPCSQniCIFslTAtL3g]
  1> --------[test][30], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=wuC_77srRDWWc9vnLYfwBg]
  1> ----shard_id [test][31]
  1> --------[test][31], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=oGz94lSOTU2NBS5mUrxNBg]
  1> --------[test][31], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=zHingh4sRUiM5S1uSR8ITQ]
  1> ----shard_id [test][32]
  1> --------[test][32], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=ZnglMc7-SRyEKz6IKpmLmg]
  1> --------[test][32], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=ZbLdfcnPSZWUGTaTTpY22g]
  1> ----shard_id [test][33]
  1> --------[test][33], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=u2nCeo7HSAye0RWPeFPK9A]
  1> --------[test][33], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=DG898wP9SCOESTgdYQnS1Q]
  1> ----shard_id [test][34]
  1> --------[test][34], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=JxZEiSDES0mrHbQVM0MEXQ]
  1> --------[test][34], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=zTtBO_j2TBCIkJ9SRABrZg]
  1> ----shard_id [test][35]
  1> --------[test][35], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=LLKBA2pUT4yc0x7T_fuKJw]
  1> --------[test][35], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=LFXtfalaQv--QPM3WjnOzg]
  1> ----shard_id [test][36]
  1> --------[test][36], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=J37QgnpbRB2af3WxUPtdPA]
  1> --------[test][36], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=7f9idJ2hSgedehHsEsDkdA]
  1> ----shard_id [test][37]
  1> --------[test][37], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=CPa6gTmsQlu1n_zWh3eOKQ]
  1> --------[test][37], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=UUSSq4AiTjuUXf37MZ56VQ]
  1> ----shard_id [test][38]
  1> --------[test][38], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=kqcMgBG6QTKjMakskQ50BA]
  1> --------[test][38], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=1b6ABJHpRvSHxothbhnaow]
  1> ----shard_id [test][39]
  1> --------[test][39], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=wJfNJOmiQ-6qFaIdNn37NQ]
  1> --------[test][39], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Ryc5wl0MRPuHoU_IC2STUA]
  1> ----shard_id [test][40]
  1> --------[test][40], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=lHprH2HfQVaHeKCubaOFoQ]
  1> --------[test][40], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=8A7ecngXRE-HQUY2vew-DA]
  1> ----shard_id [test][41]
  1> --------[test][41], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=gIEM8cD4TT2UPuaa0u5WOw]
  1> --------[test][41], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=8n5425nPQ6WyF00LmdLzxg]
  1> ----shard_id [test][42]
  1> --------[test][42], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=rQljkQ9eTnahn26TiT1Cbg]
  1> --------[test][42], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=lnWnS8nCSfa8DJlxLpV0gA]
  1> ----shard_id [test][43]
  1> --------[test][43], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=RqlSoxI7Qm26FF-h77uSlA]
  1> --------[test][43], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=Ai5QEcCzTmeJ2QK3Y4oUAQ]
  1> ----shard_id [test][44]
  1> --------[test][44], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=azwdytVWSYa5RczROskYsg]
  1> --------[test][44], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=FYgspJVcTQCRDcjxNR8AVQ]
  1> ----shard_id [test][45]
  1> --------[test][45], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=il5FyZqnTtyTXWXtT1pdZw]
  1> --------[test][45], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=J-jLCzsbTvOsNneB-mKcKg]
  1> ----shard_id [test][46]
  1> --------[test][46], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=SE6y9IskRaK6RC1qEIVfXw]
  1> --------[test][46], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=542okCcdRvCA-XAb-Tl56A]
  1> ----shard_id [test][47]
  1> --------[test][47], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=9cMvLNZ6Sk-_z60phHlnTg]
  1> --------[test][47], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=b9J0KHAVT0KUMDf6ONVOzQ]
  1> ----shard_id [test][48]
  1> --------[test][48], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=jTkiV3VkR4qDLLAex9nqvw]
  1> --------[test][48], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=qGmwJAmRSfOC6CtuOxAFyQ]
  1> ----shard_id [test][49]
  1> --------[test][49], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=0LAzPGBWQqyXJIW5biyWIQ]
  1> --------[test][49], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=Ur2fESeFTl6rQWdTUVNg1Q]

  1> routing_nodes:
  1> -----node_id[hVmxaxMrQnqidcz_30lqBA][V]
  1> --------[test][0], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTE  1> D], a[id=deGaVB2PQdGMyU1OaNXcDQ]
  1> --------[test][1], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=7VYxsFXpRwGkCqqbjdvMgg]
  1> --------[test][6], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=8auYrWZITsaIGDQIPlMpCg]
  1> --------[test][10], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=3S-lferyQmu1x-If-IJLmg]
  1> --------[test][11], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=WlebT756TiSScAgeDaaKMw]
  1> --------[test][16], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=yt8eDezxQhqSSWrXxB5Y7Q]
  1> --------[test][20], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=EzKxlA9TTPO9mK8No6q__g]
  1> --------[test][21], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=b3oTcdBJQc2pvFyBTxPhaw]
  1> --------[test][26], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=Nzk5wSXWS4G_PNfpXqF6CA]
  1> --------[test][30], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=fb_ykPCSQniCIFslTAtL3g]
  1> --------[test][31], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=oGz94lSOTU2NBS5mUrxNBg]
  1> --------[test][36], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=J37QgnpbRB2af3WxUPtdPA]
  1> --------[test][41], node[hVmxaxMrQnqidcz_30lqBA], [P], s[STARTED], a[id=gIEM8cD4TT2UPuaa0u5WOw]
  1> --------[test][29], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=bMQikTTpQFCdtyy1T6LURg]
  1> --------[test][32], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=ZnglMc7-SRyEKz6IKpmLmg]
  1> --------[test][34], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=JxZEiSDES0mrHbQVM0MEXQ]
  1> --------[test][35], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=LLKBA2pUT4yc0x7T_fuKJw]
  1> --------[test][39], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=wJfNJOmiQ-6qFaIdNn37NQ]
  1> --------[test][40], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=lHprH2HfQVaHeKCubaOFoQ]
  1> --------[test][42], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=rQljkQ9eTnahn26TiT1Cbg]
  1> --------[test][43], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=RqlSoxI7Qm26FF-h77uSlA]
  1> --------[test][44], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=azwdytVWSYa5RczROskYsg]
  1> --------[test][45], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=il5FyZqnTtyTXWXtT1pdZw]
  1> --------[test][46], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=SE6y9IskRaK6RC1qEIVfXw]
  1> --------[test][47], node[hVmxaxMrQnqidcz_30lqBA], [R], s[STARTED], a[id=9cMvLNZ6Sk-_z60phHlnTg]
  1> -----node_id[4gBVx4fvSQmkrUvpb9YwPA][V]
  1> --------[test][2], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=Ci5I848fT7Wf-NQbSojTZg]
  1> --------[test][7], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=-KGuOnHHSCWiHpwK1L9EyA]
  1> --------[test][12], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=h8q-yn2QSUGArxLqTtCx1A]
  1> --------[test][17], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=JtapsSksS4iKHyCgzVR4Dg]
  1> --------[test][22], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=MFq0Enm0SdmpENs9AnnoNA]
  1> --------[test][27], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=-2rKc6LRR-iP9NyQec1z7w]
  1> --------[test][32], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=ZbLdfcnPSZWUGTaTTpY22g]
  1> --------[test][37], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=CPa6gTmsQlu1n_zWh3eOKQ]
  1> --------[test][42], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=lnWnS8nCSfa8DJlxLpV0gA]
  1> --------[test][45], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=J-jLCzsbTvOsNneB-mKcKg]
  1> --------[test][47], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=b9J0KHAVT0KUMDf6ONVOzQ]
  1> --------[test][49], node[4gBVx4fvSQmkrUvpb9YwPA], [P], s[STARTED], a[id=0LAzPGBWQqyXJIW5biyWIQ]
  1> --------[test][0], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=hjXVyVkBRSqwCdxENtVpHQ]
  1> --------[test][3], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=LFeh8eUFRNKxsou_EZo7XQ]
  1> --------[test][4], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=nJ-nZyWDSj6MEHDYKOeEcw]
  1> --------[test][6], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=AFHX1UUHQA-xmXMBdpvdMw]
  1> --------[test][10], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=QyBP1RilSaaY1SSO3uTuzQ]
  1> --------[test][13], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=ACZtruzESOGVP-psdFpMDw]
  1> --------[test][14], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=Adc2olK2TTqs9QcyarsFWw]
  1> --------[test][16], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=y3SZY72GTXeYmXRRIGHm3g]
  1> --------[test][20], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=dr8VbJmUQzyje4CFGDsr1A]
  1> --------[test][23], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=2AV73Y_NTR22R3aSl6BhSQ]
  1> --------[test][24], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=FaLFQdrqQU2YaJlpSdjibg]
  1> --------[test][26], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=dxvqXVg-S1qMMWzgh0Gr0w]
  1> --------[test][30], node[4gBVx4fvSQmkrUvpb9YwPA], [R], s[STARTED], a[id=wuC_77srRDWWc9vnLYfwBg]
  1> -----node_id[VQ5hwxBzR1u_0saX5eJzKg][V]
  1> --------[test][3], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=LyfDArqURXuOJ7C65p8G_Q]
  1> --------[test][8], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=F2mur4pEQXe7wJEn9zXD7Q]
  1> --------[test][13], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=rPtuLHnAQlGDqEsfmHQq3A]
  1> --------[test][18], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=ehLiaBGqRAmWavoVzvbx1A]
  1> --------[test][23], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=uNCk4y5pS6-YE6aNlinjnQ]
  1> --------[test][28], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=m-NLbQHGSbal29P-IlMMDA]
  1> --------[test][33], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=u2nCeo7HSAye0RWPeFPK9A]
  1> --------[test][38], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=kqcMgBG6QTKjMakskQ50BA]
  1> --------[test][40], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=8A7ecngXRE-HQUY2vew-DA]
  1> --------[test][43], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=Ai5QEcCzTmeJ2QK3Y4oUAQ]
  1> --------[test][46], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=542okCcdRvCA-XAb-Tl56A]
  1> --------[test][48], node[VQ5hwxBzR1u_0saX5eJzKg], [P], s[STARTED], a[id=jTkiV3VkR4qDLLAex9nqvw]
  1> --------[test][1], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=b19Gr5OiQiSVDEQ9uJnWeg]
  1> --------[test][2], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=4ujnJ1jyQfShoXE10tP0UA]
  1> --------[test][5], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=DxjASXPoRf27u083gp3sPw]
  1> --------[test][9], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=wOFcXY3_TNuye_bIpDN3pg]
  1> --------[test][11], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=nvxUjNyuS9OScxj-eZ6EFg]
  1> --------[test][12], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=FB85wH_6Q8aJve1DVJjJDg]
  1> --------[test][15], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=qAMXN76CR0OPmJcxqHfKCA]
  1> --------[test][19], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=y1aksXU2SpSOUslswcGt5Q]
  1> --------[test][21], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=vhDE4A0iTDue0PD1hLgoEQ]
  1> --------[test][22], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=NbojWS97SPyQjRQG1qFndA]
  1> --------[test][25], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=phTMobimSgCo8UlSi-8zWQ]
  1> --------[test][31], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=zHingh4sRUiM5S1uSR8ITQ]
  1> --------[test][41], node[VQ5hwxBzR1u_0saX5eJzKg], [R], s[STARTED], a[id=8n5425nPQ6WyF00LmdLzxg]
  1> -----node_id[ItVjyrpyQRK4BPoNPVzKMw][V]
  1> --------[test][4], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=bkKknGCuTZq6hgHGw1DA7A]
  1> --------[test][5], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Z-iPAWRFTFGZoZK0JGlrCA]
  1> --------[test][9], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=e4-NSG9iQ82qle6XQjmzLw]
  1> --------[test][14], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=5lHnCMmuRUm5R_MWZGjSKQ]
  1> --------[test][15], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=zx-Zkzo-T7yBrPX0oK9t6A]
  1> --------[test][19], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Q5DJ1pvfTRef-tw-ySzQaA]
  1> --------[test][24], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Xn-CFyG-RtyBFn8EZiJCCg]
  1> --------[test][25], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=yMRiyBQOTZmh5j8rLREEog]
  1> --------[test][29], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=IdG6PeHZTEqkFc5D7g4y0Q]
  1> --------[test][34], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=zTtBO_j2TBCIkJ9S  1> RABrZg]
  1> --------[test][35], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=LFXtfalaQv--QPM3WjnOzg]
  1> --------[test][39], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=Ryc5wl0MRPuHoU_IC2STUA]
  1> --------[test][44], node[ItVjyrpyQRK4BPoNPVzKMw], [P], s[STARTED], a[id=FYgspJVcTQCRDcjxNR8AVQ]
  1> --------[test][7], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=81nP9Yf1QfiFlSU_zC1iaQ]
  1> --------[test][8], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=tmRh-2aAT-eDSDWO2eZ4SA]
  1> --------[test][17], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=ZmjHitbCRQaZEmTlsMghVA]
  1> --------[test][18], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=T8yV7xSiT3G_lcKODs0_gg]
  1> --------[test][27], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=-T57QsNVQdO2LuP0Wf_XPA]
  1> --------[test][28], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=_MPqHzmMTomzAty3NVQfkg]
  1> --------[test][33], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=DG898wP9SCOESTgdYQnS1Q]
  1> --------[test][36], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=7f9idJ2hSgedehHsEsDkdA]
  1> --------[test][37], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=UUSSq4AiTjuUXf37MZ56VQ]
  1> --------[test][38], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=1b6ABJHpRvSHxothbhnaow]
  1> --------[test][48], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=qGmwJAmRSfOC6CtuOxAFyQ]
  1> --------[test][49], node[ItVjyrpyQRK4BPoNPVzKMw], [R], s[STARTED], a[id=Ur2fESeFTl6rQWdTUVNg1Q]
  1> ---- unassigned

[soosinha]

@dreamer-89 Looks like the cluster health is not moving to green. Can you print and check the cluster health information while ensureGreen method is running ?

[dreamer-89]

@dreamer-89 Looks like the cluster health is not moving to green. Can you print and check the cluster health information while ensureGreen method is running ?

Thanks @soosinha for the reply. From logs, I see cluster indeed turned green post node shutdown but did not respond to cluster health call.

 1> [2023-12-12T23:49:25,492][INFO ][o.o.c.r.a.AllocationService] [node_t0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][44], [test][48]]]).
  1> [2023-12-12T23:49:27,687][INFO ][o.o.i.s.CheckpointRefreshListener] [node_t3] All permits are acquired and refresh listener is closed
  1> [2023-12-12T23:49:27,723][INFO ][o.o.i.s.CheckpointRefreshListener] [node_t4] All permits are acquired and refresh listener is closed
  1> [2023-12-12T23:49:27,960][INFO ][o.o.i.r.RecoverySourceHandler] [node_t3] [test][43][recover to node_t5] finalizing recovery took [283.4ms]
  1> [2023-12-12T23:49:28,014][INFO ][o.o.i.r.RecoverySourceHandler] [node_t4] [test][45][recover to node_t3] finalizing recovery took [328.3ms]
  1> [2023-12-12T23:49:31,721][INFO ][o.o.i.s.CheckpointRefreshListener] [node_t3] All permits are acquired and refresh listener is closed
...
  1> [2023-12-12T23:49:44,958][INFO ][o.o.i.r.RecoverySourceHandler] [node_t2] [test][48][recover to node_t4] finalizing recovery took [86.2ms]
  1> [2023-12-12T23:49:45,225][INFO ][o.o.i.r.RecoverySourceHandler] [node_t4] [test][35][recover to node_t5] finalizing recovery took [417.8ms]
  1> [2023-12-12T23:49:50,087][INFO ][o.o.i.r.RecoverySourceHandler] [node_t5] [test][49][recover to node_t4] finalizing recovery took [77.3ms]
  1> [2023-12-12T23:49:50,111][INFO ][o.o.i.r.RecoverySourceHandler] [node_t3] [test][38][recover to node_t4] finalizing recovery took [96.3ms]
  1> [2023-12-12T23:49:54,504][INFO ][o.o.i.r.RecoverySourceHandler] [node_t2] [test][37][recover to node_t4] finalizing recovery took [69.2ms]
  1> [2023-12-12T23:49:54,683][INFO ][o.o.i.r.RecoverySourceHandler] [node_t5] [test][46][recover to node_t4] finalizing recovery took [101.9ms]
  1> [2023-12-12T23:49:59,199][INFO ][o.o.i.r.SegmentReplicationAllocationIT] [testSingleIndexShardAllocation] ensureGreen timed out, cluster state:
  1> cluster uuid: QbCDs-UAQaaz2qkoD9LInA [committed: true]