Fix ScaleIndexIT testFullSearchOnlyReplicasFullLifecycle flaky test

[prudhvigodithi]

Description

Fix ScaleIndexIT testFullSearchOnlyReplicasFullLifecycle flaky test. Seen failures in:

• https://build.ci.opensearch.org/job/gradle-check/55829/testReport/junit/org.opensearch.action.admin.indices.scale.searchonly/ScaleIndexIT/testFullSearchOnlyReplicasFullLifecycle/
• https://build.ci.opensearch.org/job/gradle-check/55781/testReport/junit/org.opensearch.action.admin.indices.scale.searchonly/ScaleIndexIT/testFullSearchOnlyReplicasFullLifecycle/

I was not able to re-produce the error, but looking at the code using ensureYellow is better that a manual YELLOW assert. The ensureYellow also has a timeout value, so the line internalCluster().stopRandomNode(InternalTestCluster.nameFilter(findNodesWithSearchOnlyReplicas()[0])); should stop the node, leave the search replica unassigned and show YELLOW status.

Related Issues

#16720 and part of #15306

Flaky Issue: #17929

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[prudhvigodithi]

Adding @andrross @mch2 to please take a look.
Thanks
@getsaurabh02 @peterzhuamazon

[github-actions]

✅ Gradle check result for 282812a: SUCCESS

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.42%. Comparing base (c91cdcb) to head (282812a).
Report is 13 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17848      +/-   ##
============================================
- Coverage     72.46%   72.42%   -0.04%     
+ Complexity    66502    66494       -8     
============================================
  Files          5408     5408              
  Lines        308080   308168      +88     
  Branches      44720    44745      +25     
============================================
- Hits         223239   223196      -43     
- Misses        66536    66738     +202     
+ Partials      18305    18234      -71     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.