Add option to run BWC tests in distribution level

[zelinh]

Description

Add option to run BWC tests in distribution level, which is running tests in the test cluster with latest distribution bundle installed and all plugins (included in the latest manifest) present.

The command to run BWC tests in distribution level would be ./gradlew bwcTestSuite -Dtests.security.manager=false -PcustomDistributionDownloadType=bundle
If property customDistributionDownloadType is not set nor set to bundle, the BWC tests would be default to run in plugins level as previously configured.

Issues Resolved

Part of opensearch-project/opensearch-build#2870

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #766 (b6af132) into 2.x (8ce4493) will increase coverage by 0.06%.
The diff coverage is n/a.

❗ Current head b6af132 differs from pull request most recent head 8643fc6. Consider uploading reports for the commit 8643fc6 to get more accurate results

@@             Coverage Diff              @@
##                2.x     #766      +/-   ##
============================================
+ Coverage     79.43%   79.49%   +0.06%     
- Complexity     4293     4295       +2     
============================================
  Files           302      302              
  Lines         17956    17956              
  Branches       1906     1906              
============================================
+ Hits          14263    14274      +11     
+ Misses         2790     2776      -14     
- Partials        903      906       +3     

Flag	Coverage Δ
plugin	79.49% <ø> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ava/org/opensearch/ad/task/ADHCBatchTaskCache.java	88.88% <0.00%> (-1.24%)	⬇️
...java/org/opensearch/ad/task/ADBatchTaskRunner.java	81.76% <0.00%> (-0.76%)	⬇️
...rch/ad/transport/AnomalyResultTransportAction.java	81.10% <0.00%> (+0.23%)	⬆️
...ain/java/org/opensearch/ad/task/ADTaskManager.java	77.80% <0.00%> (+0.75%)	⬆️
.../main/java/org/opensearch/ad/NodeStateManager.java	76.47% <0.00%> (+1.30%)	⬆️
...rch/ad/transport/ForwardADTaskTransportAction.java	97.45% <0.00%> (+3.38%)	⬆️

[ohltyler]

Can this be targeted to main and backported to 2.x? Or rather, will there be a separate PR with 2.x BWC versions for main?

[zelinh]

Can this be targeted to main and backported to 2.x? Or rather, will there be a separate PR with 2.x BWC versions for main?

Yeah currently the distribution build for our 3.0.0 are failing and we don't have a stable bundle to test on. So I'm targeting to 2.5 version for now. I could draft a new PR to onboard on main when we getting close.

[ohltyler]

Can this be targeted to main and backported to 2.x? Or rather, will there be a separate PR with 2.x BWC versions for main?

Yeah currently the distribution build for our 3.0.0 are failing and we don't have a stable bundle to test on. So I'm targeting to 2.5 version for now. I could draft a new PR to onboard on main when we getting close.

Got it. I think as long as this is tracked on the infra side this is fine, thanks!