BWC Tests are failing on mixedClusterTask

[cwperks]

BWC Tests on the Security repo have been failing for the last 3 days on the mixedClusterTask which upgrades one of three nodes from 2.4.0 -> 3.0.0 and ensures the upgraded node can join the cluster. I am filing the issue on core repo because on commits where BWC Tests were previously working for Security the check is now failing leading me to believe the failure is due to an upstream change.

See a recent example of BWC test failure here: https://github.com/opensearch-project/security/actions/runs/3372773855/jobs/5611052811

This is blocking multiple PRs.

* What went wrong:
Execution failed for task ':securityBwcCluster#mixedClusterTask'.

> `cluster{::securityBwcCluster0}` failed to wait for cluster health yellow after 40 SECONDS
    IO error while waiting cluster
    503 Service Unavailable

Previously working: https://github.com/opensearch-project/security/actions/runs/3341443487

Same commit and now failing: https://github.com/cwperks/security/actions/runs/3387995860

Link to corresponding issue in security repo: opensearch-project/security#2221

[minalsha]

@cwperks feel free to assess the failure where it is failing and share. Also, note: do we need compatibility between major verions 2.x and 3.0?

[cwperks]

We do need compatibility between adjacent major versions to ensure that rolling upgrades are possible.

I believe the issue may be related to this PR: #5003

I spun up a mixed cluster using the docker images on opensearchstaging with two 2.4.0 nodes and one 3.0.0 node and received the following error;

java.lang.IllegalStateException: Received message from unsupported version: [2.4.0] minimal compatible version is: [2.5.0]

2.5.0 manifest does not exist and no builds of 2.5.0 have been performed yet. The bwcTests may fail until artifacts for 2.5.0 become available.

[peternied]

I've created a pull request to revert the change that is blocking security's backward compatibility tests and I suspect other plugins are impacted. #5090

Going to use that PR to help drive the getting the branch unblocked