[BUG] Cluster crushes after upgrade of one node (rolling upgrade) 2.4.0->3.0.0

[martin-gaievski]

Describe the bug
Cluster return

cluster failed to wait for cluster health yellow after 40 SECONDS
IO error while waiting cluster
503 Service Unavailable

after one out of three nodes got upgraded from 2.4.0 to 3.0.0. It's a test cluster, we use core build-tools to construct it and upgrade the node.

To Reproduce
We're seeing this as part of our test BWC workflow in CI example. It's also reproducible in local dev environment.

Steps to reproduce the behavior locally:

1. Clone k-NN main branch from https://github.com/opensearch-project/k-NN
2. Execute command below from dev guide - https://github.com/opensearch-project/k-NN/blob/main/DEVELOPER_GUIDE.md#backwards-compatibility-testing:

./gradlew :qa:bwcTestSuite -Dbwc.version=2.4.0

or narrowed version for rolling upgrade

./gradlew :qa:rolling-upgrade:testRollingUpgrade -Dbwc.version=2.4.0

I've constructed minimal setup that run just one test and test doesn't call any of k-NN specific API. I've captured log files from each cluster node individually, please check attached zip. In that test we started 3 nodes cluster, then node with index 0 got upgraded to 3.0.0 while nodes 1 and 2 were still running 2.4.0.

knn-minimal-test.zip

From my understanding what essentially happens is nodes on different version do not discover each other.

I do not see any errors on old 2.4.0 nodes, but on upgraded 3.0.0 node there is this error after node attempted to start:

[2022-11-03T21:12:23,365][WARN ][r.suppressed             ] [knnBwcCluster-rolling-0] path: /_cluster/health, params: {wait_for_status=yellow, wait_for_nodes=>=3}
org.opensearch.discovery.ClusterManagerNotDiscoveredException: null
	at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAction.java:305) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]

Expected behavior
Cluster shouldn't crush

Plugins
it's min distribution, we're testing k-nn plugin

[andrross]

@martin-gaievski Can you confirm if this is the same issue documented in #5091 ?

[martin-gaievski]

@martin-gaievski Can you confirm if this is the same issue documented in #5091 ?

@andrross our issue, at least symptoms look similar to this one: #5076. There are some major discrepencies: we're running main 3.0.0 snapshot against 2.4.0, and cluster is without security.

[andrross]

See #5090 (comment) for explanation, but when we created the 2.4 release, then 2.5 became the latest minor version in the 2.x line, which then makes the upgrade from 2.4->3.0 unsupported.

[dblock]

So this is by design. Assuming there's a 2.5 build this should no longer be the issue - @martin-gaievski care to track that down?

[martin-gaievski]

I want to make sure I got it right - let's skip current state of unreleased 2.4 and missing 2.5, when these are solved compatibility matrix for rolling upgrades will look like this:

for 2.x:
2.4 (and 2.0, 2.1, 2.2, 2.3) -> 2.5 (2.x became 2.5)
1.3 -> 2.5

for 3.0 (main):
2.5 -> 3.0

will all of these version pairs work, @andrross @dblock? If that's the case I think we're good from k-nn perspective.

[dblock]

I believe this is correct.

[anasalkouz]

Closing this on behalf of #5091