Enable rolling upgrades from default distribution prior to 6.3.0 to default distribution post 6.3.0

[jasontedor]

This is a meta-issue to track the work needed to enable smooth upgrades from the default distribution prior to 6.3.0 to the default distribution post 6.3.0. The sub-tasks are:

• add detection when we are communicating with a >= 6.3 transport client so that we can avoid sending it pieces of the cluster state that it would not be able to deserialize (the problem here is when an OSS transport client connects to a 6.3.0 default cluster) and avoid sending pieces of the cluster state that clients might not be able to understand Introduce client feature tracking #31020 @jasontedor
• same thing for < 6.3 transport clients @ywelsch Fix interoperability with < 6.3 transport clients #30971
• add a node attribute that X-Pack injects into the node attributes so that we can detect when we are connected to a node that understands X-Pack metadata in the cluster state @ywelsch Only allow x-pack metadata if all nodes are ready #30743 Allow rollup job creation only if cluster is x-pack ready #30963
• reject cluster state updates that contain X-Pack metadata when the cluster is not prepared for it (i.e., if there is not already X-Pack metadata in the cluster state and there are nodes in the cluster state than can not understand X-Pack metadata based on the above attribute) @ywelsch Only allow x-pack metadata if all nodes are ready #30743
• fix watcher template that uses custom index setting as an OSS master in a mixed cluster will not be able to add this template and repeatedly fail the PUT template request initiated by the x-pack node, spamming the logs @ywelsch Move watcher-history version setting to _meta field #30832
• ensure that all features (e.g., ML today) that rely on custom cluster state metadata can properly handle that metadata not being present (e.g., during a rolling upgrade scenario) @droberts195 [ML] Don't install empty ML metadata on startup #30751
• add a rolling upgrade test for 3 nodes (we have one for 2 nodes today) to trigger this scenario during testing @nik9000 QA: Switch rolling upgrade to 3 nodes #30728
• fix PersistentTaskParams so that it knows about the versions etc. Assume for example a mixed 6.2 / 6.3 x-pack cluster. If you start a rollup task, this will be put as persistent task into the cluster state. A 6.2 x-pack node cannot deserialize this task. @bleskes Make Persistent Tasks implementations version and feature aware #31045
• add more tests (e.g. that rollups cannot be created in a mixed OSS/X-Pack cluster. Mixed-cluster X-pack tests) @nik9000 QA: Check rollup job creation safety #31036

[elasticmachine]

Pinging @elastic/es-core-infra

[elasticmachine]

Pinging @elastic/es-distributed

[elasticmachine]

Pinging @elastic/ml-core

[bleskes]

Thank for putting this list up. I think we should also deal with the TokenMetaData injected here when called from here.

[bleskes]

I think we should also deal with the TokenMetaData injected here when called from here.

It looks like @ywelsch took care of it in https://github.com/elastic/elasticsearch/pull/30743/files