Do not return metadata customs by default

[jasontedor]

Returning metadata customs by default is dangerous because they could be from a plugin that is not installed on a transport client. This commit modifies the cluster state action so that metadata customs are not returned instead of explicitly asked for. We make the same change on the REST layer, and we require that if metadata customs are requested, so is metadata.

Relates #30731

[elasticmachine]

Pinging @elastic/es-distributed

[s1monw]

LGTM

[ywelsch]

The same deserialization issue exists with cluster state customs (which are still enabled by default). I'm not sure that we need to add another request parameter here, but can just reuse the existing customs parameter, defaulting it to false and have it apply to metadata customs as well.

[jasontedor]

retest this please

[jasontedor]

@ywelsch Are you okay with that being a follow-up?

[jasontedor]

retest this please

[jasontedor]

run gradle build tests

[ywelsch]

With the newly introduced flag, users with the default distribution now have to choose between upgrading to the x-pack client or not showing any custom metadata at all (not even OSS one). The PR is also a breaking change as we will stop showing OSS custom metadata by default (e.g. index graveyard, ingest pipelines, repositories). In more detail, with the current PR:

• 6.2/6.3 clients will by default stop returning custom metadata (even if that custom metadata was from OSS, e.g. index graveyard, ingest pipelines, repositories, ...).
• OSS 6.3 client cannot get clusterstate from default distrib 6.3 cluster if specifying all() on request and not explicitly excluding metadata customs afterwards while the same worked on a default distrib 6.2 cluster.
• OSS 6.3 client cannot get OSS custom metadata from a default distrib 6.3 cluster. When specifying metaCustoms, it has to choose between all or nothing.

I'm doubtful about the use of the new flag and that it truly solves the problem, and my suggestion is therefore for the short term to go with the following approach instead:

• not return x-pack custom metadata through the cluster state API. This will only affect ML metadata. It already has dedicated APIs to access the information, so there's no need to expose this through the cluster state API. If need be, we could add a system property to allow exposing this for duration of the 6.x series.
• For persistent tasks, which has moved to OSS in 6.3, we can return it if streamoutput is on 6.3 at least.

[s1monw]

I'm doubtful about the use of the new flag and that it truly solves the problem, and my suggestion is therefore for the short term to go with the following approach instead:

I am not sure this is useful to anybody and I will push hard to remove this stuff form this API in 7.0 it's been a mistake to expose this to begin with.

not return x-pack custom metadata through the cluster state API. This will only affect ML metadata. It already has dedicated APIs to access the information, so there's no need to expose this through the cluster state API. If need be, we could add a system property to allow exposing this for duration of the 6.x series.

+1 we should never return plugin metadata here at any time

For persistent tasks, which has moved to OSS in 6.3, we can return it if streamoutput is on 6.3 at least.

this sounds good to me too

[jasontedor]

Superseded by #31020