[FEATURE] Encrypt or document plaintext cluster metadata (containing field names)

[salyh]

Various metadata like mappings (containing field names) are part of the cluster state which is saved on disk unencrypted. This needs either to be encrypted (which requires modification in the opensearch core) or documentation that such metadata resides on disk unencrypted.

[kumargu]

@RajatGupta02 is doing an analysis if we will need to do encryption of cluster metadata challenges and implications. And if it can be done as a followup too.

[RajatGupta02]

The problem with encryption of metadata that resides in node-level _state directory is that we can't store the config for index.store.kms.type in _state itself, since to decrypt it we would need fetch the config -> config resides in _state itself, so we will need another place to store this config. Similar situation doesn't happen with our current shard-level encryption approach is due the index level setting of these configs being stored in the _state directory of that index which is not encrypted.
Another problem arises, that any failure to obtain the keys for _state decryption can lead to startup failures, the node won't recognize its local shards forcing full peer recovery for every shard that node should have.