[Remote Store] Override remote store and replication strategy settings during snapshot restore.

[harishbhakuni]

Is your feature request related to a problem? Please describe

To restore snapshot from remote store enabled cluster to non remote store enabled cluster or to another remote store enabled cluster, currently user need to explicitly override the remote store related settings (remote_store.repository, remote_translog.repository, remote_store.enabled) by providing them as part of request fields in restore api.

Since remote store is more of a cluster level feature and replication strategy will always be SEGMENT with remote store feature enabled. user should not have to provide override for these index settings. during restore, these settings can be pulled and applied from node attributes.

Describe the solution you'd like

As part of this issue, proposing the following change in behavior:

• Override the remote store related settings with the values present in node attributes during restore since remote store related settings are more of cluster level settings than index level settings.
• For replication strategy related settings, if remote store feature is enabled in the cluster. override the setting to SEGMENT. otherwise, if cluster.force.index.replication_type settings is present in cluster, override the replication strategy with cluster default. If this setting is not present, let the index come up with different replication_type then the cluster default since user anyway allows indices with different replication strategy in same cluster.

All the restore scenarios will work as well with this solution:

Snapshot Restore Scenario	Works/don't work	Notes
Same Version Remote Store → Same Version Non Remote Store	will work	remote store settings will be ignored during restore, replication strategy will be overriden with cluster default if `cluster.force.index.replication_type is present, else indices will be restored as segrep.
Same Version Non Remote Store → Same Version Remote Store	will work	remote store and replication strategy settings will be applied from the node attributes.
2.12 Version Remote Store → 2.10(<=) Version Non Remote Store	will not work	snapshot restoration to an older version cluster is not allowed.
(<=)2.10 Version → 2.12 Version Non Remote Store	will work
2.12 Version Non Remote Store → 2.10(<=) Version	will not work	snapshot restoration to an older version cluster is not allowed.
(<=)2.10 Version → 2.12 Version Remote Store	will work	remote store and replication strategy settings will be applied from the node attributes.

Related component

Storage:Snapshots

Describe alternatives you've considered

No response

Additional context

[TODO]: add an example after restoring without overriding settings, add an example after restoring from snapshot by overriding settings

[harishbhakuni]

tagging @sohami @mch2 @sachinpkale @gbbafna @Bukhtawar for feedback or suggestions.

[sohami]

Makes sense to me. The only thing I want to double confirm is what all combinations of Remote Store + Replication type is possible to be deployed by a user. Do we only allow SegRep indices with remote store enabled ? Or we also allow mix of doc rep + seg rep based indices where only later uses the remote store capabilities ? I think we don't allow the mix but wanted to confirm on this.

[harishbhakuni]

@sohami AFAIK, for remote store enabled clusters, user can only have seg-rep indices. for non remote store enabled clusters, user can have a mixture of doc-rep and seg-rep indices if they want.