[BUG] Cluster settings update deadlock with archived settings and read only cluster blocks

[Bukhtawar]

Describe the bug
If the cluster has both a read-only settings and also a un-recognised settings that gets archived. The cluster settings update gets into a deadlock as the archived setting blocks any new settings update and read only settings prevents archived settings to removed. Trying to set both of them together fails as read only block only allows a single setting which is to remove the read only settings.

OpenSearch/server/src/main/java/org/opensearch/action/admin/cluster/settings/TransportClusterUpdateSettingsAction.java

Lines 101 to 113 in b5cc9c3

	protected ClusterBlockException checkBlock(ClusterUpdateSettingsRequest request, ClusterState state) {
	// allow for dedicated changes to the metadata blocks, so we don't block those to allow to "re-enable" it
	if (request.transientSettings().size() + request.persistentSettings().size() == 1) {
	// only one setting
	if (Metadata.SETTING_READ_ONLY_SETTING.exists(request.persistentSettings())
	|| Metadata.SETTING_READ_ONLY_SETTING.exists(request.transientSettings())
	|| Metadata.SETTING_READ_ONLY_ALLOW_DELETE_SETTING.exists(request.transientSettings())
	|| Metadata.SETTING_READ_ONLY_ALLOW_DELETE_SETTING.exists(request.persistentSettings())) {
	// one of the settings above as the only setting in the request means - resetting the block!
	return null;
	}
	}
	return state.blocks().globalBlockedException(ClusterBlockLevel.METADATA_WRITE);

Current Settings

curl localhost:9200/_cluster/settings?pretty
{
	"persistent": {
		"archived": {
			"opendistro": {
				"index_state_management": {
					"template_migration": {
						"control": "-1"
					}
				}
			}
		},

		"blocks": {
			"create_index": "true",
			"read_only": "true"
		}
	},
	"transient": {}
}

1. Remove read-only blocks alone

curl -XPUT "localhost:9200/_cluster/settings?pretty" --data '{ "persistent": {"cluster.blocks.read_only": false} }' -H 'Content-Type:application/json'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "remote_transport_exception",
        "reason" : "[58ac970a3749b3fe2406cab8b56433dc][172.16.40.199:9300][cluster:admin/settings/update]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "unknown setting [archived.opendistro.index_state_management.template_migration.control] did you mean [opendistro.index_state_management.template_migration.control]?"
  },
  "status" : 400
}

2. Remove only archived settings

curl -XPUT "localhost:9200/_cluster/settings?pretty" --data '{ "persistent": {"opendistro.index_state_management.template_migration.control": null} }' -H 'Content-Type:application/json' 
{
  "error" : {
    "root_cause" : [
      {
        "type" : "cluster_block_exception",
        "reason" : "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
      }
    ],
    "type" : "cluster_block_exception",
    "reason" : "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
  },
  "status" : 403
}

3. Remove both archived settings and read only block together

curl -XPUT "localhost:9200/_cluster/settings?pretty" --data '{ "persistent": {"cluster.blocks.read_only": false, "opendistro.index_state_management.template_migration.control": null} }' -
H 'Content-Type:application/json'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "cluster_block_exception",
        "reason" : "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
      }
    ],
    "type" : "cluster_block_exception",
    "reason" : "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
  },
  "status" : 403
}

[nknize]

Another related issue? #8714

Looks like this has stalled. Marking as such to bump the priority if this keeps creating a bad experience.

[rajiv-kv]

Resolving as it is fixed here :- #18885