[Feature Request] New index setting/property for toggling a remote store index as "writable warm”

[kotwanikunal]

Is your feature request related to a problem? Please describe

The feature described below is related to #11703

Describe the solution you'd like

Background

Coming in from #11703 - A new mechanism is needed to configure the storage properties for an index, which can be used to modify the type of underlying node storage used by an index.

This mechanism should trigger a change on the index properties and should also provide the user with a mechanism to track the progress or cancel the operation.

Goals

• The storage type of an index can be changed by the client using an API
• The transition should be trackable by the client using an API
• The transition process should be cancelable using an API

Proposed Solution

How can the user create an index with the new properties?

The process to create an index with the new properties would utilize the existing store attribute on index settings.

Simplified approach
This approach will auto configure properties of the store with predefined configuration for the corresponding tier.
API: **PUT** /<index_name>?tier=WARM
Body:

{
   ... index settings
}

Expert user approach:
This approach can be used by expert users to define individual values for the corresponding tier properties on the store.
API: **PUT** /<index_name>/
Body:

{
    "index": {
        "store": {
            "type": "hybridfs, niofs, mmapfs, warmfs",
            // can be extended for store specific properties as follows
            // for example purposes only
            "warmfs": {
                "max_cache_usage": "ValueInPercentOrSize"
            }
        }
    }
}

How can the user know the current tier attributes of an index?
API: **GET** /<index_name>/_settings

For a warm index:
{
    "index": {
        "store": {
            "type": "hybridfs, niofs, mmapfs, warmfs",
            // for example purposes only
            "warmfs": {
                "max_cache_usage": "ValueInPercentOrSize"
            }
        }
    }
}

On similar lines, we can also list indices with their current state as follows -

Request:

**GET** _cat/indices?h=index,health,uuid,tier

Sample Response: includes an extra tier parameter to show the tier of the index

health status index            uuid                     tier    
yellow open   my-index-000001  u8FNjxh8Rfy_awN11oDKYQ   HOT     
green  open   my-index-000002  nYFWZEO7TUiOjLQXBaYJpA   WARM

How can the user migrate an existing index to a different tier?

The client will use a new, custom API to perform the migration as follows -

API: POST /<indexNameOrPattern>/_tier

{
     "type" : "HOT"/"WARM"
}

OR

(Preferred) API: POST /<indexNameOrPattern>/_tier/_warm

{
 "" : ""  // supporting body to make it extensible for future use-cases
}

How can the user track migrations for indexes?

This API would show the on-going or failed migrations across different tiers.

Request:

1. GET /_tiering/_status?source=hot&target=warm
2. GET /_tiering/{index}/_status?verbose=<true/false

OR
Preferred:

1. GET /<indexNameOrPattern>/_tier?source=hot&target=warm
2. GET /<indexNameOrPattern>/_tier?verbose=true/false

Sample Response:

{
    "tiering_status" : {
        "id": "tiering_id",
        "index" : "test1"
        "source": "hot",
        "destination": "warm",
        "progress" : "50%",
        "failed" : false
    }
}

with verbose flag
{
    "tiering_status" : {
        "id": "tiering_id",
        "index" : "test1"
        "source": "hot",
        "destination": "warm",
        "progress" : "50%",
        ...<shard_stats>...,
        ...<transfer_stats>...,
        "cancellable" : true,
        "cancelled" : false,
        "reason" : "reason of failure, if any"
    }
}

How can the user cancel a migration?

The user can utilize the tiering APIs to cancel the migration by providing the original state of the index.
If there is a current hot to warm migration going on, in order to cancel the migration, the user can trigger a warm to hot migration and return to the original state.

API: POST /<indexNameOrPattern>/``_tier

{
     "type" : "HOT"/"WARM"
}

OR

(Preferred) API: POST /<indexNameOrPattern>/_tier/_hot

Contributors: @kotwanikunal, @neetikasinghal

Related component

Search:Remote Search

[neetikasinghal]

thanks @kotwanikunal, also posting some of the alternative options that can be considered:

Alternatives considered

Index creation and migration

Utilize a new dynamic property on index settings for tiering which can be set at index creation or be updated to trigger migration.
The new index dynamic setting key can be index.access.type with values as HOT / WARM

PUT /index/_settings 
{
    "index" : {
        "store": {
            "access" : {
                "tier" : "HOT/WARM"
            }
        }
    }
}

Tracking the migration
An alternative to tracking is specifying the current migration tier of an index as follows -

Request:

1. GET /_tiering/_status?type=hot_to_warm
2. GET /_tiering/{index}/_status?verbose=<true/false>

This would require exposing transitionary statuses like hot_to_warm to the end user which can be prohibitive when considering extensibility and addition of new tiers.

Cancelation

An alternative cancellation API can be added for convenience which will be useful mainly for admin operations so that a user can cancel/remove the tiering requests which are pending completion.

Request:
API: POST /<index_name>/_tiering/<tiering_id>/_cancel (in case we decide to expose a new id for tiering to the cx)
API: POST /<index_name>/_tiering/_cancel

[neetikasinghal]

We would love to get feedback from the community for this: @andrross @sohami @reta @mch2 @shwetathareja @Bukhtawar @rohin

[rohin]

Makes sene, intent should always be to take an extensible approach.

[Bukhtawar]

With _tiering you have multiple options, like listing cluster wide migration status where you don't need to pass index pattern, while when you start the API with index pattern you won't be able to get a top-level view(unless you want * for the cluster wide view), which looks hacky. The thing to note here is the security considerations with these API. The index pattern based API structure offers more fine-grained security semantics, meaning if only the user has permission on certain indices will the user be able to view the status. So we need to think if tiering should be an administrative action or individual users should have that control. IMO it should be an administrative action.

I would go with the _tiering or better _tier/<optional-index-pattern>/_<action> API if the common pattern is executing API cluster wide and index-pattern/_tier if the common use case is operating tiers at index levels.

[ankitkala]

Thanks for the proposal @kotwanikunal @neetikasinghal

• +1 on @Bukhtawar's point for supporting index patterns.
• Should we evaluate pros & cons for using index.store.type v/s index.store.access.tier? Reusing index.store.type does makes sense semantically, but we'll be overloading the setting beyond its intended purpose. It'd to good to have more feedback around this.
• Regarding max_cache_usage, we haven't accounted for index/shard level usage. The FileCache as of now is only expected to have node level view.

[neetikasinghal]

@Bukhtawar thanks for your feedback.

while when you start the API with index pattern you won't be able to get a top-level view(unless you want * for the cluster wide view), which looks hacky

_cat/indices could also serve the use-case of showing the top-level view of the indices. _cat/indices, along with the tier of the current index can also show the MIGRATING status similar to how we see RELOCATING status in the _cat/shards API.