ILM policy with empty "warm" phase still enforces read only indices

[dakrone]

Consider the following policy:

PUT /_ilm/delete-only-policy
{
  "policy": {
    "phases": {
      "delete": {
        "after": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Where the intent is to delete the index after 30 days. Because of the injected "read-only" step in the warm phase with an after time of 0 seconds, the index is immediately marked as read only. So this returns the following:

PUT /_cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": "10s"
  }
}

PUT /bar
{
  "settings": {
    "index.number_of_shards": 1,
    "index.number_of_replicas": 0,
    "index.lifecycle.name": "delete-only-policy"
  }
}

POST /bar/1
{
  "baz": "eggplant"
}

{
  "error" : {
    "root_cause" : [
      {
        "type" : "cluster_block_exception",
        "reason" : "blocked by: [FORBIDDEN/8/index write (api)];"
      }
    ],
    "type" : "cluster_block_exception",
    "reason" : "blocked by: [FORBIDDEN/8/index write (api)];"
  },
  "status" : 403
}

[elasticmachine]

Pinging @elastic/es-core-infra

[dakrone]

In my opinion, marking an index read-only should be optionally opt-in and user configurable in the warm or cold phases, instead of hardcoded to occur at the start of the warm phase.

[talevy]

Although this will prevent optimal behavior of some actions, I think having this be explicit is good. As this bug demonstrates, by implicitly executing this, a user can be comparatively hurt by not knowing it is there.

I believe this is also an issue for empty policies

[dakrone]

We discussed this and decided with the following solution:

Inject a read-only step as the first step in the warm phase iff the phase contains a force merge or a shrink action. Otherwise the read-only step will not be added.

[colings86]

Boaz and I discussed this and we are leaning towards adding read only step to the force merge and shrink actions rather than having an infrastructure change that tries to add it a read only step as the first step in the warm phase IF the warm phase contains forcemerge or shrink.

We should discuss this further on Friday

[jasontedor]

That change sounds good to me if you want to start moving forward with it.

[AthenaEryma]

Fixed by #33907