[BUG] _list/shards API failing with index_closed_exception

[gargharsh3134]

Describe the bug

Default _list/shards API call (curl localhost:9200/_list/shards) fails with index_closed_exception if an Opensearch cluster has one or more closed indices.

The issue occurs due to the way pagination is implemented in the _list APIs. After processing the clusterState response, concrete list of indices is passed to other TransportActions down the line. For _list/shards API, all the concrete indices processed and fetched from PaginationStrategy are directly supplied to TransportIndicesStatsAction.
By default, TransportIndicesStatsAction uses IndicesOptions.strictExpandOpenAndForbidClosed() which results in index_closed_exception if any of of the concrete index is closed.

The reason this same thing doesn't fail in _list/indices is due to the fact that, a TransportGetSettingsAction call is first made with whatever indices are specified in the URL (not the concrete indices from the strategy) [Code Ref] and all the other TransportAction calls post that are made with IndicesOptions.lenientExpandHidden(); [Code Ref].

Fix:

Filter out closed indices as part of ShardPaginationStrategy (accept indicesOptions as part of strategy) and let TransportIndicesStatsAction process only the open indices.

Related component

Cluster Manager

To Reproduce

1. Have one or more closed indices in a cluster.
2. Make _list/shards (curl localhost:9200/_list/shards) call, which would fail with index_closed_exception.

Expected behavior

_cat/shards API, displays the closed shards without stats for an unfiltered query. The _list/shards API similarly, should not fail with exception and instead just filter out the response.

Additional Details

No response

[dblock]

[Catch All Triage - 1, 2, 3]