[BUG] Configuring cluster.allocator.existing_shards_allocator.batch_enabled via /_cluster/settings returns a 400 error

[drewmiranda-gl]

Describe the bug

OpenSearch 2.14 introduced a new feature/setting: cluster.allocator.existing_shards_allocator.batch_enabled.

This setting is documented as being able be set dynamically:

https://opensearch.org/docs/latest/install-and-configure/configuring-opensearch/cluster-settings/#cluster-level-routing-and-allocation-settings

All settings in this list are dynamic:

However, when attempting to set using /_cluster/settings API endpoint, an error is returned:

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "persistent setting [cluster.allocator.existing_shards_allocator.batch_enabled], not dynamically updateable"
            }
        ],
        "type": "illegal_argument_exception",
        "reason": "persistent setting [cluster.allocator.existing_shards_allocator.batch_enabled], not dynamically updateable"
    },
    "status": 400
}

Payload used in PUT request:

{
    "persistent": {
        "cluster.allocator.existing_shards_allocator.batch_enabled": true
    }
}

Related component

Other

To Reproduce

1. use OpenSearch 2.14 or 2.15
2. Send a PUT request to /_cluster/settings

{
    "persistent": {
        "cluster.allocator.existing_shards_allocator.batch_enabled": true
    }
}

Expected behavior

Expectation is that if the documentation says this is a dynamic setting, it can be set dynamically. If this must be set via opensearch.yml (which does work), this should be communicated/documented.

Additional Details

Plugins
Stock/out of box

Host/Environment (please complete the following information):

• OS: Ubuntu Server 22.04 LTS
• OpenSearch Version: 2.15.0

Additional context

First learned about this from the following WARN in opensearch's log:

Falling back to single shard assignment since batch mode disable or multiple custom allocators set

[peternied]

[Triage - attendees 1 2 3]
@drewmiranda-gl Thanks for creating this issue, what would you think about making a pull request to address this issue or make a test case that reproduces this issue?

[jed326]

From code comment it says this setting is experimental for now (although it's not behind a feature flag or anything).

OpenSearch/server/src/main/java/org/opensearch/cluster/routing/allocation/ExistingShardsAllocator.java

Lines 64 to 82 in a2cef8f

	/**
	* Boolean setting to enable/disable batch allocation of unassigned shards already existing on disk.
	* This will allow sending all Unassigned Shards to the ExistingShard Allocator to make decision to allocate
	* in one or more go.
	*
	* Enable this setting if your ExistingShardAllocator is implementing the
	* {@link ExistingShardsAllocator#allocateAllUnassignedShards(RoutingAllocation, boolean)} method.
	* The default implementation of this method is not optimized and assigns shards one by one.
	*
	* If no plugin overrides {@link ExistingShardsAllocator} then default implementation will be use for it , i.e,
	* {@link ShardsBatchGatewayAllocator}.
	*
	* This setting is experimental at this point.
	*/
	Setting<Boolean> EXISTING_SHARDS_ALLOCATOR_BATCH_MODE = Setting.boolSetting(
	"cluster.allocator.existing_shards_allocator.batch_enabled",
	false,
	Setting.Property.NodeScope
	);

FYI @Gaurav614

[Gaurav614]

Thanks for pointing out, I dont think team has planned to support dynamic setting for now.
We need to update the documentation here for now.

Feel free to raise the Proposal for making it a dynamic setting. It involves various edge cases while assigning shards during node drops(or via GatewayAllocator), like how to switch dynamically to batch mode when shards are in assignment mode in batches

CC: @shwetathareja

[shwetathareja]

@drewmiranda-gl / @jed326 : Thanks for pointing out. This setting is not supposed to be dynamic. We should fix the documentation to reflect the same. @rwali-aws @rajiv-kv @sumitasr

[sumitasr]

Looking into it.

[mark-VIII]

How safe is it to enable this "experimental" setting? The documentation seems to indicate this new allocation mode is better and does not describe it as "experimental". Given the WARN log messages, should most users be setting this to true and the reason the default is false is to not change behaviour on upgrade? I think it'd be worth adding some guidance somewhere on upgrade considerations for new settings such as this. Even if something is added to release notes, that would be a good start.