Skip to content

Add cluster.blocks.read.auto_release setting#20633

Open
drtootsie wants to merge 2 commits intoopensearch-project:mainfrom
drtootsie:add-read-block-auto-release-setting
Open

Add cluster.blocks.read.auto_release setting#20633
drtootsie wants to merge 2 commits intoopensearch-project:mainfrom
drtootsie:add-read-block-auto-release-setting

Conversation

@drtootsie
Copy link
Copy Markdown

Addresses #20539

Problem

DiskThresholdMonitor unconditionally auto-releases index.blocks.read on all indices, even when manually set by users. This setting was added to support warm node file cache threshold protection, but the implementation assumes index.blocks.read is only used by the system.

Unlike cluster.blocks.create_index which has cluster.blocks.create_index.auto_release to control this behavior, there is no equivalent setting for index.blocks.read.

Solution

Add cluster.blocks.read.auto_release setting to control read block auto-release behavior, following the same pattern as cluster.blocks.create_index.auto_release.

Changes

+ public static final Setting<Boolean> CLUSTER_READ_BLOCK_AUTO_RELEASE = Setting.boolSetting(
+     "cluster.blocks.read.auto_release",
+     true,
+     Setting.Property.Dynamic,
+     Setting.Property.NodeScope
+ );

+ private volatile boolean readBlockAutoReleaseEnabled;

+ public boolean isReadBlockAutoReleaseEnabled() {
+     return readBlockAutoReleaseEnabled;
+ }

In DiskThresholdMonitor.handleReadBlocks():

+ if (diskThresholdSettings.isReadBlockAutoReleaseEnabled()) {
      indicesToReleaseReadBlock = StreamSupport.stream(...)
          .filter(index -> indicesToBlockRead.contains(index) == false)
          .filter(index -> state.getBlocks().hasIndexBlock(index, IndexMetadata.INDEX_READ_BLOCK))
          .collect(Collectors.toSet());
+ } else {
+     indicesToReleaseReadBlock = Set.of();
+ }

Usage

Users can disable auto-release to preserve manually-set read blocks:

PUT /_cluster/settings
{
  "persistent": {
    "cluster.blocks.read.auto_release": false
  }
}

Testing

  • Set index.blocks.read: true manually on an index
  • With default setting (true): block is auto-released
  • With cluster.blocks.read.auto_release: false: block persists
  • Warm node file cache protection still works (sets blocks that get auto-released)

Impact

  • Default behavior unchanged (auto-release enabled)
  • Users can now control read block auto-release
  • Manually-set read blocks can persist until explicitly removed

@drtootsie drtootsie requested a review from a team as a code owner February 15, 2026 20:15
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Feb 15, 2026

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR adds conditional auto-release functionality for read-blocks in disk threshold management. It introduces a new dynamic cluster setting that gates automatic read-block release logic, allowing operators to control whether blocked indices are automatically unblocked when disk conditions improve.

Changes

Cohort / File(s) Summary
Read-Block Auto-Release Configuration
server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdSettings.java
Introduces new cluster setting CLUSTER_READ_BLOCK_AUTO_RELEASE (boolean, dynamic, node-scoped) with corresponding volatile field, getter, and settings update listener for dynamic updates.
Read-Block Release Logic
server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdMonitor.java
Adds conditional branching in handleReadBlocks() that computes indicesToReleaseReadBlock based on the isReadBlockAutoReleaseEnabled() flag; when disabled, release set is empty, otherwise computed as currently blocked indices not in the block-list that retain INDEX_READ_BLOCK.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~18 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 1
❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title directly and concisely summarizes the main change: adding a new cluster setting to control read block auto-release behavior.
Description check ✅ Passed The description includes all required sections: Problem statement, Solution, Changes with code examples, Usage instructions, and Testing details. It comprehensively addresses the issue and follows the template structure.
Merge Conflict Detection ✅ Passed ✅ No merge conflicts detected when merging into main

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdMonitor.java (1)

496-527: ⚠️ Potential issue | 🟠 Major

Apply indicesNotToAutoRelease filter to indicesToReleaseReadBlock in handleReadBlocks.

The read-only block auto-release in handleReadOnlyBlocks (line 456) filters out indicesNotToAutoRelease to prevent releasing blocks on flood-stage, high-watermark, or nodes with missing usage data. However, handleReadBlocks does not apply this same filter to indicesToReleaseReadBlock, allowing read blocks to be auto-released on indices that should remain blocked. This creates an asymmetry where read-only blocks are protected but read blocks are not.

Pass indicesNotToAutoRelease to handleReadBlocks and apply the same filter to indicesToReleaseReadBlock (around line 499-506) to match the pattern in handleReadOnlyBlocks.

📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8c244c0 and 10ce8c8.

📒 Files selected for processing (2)
  • server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdMonitor.java
  • server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdSettings.java
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2026-01-13T17:40:27.167Z
Learnt from: reta
Repo: opensearch-project/OpenSearch PR: 20411
File: server/src/main/java/org/opensearch/index/codec/CodecService.java:112-133
Timestamp: 2026-01-13T17:40:27.167Z
Learning: Avoid capturing or evaluating a supplier (e.g., this::defaultCodec) upfront when passing it to a registry during object construction. If registries may replace defaults during iteration (as in EnginePlugin.getAdditionalCodecs), pass the supplier itself and only resolve it at use time. This ensures dynamic behavior is preserved during initialization and prevents premature binding of defaults in codecs/registry setup. This pattern should apply to similar initialization paths in Java server code where registries may mutate defaults during construction.

Applied to files:

  • server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdMonitor.java
  • server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdSettings.java
🔇 Additional comments (2)
server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdSettings.java (1)

117-122: Clean, consistent addition mirroring the existing createIndexBlockAutoReleaseEnabled pattern.

Setting definition, volatile field, constructor init, update consumer registration, setter, and getter all follow the established convention. LGTM.

Also applies to: 132-132, 163-163, 175-175, 377-379, 439-441

server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdMonitor.java (1)

497-509: Correct conditional gating of read-block auto-release.

The toggle cleanly wraps the existing release computation and falls back to an immutable empty set when disabled. This is consistent with how isCreateIndexBlockAutoReleaseEnabled() gates the create-index block release in handleClusterCreateIndexBlocks.

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 10ce8c8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Addresses opensearch-project#20539

DiskThresholdMonitor unconditionally auto-releases index.blocks.read
on all indices, even when manually set by users. This was designed for
warm node file cache threshold protection but affects all nodes.

Changes:
- Add CLUSTER_READ_BLOCK_AUTO_RELEASE setting (default: true)
- Check isReadBlockAutoReleaseEnabled() before auto-releasing read blocks
- Follows same pattern as CLUSTER_CREATE_INDEX_BLOCK_AUTO_RELEASE

Users can now disable auto-release:
PUT /_cluster/settings
{
  "persistent": {
    "cluster.blocks.read.auto_release": false
  }
}

This allows manually-set read blocks to persist until explicitly removed.

Signed-off-by: Pepper Pancoast <ppancoast@ntst.com>
@drtootsie drtootsie force-pushed the add-read-block-auto-release-setting branch from 10ce8c8 to a493f8b Compare February 15, 2026 20:27
@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for a493f8b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 1, 2026

PR Reviewer Guide 🔍

(Review updated until commit 26981d2)

Here are some key observations to aid the review process:

🧪 PR contains tests
🔒 No security concerns identified
✅ No TODO sections
🔀 No multiple PR themes
⚡ Recommended focus areas for review

Incomplete Test Coverage

The test testReadBlockAutoReleaseDisabled only verifies that indicesToReleaseReadBlock remains null when auto-release is disabled. It does not verify the complementary case: that when cluster.blocks.read.auto_release is true (default), the read block IS auto-released. Additionally, there is no test verifying that blocks are still correctly applied (indicesToBlockRead is set) when auto-release is disabled, to ensure the blocking side of the feature still works.

public void testReadBlockAutoReleaseDisabled() {
    AtomicReference<Set<String>> indicesToBlockRead = new AtomicReference<>();
    AtomicReference<Set<String>> indicesToReleaseReadBlock = new AtomicReference<>();

    AllocationService allocation = createAllocationService(
        Settings.builder().put("cluster.routing.allocation.node_concurrent_recoveries", 10).build()
    );

    IndexMetadata indexMetadata = IndexMetadata.builder("test_read_block")
        .settings(settings(Version.CURRENT).put(IndexMetadata.INDEX_BLOCKS_READ_SETTING.getKey(), true))
        .numberOfShards(1)
        .numberOfReplicas(0)
        .build();

    Metadata metadata = Metadata.builder().put(indexMetadata, false).build();
    RoutingTable routingTable = RoutingTable.builder().addAsNew(metadata.index("test_read_block")).build();

    DiscoveryNode warmNode = newNode("warm_node", Collections.singleton(DiscoveryNodeRole.WARM_ROLE));

    final ClusterState clusterState = applyStartedShardsUntilNoChange(
        ClusterState.builder(ClusterName.CLUSTER_NAME_SETTING.getDefault(Settings.EMPTY))
            .metadata(metadata)
            .routingTable(routingTable)
            .nodes(DiscoveryNodes.builder().add(warmNode))
            .blocks(ClusterBlocks.builder().addBlocks(indexMetadata).build())
            .build(),
        allocation
    );

    assertTrue(clusterState.blocks().indexBlocked(ClusterBlockLevel.READ, "test_read_block"));

    Settings settings = Settings.builder().put("cluster.blocks.read.auto_release", false).build();

    DiskThresholdMonitor monitor = new DiskThresholdMonitor(
        settings,
        () -> clusterState,
        new ClusterSettings(settings, ClusterSettings.BUILT_IN_CLUSTER_SETTINGS),
        null,
        () -> 0L,
        (reason, priority, listener) -> listener.onResponse(clusterState),
        () -> 2.0
    ) {
        @Override
        protected void updateIndicesReadOnly(Set<String> indicesToUpdate, ActionListener<Void> listener, boolean readOnly) {
            listener.onResponse(null);
        }

        @Override
        protected void updateIndicesReadBlock(Set<String> indicesToUpdate, ActionListener<Void> listener, boolean readBlock) {
            if (readBlock) {
                indicesToBlockRead.set(indicesToUpdate);
            } else {
                indicesToReleaseReadBlock.set(indicesToUpdate);
            }
            listener.onResponse(null);
        }

        @Override
        protected void setIndexCreateBlock(ActionListener<Void> listener, boolean indexCreateBlock) {
            listener.onResponse(null);
        }
    };

    // Node is healthy (free space 40 > 30 threshold)
    Map<String, DiskUsage> builder = new HashMap<>();
    builder.put("warm_node", new DiskUsage("warm_node", "warm_node", "/foo/bar", 200, 40));

    monitor.onNewInfo(clusterInfo(builder));

    // Since auto-release is disabled, indicesToReleaseReadBlock should remain null even though node is healthy
    assertNull(indicesToReleaseReadBlock.get());
}
Warm Node Interaction

When readBlockAutoReleaseEnabled is false, indicesToReleaseReadBlock is set to an empty set, which means blocks set by the warm node file cache threshold protection will also never be auto-released. The PR description claims "Warm node file cache protection still works (sets blocks that get auto-released)" but this is incorrect — if a user sets cluster.blocks.read.auto_release: false, warm-node-triggered read blocks will also never be released automatically. This behavioral impact should be documented and potentially guarded.

if (diskThresholdSettings.isReadBlockAutoReleaseEnabled()) {
    indicesToReleaseReadBlock = StreamSupport.stream(
        Spliterators.spliterator(state.routingTable().indicesRouting().entrySet(), 0),
        false
    )
        .map(Map.Entry::getKey)
        .filter(index -> indicesToBlockRead.contains(index) == false)
        .filter(index -> state.getBlocks().hasIndexBlock(index, IndexMetadata.INDEX_READ_BLOCK))
        .collect(Collectors.toSet());
} else {
    indicesToReleaseReadBlock = Set.of();
}

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 1, 2026

PR Code Suggestions ✨

Latest suggestions up to 26981d2

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
General
Strengthen test assertions for disabled auto-release

The test only verifies that indicesToReleaseReadBlock is null (i.e.,
updateIndicesReadBlock was never called with readBlock=false), but it doesn't assert
that the existing read block on the index is still present in the cluster state.
Additionally, it would be valuable to also assert that indicesToBlockRead is
null/empty to confirm no spurious block operations occurred.

server/src/test/java/org/opensearch/cluster/routing/allocation/DiskThresholdMonitorTests.java [931-938]

 // Node is healthy (free space 40 > 30 threshold)
 Map<String, DiskUsage> builder = new HashMap<>();
 builder.put("warm_node", new DiskUsage("warm_node", "warm_node", "/foo/bar", 200, 40));
 
 monitor.onNewInfo(clusterInfo(builder));
 
 // Since auto-release is disabled, indicesToReleaseReadBlock should remain null even though node is healthy
 assertNull(indicesToReleaseReadBlock.get());
+// No new read blocks should have been applied either
+assertNull(indicesToBlockRead.get());
+// The existing read block should still be present on the index
+assertTrue(clusterState.blocks().indexBlocked(ClusterBlockLevel.READ, "test_read_block"));
Suggestion importance[1-10]: 4

__

Why: Adding assertions for indicesToBlockRead being null and verifying the read block still exists on the cluster state would make the test more comprehensive. However, the clusterState is a local variable and the monitor doesn't modify it directly, so the last assertion is trivially true and adds limited value.

Low
Add logging when auto-release is disabled

When isReadBlockAutoReleaseEnabled() returns false, existing read blocks on indices
that are no longer above the threshold will never be released, even if an
administrator manually re-enables the setting later and the indices are still
blocked. Consider logging a debug/info message when auto-release is disabled and
there are indices that would otherwise have been released, to aid in operational
visibility.

server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdMonitor.java [510-512]

 } else {
     indicesToReleaseReadBlock = Set.of();
+    if (logger.isDebugEnabled()) {
+        long blockedCount = StreamSupport.stream(
+            Spliterators.spliterator(state.routingTable().indicesRouting().entrySet(), 0), false)
+            .map(Map.Entry::getKey)
+            .filter(index -> indicesToBlockRead.contains(index) == false)
+            .filter(index -> state.getBlocks().hasIndexBlock(index, IndexMetadata.INDEX_READ_BLOCK))
+            .count();
+        if (blockedCount > 0) {
+            logger.debug("[{}] indices have read blocks that would be released, but auto-release is disabled", blockedCount);
+        }
+    }
 }
Suggestion importance[1-10]: 2

__

Why: Adding debug logging for operational visibility is a minor improvement, but the suggested implementation re-runs the entire stream computation just for logging purposes, which is inefficient. This is a low-impact suggestion that adds complexity without significant benefit.

Low

Previous suggestions

Suggestions up to commit 6ad6837
CategorySuggestion                                                                                                                                    Impact
General
Log warning when auto-release is disabled

When isReadBlockAutoReleaseEnabled() returns false, existing read blocks on indices
will never be released, even if disk usage has recovered. This could leave indices
permanently blocked if the setting is toggled off after blocks were applied.
Consider logging a warning or providing documentation that disabling auto-release
means manual intervention is required to remove read blocks.

server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdMonitor.java [510-512]

 } else {
     indicesToReleaseReadBlock = Set.of();
+    logger.debug(
+        "Auto-release of read blocks is disabled; existing read blocks on indices will not be automatically removed"
+    );
 }
Suggestion importance[1-10]: 3

__

Why: Adding a debug log when isReadBlockAutoReleaseEnabled() returns false is a minor improvement for observability. However, it's a low-impact change that only adds a debug message and doesn't address any correctness issue.

Low
Possible issue
Register new setting in cluster settings registry

The new CLUSTER_READ_BLOCK_AUTO_RELEASE setting needs to be registered in the
cluster settings registry. Without registration, the setting won't be recognized and
will cause an exception when set. Check where
CLUSTER_CREATE_INDEX_BLOCK_AUTO_RELEASE is registered (likely in a settings module
or plugin class) and add CLUSTER_READ_BLOCK_AUTO_RELEASE alongside it.

server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdSettings.java [121-126]

 public static final Setting<Boolean> CLUSTER_READ_BLOCK_AUTO_RELEASE = Setting.boolSetting(
     "cluster.blocks.read.auto_release",
     true,
     Setting.Property.Dynamic,
     Setting.Property.NodeScope
 );
+// Ensure CLUSTER_READ_BLOCK_AUTO_RELEASE is added to the list of registered cluster settings
+// in the same location where CLUSTER_CREATE_INDEX_BLOCK_AUTO_RELEASE is registered.
Suggestion importance[1-10]: 2

__

Why: The suggestion asks to verify that CLUSTER_READ_BLOCK_AUTO_RELEASE is registered in the cluster settings registry, but the improved_code only adds a comment rather than actual code. This is a "verify/ensure" type suggestion, and the actual registration likely happens in an external file not shown in the diff. The suggestion score is low as it doesn't provide actionable code changes.

Low
Suggestions up to commit 1cb260f
CategorySuggestion                                                                                                                                    Impact
General
Add logging when auto-release is disabled

When isReadBlockAutoReleaseEnabled() returns false, existing read blocks on indices
will never be released, even if disk usage has recovered. This could leave indices
permanently blocked if the setting is toggled from true to false and back. Consider
logging a warning or providing a mechanism to manually release blocks when
auto-release is disabled.

server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdMonitor.java [510-512]

 } else {
+    logger.debug(
+        "Auto-release of read blocks is disabled; skipping release of read blocks on indices"
+    );
     indicesToReleaseReadBlock = Set.of();
 }
Suggestion importance[1-10]: 3

__

Why: Adding a debug log when auto-release is disabled improves observability, but this is a minor enhancement. The concern about permanent blocking when toggling the setting is valid but the suggested fix (just logging) doesn't address the underlying issue.

Low
Possible issue
Register new setting in cluster settings registry

The new CLUSTER_READ_BLOCK_AUTO_RELEASE setting needs to be registered in the
cluster settings registry. Without registration, the setting will not be recognized
by OpenSearch and attempts to use it will result in errors. Check where
CLUSTER_CREATE_INDEX_BLOCK_AUTO_RELEASE is registered (likely in a settings
registration class) and add CLUSTER_READ_BLOCK_AUTO_RELEASE alongside it.

server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdSettings.java [121-126]

 public static final Setting<Boolean> CLUSTER_READ_BLOCK_AUTO_RELEASE = Setting.boolSetting(
     "cluster.blocks.read.auto_release",
     true,
     Setting.Property.Dynamic,
     Setting.Property.NodeScope
 );
+// Ensure this setting is also added to the list of registered cluster settings
+// (e.g., in ClusterSettings.BUILT_IN_CLUSTER_SETTINGS or equivalent registry)
Suggestion importance[1-10]: 2

__

Why: The suggestion asks to verify that CLUSTER_READ_BLOCK_AUTO_RELEASE is registered in the cluster settings registry, but the improved_code only adds a comment rather than actual code. This is likely handled elsewhere in the codebase (similar to CLUSTER_CREATE_INDEX_BLOCK_AUTO_RELEASE), and the suggestion doesn't provide actionable code changes within the PR diff.

Low

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 1, 2026

❌ Gradle check result for 1cb260f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@drtootsie drtootsie force-pushed the add-read-block-auto-release-setting branch from 1cb260f to 6ad6837 Compare March 10, 2026 04:10
@github-actions
Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 6ad6837

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 6ad6837: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: drtootsie <peptz7@gmail.com>
@drtootsie drtootsie force-pushed the add-read-block-auto-release-setting branch from 6ad6837 to 26981d2 Compare March 11, 2026 02:17
@github-actions
Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 26981d2

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 26981d2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant