Adding configurable resiliency features to MergedSegmentWarmer by kh3ra · Pull Request #19629 · opensearch-project/OpenSearch

kh3ra · 2025-10-14T13:12:06Z

Description

Merge-related config changes

Standardized merged segment warmer - related setting names.
Added a threshold property that controls which merged segments get pre-copied (warmed), ensuring only segments larger than the specified size are warmed.
Added cluster defaults for max_merge_count and max_merge_threads.
AsyncPublishReferencedSegmentsTask only runs when INDICES_REPLICATION_MERGES_WARMER_ENABLED_SETTING = true

Related Issues

Public documentation - opensearch-project/documentation-website#11410

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-10-14T13:59:27Z

❌ Gradle check result for 3aeef75: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/index/engine/MergedSegmentWarmer.java

server/src/main/java/org/opensearch/indices/recovery/RecoverySettings.java

server/src/main/java/org/opensearch/index/engine/InternalEngine.java

server/src/main/java/org/opensearch/index/engine/MergedSegmentWarmer.java

github-actions · 2025-10-15T00:29:48Z

❌ Gradle check result for f0feabe: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-10-15T09:10:45Z

❌ Gradle check result for bbd9dd4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-10-15T10:19:23Z

❌ Gradle check result for 2459597: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-10-15T15:26:30Z

❌ Gradle check result for 36d0a37: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-10-15T16:42:28Z

❌ Gradle check result for 4aec3d9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-10-15T17:09:51Z

❌ Gradle check result for b06e2ac: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kh3ra · 2025-10-23T12:41:06Z

Flaky test failure - #18484

github-actions · 2025-10-23T16:18:08Z

❌ Gradle check result for af90e55: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-10-23T16:45:21Z

❌ Gradle check result for 32a46cc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-10-23T17:38:48Z

❌ Gradle check result for b07e5da: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-10-23T19:29:18Z

❌ Gradle check result for f9c1f3d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-10-24T06:06:48Z

❌ Gradle check result for f9c1f3d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Standardized merged segment warmer - related setting names. Added a threshold property that controls which merged segments get pre-copied (warmed), ensuring only segments larger than the specified size are warmed. Added cluster defaults for max_merge_count and max_merge_threads. AsyncPublishReferencedSegmentsTask only runs when INDICES_REPLICATION_MERGES_WARMER_ENABLED_SETTING = true Signed-off-by: Aditya Khera <kheraadi@amazon.com>

Signed-off-by: Aditya Khera <kheraadi@amazon.com>

github-actions · 2025-10-24T09:56:09Z

✅ Gradle check result for 677c46a: SUCCESS

…earch-project#19629) * Merge-related config changes Standardized merged segment warmer - related setting names. Added a threshold property that controls which merged segments get pre-copied (warmed), ensuring only segments larger than the specified size are warmed. Added cluster defaults for max_merge_count and max_merge_threads. AsyncPublishReferencedSegmentsTask only runs when INDICES_REPLICATION_MERGES_WARMER_ENABLED_SETTING = true Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Empty commit to trigger build Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Fixing flakiness in ClusterMergeSchedulerConfigsIT Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Fixing breaking changes Signed-off-by: Aditya Khera <kheraadi@amazon.com> * API changes Signed-off-by: Aditya Khera <kheraadi@amazon.com> --------- Signed-off-by: Aditya Khera <kheraadi@amazon.com> Co-authored-by: Aditya Khera <kheraadi@amazon.com>

andrross · 2025-11-13T21:45:54Z

server/src/test/java/org/opensearch/index/engine/MergedSegmentWarmerTests.java

+    }
+
+    public void testShouldWarm_failure() throws IOException {
+        ClassLoader.getSystemClassLoader().setDefaultAssertionStatus(false);


@kh3ra @ashking94 Why was this added? We have tests that depend on assertions tripping when we test the invariants they are asserting. As I understand it this will disable assertions for this particular JVM instance, which JUnit will reuse to run subsequent tests on, though it's non-deterministic which tests will run on which JVM in what order. See #19765 where @varunbharadwaj and I have been struggling with an unrelated test failing due to assertions being disabled.

…earch-project#19629) * Merge-related config changes Standardized merged segment warmer - related setting names. Added a threshold property that controls which merged segments get pre-copied (warmed), ensuring only segments larger than the specified size are warmed. Added cluster defaults for max_merge_count and max_merge_threads. AsyncPublishReferencedSegmentsTask only runs when INDICES_REPLICATION_MERGES_WARMER_ENABLED_SETTING = true Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Empty commit to trigger build Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Fixing flakiness in ClusterMergeSchedulerConfigsIT Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Fixing breaking changes Signed-off-by: Aditya Khera <kheraadi@amazon.com> * API changes Signed-off-by: Aditya Khera <kheraadi@amazon.com> --------- Signed-off-by: Aditya Khera <kheraadi@amazon.com> Co-authored-by: Aditya Khera <kheraadi@amazon.com>

kh3ra requested review from a team, Bukhtawar, CEHENKLE, Rishikesh1159, anasalkouz, andrross, ashking94, cwperks, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh, owaiskazi19, reta, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners October 14, 2025 13:12

ashking94 requested changes Oct 14, 2025

View reviewed changes

kh3ra force-pushed the dev/merged-segment-warmer-configs branch from f0feabe to 34610d9 Compare October 15, 2025 08:49

kh3ra force-pushed the dev/merged-segment-warmer-configs branch from bbd9dd4 to 2459597 Compare October 15, 2025 10:08

opensearch-ci-bot mentioned this pull request Oct 23, 2025

[AUTOCUT] Gradle Check Flaky Test Report for ExampleSystemIngestProcessorClientYamlTestSuiteIT #18484

Open

kh3ra mentioned this pull request Oct 23, 2025

[DOC] Add docs for merged segment warmer and related configs opensearch-project/documentation-website#11410

Closed

4 tasks

kh3ra force-pushed the dev/merged-segment-warmer-configs branch from 32a46cc to 1b635d9 Compare October 23, 2025 17:09

kh3ra force-pushed the dev/merged-segment-warmer-configs branch from b07e5da to f9c1f3d Compare October 23, 2025 18:46

Aditya Khera added 5 commits October 24, 2025 14:14

Empty commit to trigger build

ad9a744

Signed-off-by: Aditya Khera <kheraadi@amazon.com>

Fixing flakiness in ClusterMergeSchedulerConfigsIT

2028d7e

Signed-off-by: Aditya Khera <kheraadi@amazon.com>

Fixing breaking changes

80c393a

Signed-off-by: Aditya Khera <kheraadi@amazon.com>

API changes

677c46a

Signed-off-by: Aditya Khera <kheraadi@amazon.com>

kh3ra force-pushed the dev/merged-segment-warmer-configs branch from f9c1f3d to 677c46a Compare October 24, 2025 08:44

ashking94 merged commit 8f838f1 into opensearch-project:main Oct 24, 2025
33 checks passed

This was referenced Oct 25, 2025

[AUTOCUT] Gradle Check Flaky Test Report for MergedSegmentWarmerIT #19042

Open

[AUTOCUT] Gradle Check Flaky Test Report for S3BlobContainerRetriesTests #17551

Open

RS146BIJAY mentioned this pull request Nov 2, 2025

[BUG] MergeSchedulerSettingsTests.testMaxThreadAndMergeCount test case is flaky #19866

Open

opensearch-ci-bot mentioned this pull request Nov 12, 2025

[AUTOCUT] Gradle Check Flaky Test Report for UpdateByQueryBasicTests #16439

Open

andrross reviewed Nov 13, 2025

View reviewed changes

opensearch-ci-bot mentioned this pull request Nov 21, 2025

[AUTOCUT] Gradle Check Flaky Test Report for InternalEngineTests #15838

Open

BrewTestBot mentioned this pull request Dec 17, 2025

opensearch 3.4.0 Homebrew/homebrew-core#258986

Merged

tkykenmt mentioned this pull request Jan 11, 2026

docs: add merged segment warmer report for v3.4.0 tkykenmt/opensearch-feature-explorer#1803

Merged

opensearch-ci-bot mentioned this pull request Feb 18, 2026

[AUTOCUT] Gradle Check Flaky Test Report for Gradle Test Run :modules:mapper-extras:test #20661

Closed

Conversation

kh3ra commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Oct 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 15, 2025

Uh oh!

github-actions bot commented Oct 15, 2025

Uh oh!

github-actions bot commented Oct 15, 2025

Uh oh!

github-actions bot commented Oct 15, 2025

Uh oh!

github-actions bot commented Oct 15, 2025

Uh oh!

github-actions bot commented Oct 15, 2025

Uh oh!

kh3ra commented Oct 23, 2025

Uh oh!

github-actions bot commented Oct 23, 2025

Uh oh!

github-actions bot commented Oct 23, 2025

Uh oh!

github-actions bot commented Oct 23, 2025

Uh oh!

github-actions bot commented Oct 23, 2025

Uh oh!

github-actions bot commented Oct 24, 2025

Uh oh!

github-actions bot commented Oct 24, 2025

Uh oh!

Uh oh!

andrross Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kh3ra commented Oct 14, 2025 •

edited

Loading