Updated nomenclature to differentiate between hot and warm tiering implementation. by mayanksharma27 · Pull Request #17490 · opensearch-project/OpenSearch

mayanksharma27 · 2025-02-28T14:01:51Z

Description

We currently only have tiering implementation for warm indices.
The current nomenclature for feature flags and settings are too generic and cant differentiate between hot and warm indices.
Since we want to implement tiering for hot indices as well in near future; we would want to update the nomenclature to be extensible and easy to understand, this PR present the nomenclature changes.

Related Issues

Resolves #[17489]

Meta Issue #13149

Check List

[NA] Functionality includes testing.
[NA ] API changes companion pull request created, if applicable.
[NA] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-02-28T15:17:10Z

❌ Gradle check result for e920df4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/cluster/routing/RoutingPool.java

Signed-off-by: Mayank Sharma <smynk@amazon.com>

mayanksharma27

Addressed comments from @gbbafna

github-actions · 2025-03-06T12:08:55Z

✅ Gradle check result for 5552edd: SUCCESS

codecov · 2025-03-06T12:09:33Z

Codecov Report

Attention: Patch coverage is 69.56522% with 7 lines in your changes missing coverage. Please review.

Project coverage is 72.42%. Comparing base (2ee8660) to head (a32221e).
Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
.../opensearch/index/engine/NRTReplicationEngine.java	0.00%	0 Missing and 2 partials ⚠️
...in/java/org/opensearch/index/shard/IndexShard.java	0.00%	1 Missing and 1 partial ⚠️
...ing/allocation/allocator/RemoteShardsBalancer.java	0.00%	1 Missing ⚠️
...c/main/java/org/opensearch/index/IndexService.java	50.00%	0 Missing and 1 partial ⚠️
.../indices/replication/SegmentReplicationTarget.java	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17490      +/-   ##
============================================
- Coverage     72.43%   72.42%   -0.02%     
- Complexity    65694    65728      +34     
============================================
  Files          5311     5311              
  Lines        304937   304933       -4     
  Branches      44226    44225       -1     
============================================
- Hits         220872   220836      -36     
- Misses        65912    65995      +83     
+ Partials      18153    18102      -51

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…ntation for warm and upcoming hot implementation commit 5552edd Author: Mayank Sharma <smynk@amazon.com> Date: Thu Mar 6 16:38:38 2025 +0530 Empty commit to rerun flaky test Signed-off-by: Mayank Sharma <smynk@amazon.com> commit d18bd6e Author: Mayank Sharma <smynk@amazon.com> Date: Thu Mar 6 14:25:33 2025 +0530 Ran spotlessapply Signed-off-by: Mayank Sharma <smynk@amazon.com> commit d5ac854 Author: Mayank Sharma <smynk@amazon.com> Date: Wed Mar 5 19:59:05 2025 +0530 Empty commit to rerun flaky test Signed-off-by: Mayank Sharma <smynk@amazon.com> commit 44000ee Author: Mayank Sharma <smynk@amazon.com> Date: Wed Mar 5 19:17:00 2025 +0530 Fixed failing UTs for ShardsTieringAllocationTests Signed-off-by: Mayank Sharma <smynk@amazon.com> commit 03a9305 Author: Mayank Sharma <smynk@amazon.com> Date: Wed Mar 5 17:54:33 2025 +0530 Empty commit to rerun flaky test Signed-off-by: Mayank Sharma <smynk@amazon.com> commit 795936a Author: Mayank Sharma <smynk@amazon.com> Date: Wed Mar 5 12:10:18 2025 +0530 Updated settings from "index.is.warm" to "index.warm" as part of PR comments opensearch-project#17490 Signed-off-by: Mayank Sharma <smynk@amazon.com> commit 724b934 Author: Mayank Sharma <smynk@amazon.com> Date: Tue Mar 4 17:09:19 2025 +0530 Addressed comments for PR opensearch-project#17490 Signed-off-by: Mayank Sharma <smynk@amazon.com> commit f1a249f Author: Mayank Sharma <smynk@amazon.com> Date: Fri Feb 28 19:02:24 2025 +0530 Updated nomenclature to differentiate between current tiering implementation for warm and upcoming hot implementation Signed-off-by: Mayank Sharma <smynk@amazon.com> Signed-off-by: Mayank Sharma <smynk@amazon.com>

…current tiering implementation for warm and upcoming hot implementation commit 9cdec49 Author: Mayank Sharma <smynk@amazon.com> Date: Thu Mar 6 18:04:16 2025 +0530 Updated nomenclature to differentiate between current tiering implementation for warm and upcoming hot implementation commit 5552edd Author: Mayank Sharma <smynk@amazon.com> Date: Thu Mar 6 16:38:38 2025 +0530 Empty commit to rerun flaky test Signed-off-by: Mayank Sharma <smynk@amazon.com> commit d18bd6e Author: Mayank Sharma <smynk@amazon.com> Date: Thu Mar 6 14:25:33 2025 +0530 Ran spotlessapply Signed-off-by: Mayank Sharma <smynk@amazon.com> commit d5ac854 Author: Mayank Sharma <smynk@amazon.com> Date: Wed Mar 5 19:59:05 2025 +0530 Empty commit to rerun flaky test Signed-off-by: Mayank Sharma <smynk@amazon.com> commit 44000ee Author: Mayank Sharma <smynk@amazon.com> Date: Wed Mar 5 19:17:00 2025 +0530 Fixed failing UTs for ShardsTieringAllocationTests Signed-off-by: Mayank Sharma <smynk@amazon.com> commit 03a9305 Author: Mayank Sharma <smynk@amazon.com> Date: Wed Mar 5 17:54:33 2025 +0530 Empty commit to rerun flaky test Signed-off-by: Mayank Sharma <smynk@amazon.com> commit 795936a Author: Mayank Sharma <smynk@amazon.com> Date: Wed Mar 5 12:10:18 2025 +0530 Updated settings from "index.is.warm" to "index.warm" as part of PR comments opensearch-project#17490 Signed-off-by: Mayank Sharma <smynk@amazon.com> commit 724b934 Author: Mayank Sharma <smynk@amazon.com> Date: Tue Mar 4 17:09:19 2025 +0530 Addressed comments for PR opensearch-project#17490 Signed-off-by: Mayank Sharma <smynk@amazon.com> commit f1a249f Author: Mayank Sharma <smynk@amazon.com> Date: Fri Feb 28 19:02:24 2025 +0530 Updated nomenclature to differentiate between current tiering implementation for warm and upcoming hot implementation Signed-off-by: Mayank Sharma <smynk@amazon.com> Signed-off-by: Mayank Sharma <smynk@amazon.com> Signed-off-by: Mayank Sharma <smynk@amazon.com>

github-actions · 2025-03-06T14:20:54Z

❌ Gradle check result for 3c323e2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

mayanksharma27 · 2025-03-06T20:09:05Z

Flaky integ tests failing - #11037

github-actions · 2025-03-10T05:10:41Z

❌ Gradle check result for 33f94d1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-12T08:56:17Z

❌ Gradle check result for e6e82ca: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-12T09:20:57Z

❌ Gradle check result for 33f94d1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

…current tiering implementation for warm and upcoming hot implementation commit 9cdec49 Author: Mayank Sharma <smynk@amazon.com> Date: Thu Mar 6 18:04:16 2025 +0530 Updated nomenclature to differentiate between current tiering implementation for warm and upcoming hot implementation commit 5552edd Author: Mayank Sharma <smynk@amazon.com> Date: Thu Mar 6 16:38:38 2025 +0530 Empty commit to rerun flaky test Signed-off-by: Mayank Sharma <smynk@amazon.com> commit d18bd6e Author: Mayank Sharma <smynk@amazon.com> Date: Thu Mar 6 14:25:33 2025 +0530 Ran spotlessapply Signed-off-by: Mayank Sharma <smynk@amazon.com> commit d5ac854 Author: Mayank Sharma <smynk@amazon.com> Date: Wed Mar 5 19:59:05 2025 +0530 Empty commit to rerun flaky test Signed-off-by: Mayank Sharma <smynk@amazon.com> commit 44000ee Author: Mayank Sharma <smynk@amazon.com> Date: Wed Mar 5 19:17:00 2025 +0530 Fixed failing UTs for ShardsTieringAllocationTests Signed-off-by: Mayank Sharma <smynk@amazon.com> commit 03a9305 Author: Mayank Sharma <smynk@amazon.com> Date: Wed Mar 5 17:54:33 2025 +0530 Empty commit to rerun flaky test Signed-off-by: Mayank Sharma <smynk@amazon.com> commit 795936a Author: Mayank Sharma <smynk@amazon.com> Date: Wed Mar 5 12:10:18 2025 +0530 Updated settings from "index.is.warm" to "index.warm" as part of PR comments opensearch-project#17490 Signed-off-by: Mayank Sharma <smynk@amazon.com> commit 724b934 Author: Mayank Sharma <smynk@amazon.com> Date: Tue Mar 4 17:09:19 2025 +0530 Addressed comments for PR opensearch-project#17490 Signed-off-by: Mayank Sharma <smynk@amazon.com> commit f1a249f Author: Mayank Sharma <smynk@amazon.com> Date: Fri Feb 28 19:02:24 2025 +0530 Updated nomenclature to differentiate between current tiering implementation for warm and upcoming hot implementation Signed-off-by: Mayank Sharma <smynk@amazon.com> Signed-off-by: Mayank Sharma <smynk@amazon.com> Signed-off-by: Mayank Sharma <smynk@amazon.com>

github-actions · 2025-03-12T21:55:10Z

❌ Gradle check result for a32221e: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-13T07:08:15Z

❌ Gradle check result for a32221e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-13T09:23:07Z

❌ Gradle check result for a32221e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Bukhtawar

Is there a difference between searchable snapshot index vs tiered warm index?

github-actions · 2025-03-17T07:20:38Z

✅ Gradle check result for a32221e: SUCCESS

gbbafna · 2025-03-17T08:07:17Z

Is there a difference between searchable snapshot index vs tiered warm index?

Yes, index settings are different for both of them . For tiered warm index , store.data_locality is set to partial , whereas for searchable snapshots index.store.type: remote_snapshot is used.

… current tiering implementation for warm and upcoming hot implementation (opensearch-project#17490) Signed-off-by: Mayank Sharma <smynk@amazon.com> Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

skumawat2025 · 2025-04-16T14:41:43Z

server/src/main/java/org/opensearch/index/IndexSettings.java

-            IndexModule.INDEX_STORE_LOCALITY_SETTING.getKey(),
-            IndexModule.DataLocalityType.FULL.toString()
-        ).equalsIgnoreCase(IndexModule.DataLocalityType.PARTIAL.toString());
+


I am sorry if i missed anything earlier. but why are we removing this setting?

We are not removing , but renaming the setting .

Ok. So after this change. To create a warm index we need to provide setting
warm.index: true and no need to specify the data_locality setting is it?
In this case it should pick up the index.data_locality: PARTIAL value by default?

… current tiering implementation for warm and upcoming hot implementation (opensearch-project#17490) Signed-off-by: Mayank Sharma <smynk@amazon.com>

opensearch-ci-bot mentioned this pull request Feb 28, 2025

[AUTOCUT] Gradle Check Flaky Test Report for DedicatedClusterSnapshotRestoreIT #15806

Open

gbbafna reviewed Mar 1, 2025

View reviewed changes

server/src/main/java/org/opensearch/cluster/routing/RoutingPool.java Outdated Show resolved Hide resolved

mayanksharma27 added a commit to mayanksharma27/OpenSearch that referenced this pull request Mar 4, 2025

Addressed comments for PR opensearch-project#17490

38cf2ad

Signed-off-by: Mayank Sharma <smynk@amazon.com>

mayanksharma27 commented Mar 4, 2025

View reviewed changes

mayanksharma27 force-pushed the smynk-tiering-1 branch from 5552edd to 9cdec49 Compare March 6, 2025 13:22

mayanksharma27 force-pushed the smynk-tiering-1 branch from 9cdec49 to 3c323e2 Compare March 6, 2025 13:56

opensearch-ci-bot mentioned this pull request Mar 7, 2025

[AUTOCUT] Gradle Check Flaky Test Report for S3BlobContainerRetriesTests #17551

Open

mayanksharma27 force-pushed the smynk-tiering-1 branch from e6e82ca to 33f94d1 Compare March 12, 2025 09:13

mayanksharma27 force-pushed the smynk-tiering-1 branch from 33f94d1 to a32221e Compare March 12, 2025 19:51

opensearch-ci-bot mentioned this pull request Mar 12, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RemotePrimaryRelocationIT #17364

Closed

Bukhtawar reviewed Mar 15, 2025

View reviewed changes

gbbafna approved these changes Mar 17, 2025

View reviewed changes

gbbafna merged commit e27cf58 into opensearch-project:main Mar 17, 2025
31 of 32 checks passed

opensearch-ci-bot mentioned this pull request Mar 17, 2025

[AUTOCUT] Gradle Check Flaky Test Report for S3BlobStoreRepositoryTests #14299

Open

skumawat2025 reviewed Apr 16, 2025

View reviewed changes

BrewTestBot mentioned this pull request May 6, 2025

opensearch 3.0.0 Homebrew/homebrew-core#222665

Merged

opensearch-ci-bot mentioned this pull request Apr 15, 2025

[AUTOCUT] Gradle Check Flaky Test Report for SearchReplicaFilteringAllocationIT #17221

Open

Conversation

mayanksharma27 commented Feb 28, 2025

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Feb 28, 2025

Uh oh!

Uh oh!

mayanksharma27 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 6, 2025

Uh oh!

codecov bot commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Mar 6, 2025

Uh oh!

mayanksharma27 commented Mar 6, 2025

Uh oh!

github-actions bot commented Mar 10, 2025

Uh oh!

github-actions bot commented Mar 12, 2025

Uh oh!

github-actions bot commented Mar 12, 2025

Uh oh!

github-actions bot commented Mar 12, 2025

Uh oh!

github-actions bot commented Mar 13, 2025

Uh oh!

github-actions bot commented Mar 13, 2025

Uh oh!

Bukhtawar left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 17, 2025

Uh oh!

gbbafna commented Mar 17, 2025

Uh oh!

Uh oh!

skumawat2025 Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gbbafna Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

skumawat2025 Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Mar 6, 2025 •

edited

Loading

skumawat2025 Apr 16, 2025 •

edited

Loading