Renaming the node role search to warm

[vinaykpud]

In this PR we are renaming the existing Node "Search Role" to "Warm Role"

Description

This is done based on the decision taken as part of the discussion in this thread: #17422 (comment)

Related Issues

Related to #15306
Related to #17422

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR [created] [DOC] Rename Search node role to Warm node role for searchable snapshots documentation-website#9392

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[mch2]

This is a straight rename correct? It does not resolve #17422 ?

[github-actions]

❌ Gradle check result for 835fd60: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[vinaykpud]

This is a straight rename correct? It does not resolve #17422 ?

Yes, Do you want me to add a new issue for this?

[github-actions]

❌ Gradle check result for b0d026a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for b0d026a: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 92.85714% with 2 lines in your changes missing coverage. Please review.

Project coverage is 72.49%. Comparing base (1166998) to head (d05b81a).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
.../java/org/opensearch/env/NodeRepurposeCommand.java	81.81%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17573      +/-   ##
============================================
+ Coverage     72.48%   72.49%   +0.01%     
+ Complexity    65771    65752      -19     
============================================
  Files          5311     5311              
  Lines        304973   304973              
  Branches      44229    44229              
============================================
+ Hits         221045   221085      +40     
+ Misses        65830    65760      -70     
- Partials      18098    18128      +30     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andrross]

Let's make sure this aligns with what @gbbafna mentioned here before merging.

We'd generally want to do a deprecation cycle before just replacing a term like this. However, it does appear that search was a bit of a misnomer and it is not in wide use beyond searchable snapshots, so I'm not totally opposed with doing this to unblock the usage of the search role where it might actually make more sense.

[gbbafna]

@vinaykpud : Can we verify if we upgrade the cluster from 2.x to 3.0 with this change, searchable snapshots continue to work ? There might be few nuances here like changing the cluster-manager first . Want to make sure that rolling restarts works with this change

[vinaykpud]

@vinaykpud : Can we verify if we upgrade the cluster from 2.x to 3.0 with this change, searchable snapshots continue to work ? There might be few nuances here like changing the cluster-manager first . Want to make sure that rolling restarts works with this change

Sure @gbbafna

Performed a test to check the cluster upgrade scenario. Using opensearch-cluster-cdk I have setup cluster with 3 cluster manager and 2 data nodes.
Here is the summary of the steps performed.

1. Created index with 1P and 1R and indexed few documents
2. Registered a S3 repo for snapshots
3. took a snapshot
4. Deleted the index
5. Updated the node role from data to search by changing the yaml file, restarted the process in both data nodes to make them snapshot search nodes
6. restored searchable snapshot and made sure shards are assigned
7. created tarball of the OpenSearch with changes in this PR and copied to all the nodes
8. Restarted the OS process with latest binaries first in each cluster managers one by one and cluster was healthy.
9. Then restarted the OS process with latest binaries in each search node one after another.
10. During this process I had _cat/nodes, _cat/shards, health APIs running in loop for monitoring the cluster.
11. Cluster din't go Red and upgraded to OS 3.0.0
12. Also verified by querying _search on the searchable snapshot index and its working.

Let me know if we missed anything here.

cc @mch2 @andrross

[mch2]

@vinaykpud : Can we verify if we upgrade the cluster from 2.x to 3.0 with this change, searchable snapshots continue to work ? There might be few nuances here like changing the cluster-manager first . Want to make sure that rolling restarts works with this change

Sure @gbbafna

Performed a test to check the cluster upgrade scenario. Using opensearch-cluster-cdk I have setup cluster with 3 cluster manager and 2 data nodes. Here is the summary of the steps performed.

1. Created index with 1P and 1R and indexed few documents
2. Registered a S3 repo for snapshots
3. took a snapshot
4. Deleted the index
5. Updated the node role from data to search by changing the yaml file, restarted the process in both data nodes to make them snapshot search nodes
6. restored searchable snapshot and made sure shards are assigned
7. created tarball of the OpenSearch with changes in this PR and copied to all the nodes
8. Restarted the OS process with latest binaries first in each cluster managers one by one and cluster was healthy.
9. Then restarted the OS process with latest binaries in each search node one after another.
10. During this process I had _cat/nodes, _cat/shards, health APIs running in loop for monitoring the cluster.
11. Cluster din't go Red and upgraded to OS 3.0.0
12. Also verified by querying _search on the searchable snapshot index and its working.

Let me know if we missed anything here.

cc @mch2 @andrross

So to summarize, you did a successful rolling upgrade from 2.19 to 3.0 but it required cm's to migrate first. Did you not have to update the routing pool logic for bwc? I don't see this in the pr.

    public static RoutingPool getNodePool(DiscoveryNode node) {
        if (node.isWarmNode() || (node.isSearchNode() && (node.getVersion().before(Version.V_3_0_0)))) {
            return REMOTE_CAPABLE;
        }
        return LOCAL_ONLY;
    }

[vinaykpud]

@vinaykpud : Can we verify if we upgrade the cluster from 2.x to 3.0 with this change, searchable snapshots continue to work ? There might be few nuances here like changing the cluster-manager first . Want to make sure that rolling restarts works with this change

Sure @gbbafna
Performed a test to check the cluster upgrade scenario. Using opensearch-cluster-cdk I have setup cluster with 3 cluster manager and 2 data nodes. Here is the summary of the steps performed.

1. Created index with 1P and 1R and indexed few documents
2. Registered a S3 repo for snapshots
3. took a snapshot
4. Deleted the index
5. Updated the node role from data to search by changing the yaml file, restarted the process in both data nodes to make them snapshot search nodes
6. restored searchable snapshot and made sure shards are assigned
7. created tarball of the OpenSearch with changes in this PR and copied to all the nodes
8. Restarted the OS process with latest binaries first in each cluster managers one by one and cluster was healthy.
9. Then restarted the OS process with latest binaries in each search node one after another.
10. During this process I had _cat/nodes, _cat/shards, health APIs running in loop for monitoring the cluster.
11. Cluster din't go Red and upgraded to OS 3.0.0
12. Also verified by querying _search on the searchable snapshot index and its working.

Let me know if we missed anything here.
cc @mch2 @andrross

So to summarize, you did a successful rolling upgrade from 2.19 to 3.0 but it required cm's to migrate first. Did you not have to update the routing pool logic for bwc? I don't see this in the pr.

    public static RoutingPool getNodePool(DiscoveryNode node) {
        if (node.isWarmNode() || (node.isSearchNode() && (node.getVersion().before(Version.V_3_0_0)))) {
            return REMOTE_CAPABLE;
        }
        return LOCAL_ONLY;
    }

@mch2 , No, I haven't updated the routing pool logic, it worked without adding any new logic.
But I haven't tried other way ie: First Upgrading Search/Warm Nodes and Upgrading CM's later. We might need to add this logic for that.

[mch2]

First Upgrading Search/Warm Nodes and Upgrading CM's later

@vinaykpud No this logic would be on 3.0 cms not 2.x. I was thinking this would provide added safety for mixed cluster cases where we need to make alloc decisions and assign ss shards in that mixed state. Though in the normal upgrade case this isn't required. I think we can go ahead without this.

[github-actions]

❕ Gradle check result for c6d46d2: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testSnapshotWithStuckNode
      1 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

✅ Gradle check result for d05b81a: SUCCESS