[BUG] Fix remote shards balancer when filtering throttled nodes by bugmakerrrrrr · Pull Request #11724 · opensearch-project/OpenSearch

bugmakerrrrrr · 2024-01-03T13:47:28Z

Description

Today, RemoteShardsBalancer uses AllocationDecider#canAllocateAnyShardToNode to filter throttled or ineligible nodes during allocating unassigned shards. If all eligible nodes of an unassigned shard are filtered before trying to allocate this shard, the shard will be marked as ignored with UnassignedInfo.AllocationStatus.DECIDERS_NO status. As a result, the corresponding ShardRestoreStatus will be set to Failure (RestoreService.RestoreInProgressUpdater#unassignedInfoUpdated). This pull request takes throttled nodes into account and ensures that shards are marked with the appropriate status.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-01-03T14:10:13Z

Compatibility status:

Checks if related components are compatible with change c8d000d

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/sql.git]

github-actions · 2024-01-03T14:40:32Z

❕ Gradle check result for 4c5408b: UNSTABLE

TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotAndRestore
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.classMethod
      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

bugmakerrrrrr · 2024-01-08T11:48:23Z

@kotwanikunal could you take a look at this additional one?

.../src/main/java/org/opensearch/cluster/routing/allocation/allocator/RemoteShardsBalancer.java

Signed-off-by: panguixin <panguixin@bytedance.com>

bugmakerrrrrr · 2024-01-23T13:12:44Z

@kotwanikunal can we merge this

github-actions · 2024-01-23T13:59:23Z

✅ Gradle check result for ec30c58: SUCCESS

kotwanikunal · 2024-01-23T22:15:51Z

@andrross / @linuxpi - Mind giving this one another review.

Question about tests

andrross · 2024-01-24T18:06:39Z

@bugmakerrrrrr Can we add a unit test for the bug being fixed here?

bugmakerrrrrr · 2024-01-25T03:02:12Z

@bugmakerrrrrr Can we add a unit test for the bug being fixed here?

@andrross I think that RemoteShardsAllocateUnassignedTests#testNoRemoteAllocation can cover this bug fix after modifying RemoteShardsBalancerBaseTestCase.

github-actions · 2024-01-25T17:36:57Z

❕ Gradle check result for c7ef861: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Andrew Ross <andrross@amazon.com>

github-actions · 2024-01-25T19:01:17Z

❌ Gradle check result for c8d000d:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-25T19:04:17Z

❌ Gradle check result for c8d000d:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-25T21:50:32Z

❕ Gradle check result for c8d000d: UNSTABLE

TEST FAILURES:

      3 org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT.testConcurrentDecommissionAction
      1 org.opensearch.search.SearchWeightedRoutingIT.testShardRoutingWithNetworkDisruption_FailOpenEnabled
      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

* fix remote shards balancer Signed-off-by: panguixin <panguixin@bytedance.com> * add change log Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com> (cherry picked from commit 9f649e0) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…) (#12024) * fix remote shards balancer * add change log --------- (cherry picked from commit 9f649e0) Signed-off-by: panguixin <panguixin@bytedance.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Andrew Ross <andrross@amazon.com>

…search-project#11724) * fix remote shards balancer Signed-off-by: panguixin <panguixin@bytedance.com> * add change log Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com>

…search-project#11724) * fix remote shards balancer Signed-off-by: panguixin <panguixin@bytedance.com> * add change log Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

kotwanikunal approved these changes Jan 9, 2024

View reviewed changes

linuxpi reviewed Jan 11, 2024

View reviewed changes

.../src/main/java/org/opensearch/cluster/routing/allocation/allocator/RemoteShardsBalancer.java Show resolved Hide resolved

.../src/main/java/org/opensearch/cluster/routing/allocation/allocator/RemoteShardsBalancer.java Show resolved Hide resolved

bugmakerrrrrr added 2 commits January 23, 2024 21:03

fix remote shards balancer

c9ecc72

Signed-off-by: panguixin <panguixin@bytedance.com>

add change log

ec30c58

Signed-off-by: panguixin <panguixin@bytedance.com>

bugmakerrrrrr force-pushed the fix_remote_shards_balancer branch from 7ab39d9 to ec30c58 Compare January 23, 2024 13:11

kotwanikunal approved these changes Jan 23, 2024

View reviewed changes

andrross previously approved these changes Jan 24, 2024

View reviewed changes

andrross self-requested a review January 24, 2024 18:05

Merge branch 'main' into fix_remote_shards_balancer

c7ef861

andrross added the backport 2.x Backport to 2.x branch label Jan 25, 2024

andrross approved these changes Jan 25, 2024

View reviewed changes

Merge branch 'main' into fix_remote_shards_balancer

c8d000d

Signed-off-by: Andrew Ross <andrross@amazon.com>

andrross approved these changes Jan 25, 2024

View reviewed changes

kotwanikunal merged commit 9f649e0 into opensearch-project:main Jan 25, 2024

opensearch-trigger-bot bot mentioned this pull request Jan 25, 2024

[Backport 2.x] [BUG] Fix remote shards balancer when filtering throttled nodes #12024

Merged

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

Conversation

bugmakerrrrrr commented Jan 3, 2024

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Jan 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Compatibility status:

Incompatible components

Skipped components

Compatible components

Uh oh!

github-actions bot commented Jan 3, 2024

Uh oh!

bugmakerrrrrr commented Jan 8, 2024

Uh oh!

Uh oh!

Uh oh!

bugmakerrrrrr commented Jan 23, 2024

Uh oh!

github-actions bot commented Jan 23, 2024

Uh oh!

kotwanikunal commented Jan 23, 2024

Uh oh!

andrross commented Jan 24, 2024

Uh oh!

bugmakerrrrrr commented Jan 25, 2024

Uh oh!

github-actions bot commented Jan 25, 2024

Uh oh!

github-actions bot commented Jan 25, 2024

Uh oh!

github-actions bot commented Jan 25, 2024

Uh oh!

github-actions bot commented Jan 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Jan 3, 2024 •

edited

Loading