Skip to content

Cache index shard limit to optimise ShardLimitsAllocationDecider#14962

Merged
Bukhtawar merged 11 commits intoopensearch-project:mainfrom
imRishN:optimise-setting-parse
Jul 29, 2024
Merged

Cache index shard limit to optimise ShardLimitsAllocationDecider#14962
Bukhtawar merged 11 commits intoopensearch-project:mainfrom
imRishN:optimise-setting-parse

Conversation

@imRishN
Copy link
Copy Markdown
Member

@imRishN imRishN commented Jul 24, 2024 •
edited
Loading

Description

As described in the issue, with large number of shards, ShardsLimitAllocationDecider spends lots of time parsing settings. As a part of this PR, we are caching the setting value to optimise the decider run time.

Benchmark Before Changes

Benchmark                                              (indicesNodes)  Mode  Cnt       Score       Error  Units
RerouteBenchmark.measureShardAllocationEmptyCluster      10000|  500|  avgt    3  839663.255 ± 48190.442  ms/op

Benchmark After Changes

Benchmark                                              (indicesNodes)  Mode  Cnt       Score       Error  Units
RerouteBenchmark.measureShardAllocationEmptyCluster      10000|  500|  avgt    3  755228.219 ± 41246.861  ms/op

Percentage improvement while assigning 200k shards - 10.05%

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#14965

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

imRishN added 2 commits July 24, 2024 21:56
@imRishN
Cache index shard limit per node
0c49889 
Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
@imRishN
Cache index shard limit to optimise ShardLimitsAllocationDecider
d1e702f 
Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 24, 2024

❌ Gradle check result for d1e702f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@imRishN imRishN mentioned this pull request Jul 24, 2024
Closed
imRishN added 2 commits July 25, 2024 02:34
@imRishN
Add RerouteBenchmark
ec6ce0b 
Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
@imRishN
Fix imports
fe6e5d9 
Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
@imRishN imRishN marked this pull request as ready for review July 24, 2024 21:09
@imRishN
Fix spotless
4ed2164 
Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 24, 2024

❌ Gradle check result for ec6ce0b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 24, 2024

❌ Gradle check result for fe6e5d9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 24, 2024

❌ Gradle check result for 4ed2164: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@imRishN
Trigger Build
439bce2 
Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 25, 2024

❌ Gradle check result for 439bce2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@imRishN
Trigger Build
818a902 
Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 25, 2024

❌ Gradle check result for 818a902: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@imRishN
Trigger Build
2f3b3e1 
Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 25, 2024

❌ Gradle check result for 2f3b3e1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 27, 2024

❕ Gradle check result for a93f11d: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 27, 2024

❌ Gradle check result for 950a5aa: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@codecov
Copy link
Copy Markdown

codecov bot commented Jul 27, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.86%. Comparing base (1fe58b5) to head (b937cfc).
Report is 6 commits behind head on main.

Additional details and impacted files 
@@             Coverage Diff              @@
##               main   #14962      +/-   ##
============================================
+ Coverage     71.78%   71.86%   +0.07%     
- Complexity    62694    62752      +58     
============================================
  Files          5160     5161       +1     
  Lines        294211   294363     +152     
  Branches      42553    42577      +24     
============================================
+ Hits         211212   211540     +328     
+ Misses        65599    65403     -196     
- Partials      17400    17420      +20

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@imRishN
Trigger Build
b937cfc 
Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 27, 2024

✅ Gradle check result for b937cfc: SUCCESS

@Bukhtawar Bukhtawar merged commit 122f3f0 into opensearch-project:main Jul 29, 2024
@Bukhtawar Bukhtawar added the backport 2.x Backport to 2.x branch label Jul 29, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 29, 2024
@github-actions
Cache index shard limit to optimise ShardLimitsAllocationDecider (#14962
4544af7 
)

* Cache index shard limit per node

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
(cherry picked from commit 122f3f0)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Jul 29, 2024
Merged
dblock pushed a commit that referenced this pull request Aug 7, 2024
@opensearch-trigger-bot @github-actions
Cache index shard limit to optimise ShardLimitsAllocationDecider (#14962
8d664b3 
) (#15000)

* Cache index shard limit per node


(cherry picked from commit 122f3f0)

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
harshavamsi pushed a commit to harshavamsi/OpenSearch that referenced this pull request Aug 20, 2024
@imRishN @harshavamsi
Cache index shard limit to optimise ShardLimitsAllocationDecider (ope…
c161813 
…nsearch-project#14962)

* Cache index shard limit per node

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
wdongyu pushed a commit to wdongyu/OpenSearch that referenced this pull request Aug 22, 2024
@imRishN
Cache index shard limit to optimise ShardLimitsAllocationDecider (ope…
9eabd42 
…nsearch-project#14962)

* Cache index shard limit per node

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
This was referenced Sep 11, 2024
Open
Open
Closed
Open
@prudhvigodithi prudhvigodithi mentioned this pull request Oct 16, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Bukhtawar Bukhtawar Bukhtawar approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz

@andrross andrross Awaiting requested review from andrross

@ashking94 ashking94 Awaiting requested review from ashking94

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE

@dblock dblock Awaiting requested review from dblock

@dbwiddis dbwiddis Awaiting requested review from dbwiddis

@gbbafna gbbafna Awaiting requested review from gbbafna

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal

@mch2 mch2 Awaiting requested review from mch2

@msfroh msfroh Awaiting requested review from msfroh

@nknize nknize Awaiting requested review from nknize

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19

@reta reta Awaiting requested review from reta

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159

@sachinpkale sachinpkale Awaiting requested review from sachinpkale

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli

@shwetathareja shwetathareja Awaiting requested review from shwetathareja

@sohami sohami Awaiting requested review from sohami

@VachaShah VachaShah Awaiting requested review from VachaShah

Assignees

No one assigned

Labels

backport 2.x Backport to 2.x branch skip-changelog

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@imRishN @Bukhtawar