Download functionality of global metadata from remote store by dhwanilpatel · Pull Request #10535 · opensearch-project/OpenSearch

dhwanilpatel · 2023-10-10T15:20:55Z

Description

Changes for downloading global metadata to remote store.

It will download global metadata file and de-serialize it in Metadata object.

Related Issues

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-10-10T15:46:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/27497/
CommitID: ca03acf
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-10T15:51:27Z

Compatibility status:

Checks if related components are compatible with change 4f377d2

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

…_download Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-10-11T09:15:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/27586/
CommitID: 41ed4d3

codecov · 2023-10-11T09:17:09Z

Codecov Report

Merging #10535 (41ed4d3) into main (e942483) will increase coverage by 0.12%.
Report is 2 commits behind head on main.
The diff coverage is 86.66%.

❗ Current head 41ed4d3 differs from pull request most recent head 4f377d2. Consider uploading reports for the commit 4f377d2 to get more accurate results

@@             Coverage Diff              @@
##               main   #10535      +/-   ##
============================================
+ Coverage     71.12%   71.25%   +0.12%     
+ Complexity    58503    58452      -51     
============================================
  Files          4853     4844       -9     
  Lines        275915   275299     -616     
  Branches      40153    40084      -69     
============================================
- Hits         196256   196162      -94     
+ Misses        63247    62689     -558     
- Partials      16412    16448      +36

Files	Coverage Δ
...search/gateway/remote/ClusterMetadataManifest.java	`97.65% <100.00%> (+1.85%)`	⬆️
...arch/gateway/remote/RemoteClusterStateService.java	`68.36% <84.61%> (+0.29%)`	⬆️

... and 534 files with indirect coverage changes

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-10-11T11:57:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/27602/
CommitID: 838d513
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

CHANGELOG.md

server/src/main/java/org/opensearch/gateway/remote/ClusterMetadataManifest.java

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-10-12T09:50:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/27735/
CommitID: 055fbb1
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java

amkhar

Minor comments.

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-10-13T08:22:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL:
CommitID: 133c2ca
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

shwetathareja

minor comments, lets rebase it once upload PR is merged.

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java

…_download Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-10-18T12:49:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28246/
CommitID: bd9fbeb
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-18T12:57:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28248/
CommitID: 8055dfd
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-10-18T13:48:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28257/
CommitID: 4f377d2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…ch-project#10535) * Download funcationality of global metadata from remote store Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

* Download funcationality of global metadata from remote store Signed-off-by: Dhwanil Patel <dhwanip@amazon.com> (cherry picked from commit 3a36c22) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…10733) * Download funcationality of global metadata from remote store (cherry picked from commit 3a36c22) Signed-off-by: Dhwanil Patel <dhwanip@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ch-project#10535) * Download funcationality of global metadata from remote store Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

…ch-project#10535) * Download funcationality of global metadata from remote store Signed-off-by: Dhwanil Patel <dhwanip@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

Download funcationality of global metadata from remote store

ca03acf

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

Merge remote-tracking branch 'upstream/main' into remote_global_state…

41ed4d3

…_download Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

Add Changelog

838d513

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

dhwanilpatel changed the title ~~Download funcationality of global metadata from remote store~~ Download functionality of global metadata from remote store Oct 11, 2023

dhwanilpatel marked this pull request as ready for review October 11, 2023 11:59

dhwanilpatel requested review from Bukhtawar, CEHENKLE, Rishikesh1159, VachaShah, anasalkouz, andrross, dblock, dreamer-89, gbbafna, kartg, kotwanikunal, mch2, nknize, owaiskazi19, peternied, reta, ryanbogan, saratvemulapalli, setiah and shwetathareja as code owners October 11, 2023 11:59

dhwanilpatel requested review from msfroh and sohami as code owners October 11, 2023 11:59

stephen-crawford reviewed Oct 11, 2023

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/gateway/remote/ClusterMetadataManifest.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/gateway/remote/ClusterMetadataManifest.java Outdated Show resolved Hide resolved

linuxpi reviewed Oct 12, 2023

View reviewed changes

Incorporated comments

055fbb1

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

linuxpi reviewed Oct 12, 2023

View reviewed changes

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java Show resolved Hide resolved

amkhar reviewed Oct 13, 2023

View reviewed changes

Fixed UTs and incorporated comments

133c2ca

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

shwetathareja reviewed Oct 16, 2023

View reviewed changes

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java Outdated Show resolved Hide resolved

dhwanilpatel added 2 commits October 18, 2023 17:20

Merge remote-tracking branch 'upstream/main' into remote_global_state…

e77733e

…_download Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

Rebased with upload changed, completed todo

8055dfd

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

dhwanilpatel force-pushed the remote_global_state_download branch from bd9fbeb to 8055dfd Compare October 18, 2023 12:22

Fixed UT

4f377d2

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

shwetathareja approved these changes Oct 18, 2023

View reviewed changes

shwetathareja mentioned this pull request Oct 18, 2023

[BUG] Flaky Test - org.opensearch.index.shard.IndexShardTests.testCommitLevelRestoreShardFromRemoteStore #10704

Closed

shwetathareja merged commit 3a36c22 into opensearch-project:main Oct 18, 2023

shwetathareja added the backport 2.x Backport to 2.x branch label Oct 19, 2023

opensearch-trigger-bot bot mentioned this pull request Oct 19, 2023

[Backport 2.x] Download functionality of global metadata from remote store #10733

Merged

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

gbbafna mentioned this pull request Aug 8, 2024

Add Varun Bansal as maintainer #15163

Merged

Conversation

dhwanilpatel commented Oct 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Oct 10, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Oct 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Compatibility status:

Incompatible components

Skipped components

Compatible components

Uh oh!

github-actions bot commented Oct 11, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

codecov bot commented Oct 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Oct 11, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 12, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Uh oh!

Uh oh!

amkhar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 13, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

shwetathareja left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 18, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Oct 18, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Oct 18, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dhwanilpatel commented Oct 10, 2023 •

edited

Loading

github-actions bot commented Oct 10, 2023 •

edited

Loading

codecov bot commented Oct 11, 2023 •

edited

Loading