Skip to content

[Remote Migration] Changes for Primary Relocation during migration #12494

Merged
sachinpkale merged 14 commits intoopensearch-project:mainfrom
gbbafna:mixed-relocation
Mar 21, 2024
Merged

[Remote Migration] Changes for Primary Relocation during migration #12494
sachinpkale merged 14 commits intoopensearch-project:mainfrom
gbbafna:mixed-relocation

Conversation

@gbbafna
Copy link
Copy Markdown
Contributor

@gbbafna gbbafna commented Feb 29, 2024 •
edited
Loading

Description

Allows relocation from docrep primary to remote primary and remote primary to remote primary .

This also seeds remote store while moving from docrep to primary .

Related Issues

Resolves #12689

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@gbbafna gbbafna changed the title [Remote Store Migration] Changes for Primary Relocation during migration [Remote Migration] Changes for Primary Relocation during migration Feb 29, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Feb 29, 2024 •
edited
Loading

Compatibility status:

Checks if related components are compatible with change 5601758

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git]

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Feb 29, 2024

❌ Gradle check result for d4fce02: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Bukhtawar
Bukhtawar reviewed Mar 3, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@Bukhtawar Bukhtawar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need changes behind a Feature flag?

@gbbafna
Copy link
Copy Markdown
Contributor Author

gbbafna commented Mar 4, 2024

Do we need changes behind a Feature flag?

The mixed mode is already behind a feature flag . So by virtue of that most of the risky code is already under feature flag .

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 4, 2024

❌ Gradle check result for 9cb67fc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale
sachinpkale reviewed Mar 5, 2024
View reviewed changes
sachinpkale
sachinpkale reviewed Mar 11, 2024
View reviewed changes
ashking94
ashking94 reviewed Mar 14, 2024
View reviewed changes
ashking94
ashking94 reviewed Mar 14, 2024
View reviewed changes
ashking94
ashking94 reviewed Mar 14, 2024
View reviewed changes
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 14, 2024

❌ Gradle check result for 463eb84: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

ashking94
ashking94 reviewed Mar 14, 2024
View reviewed changes
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 15, 2024

❌ Gradle check result for 77c9b3a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@gbbafna gbbafna marked this pull request as ready for review March 15, 2024 08:39
@sachinpkale
Copy link
Copy Markdown
Member

sachinpkale commented Mar 21, 2024

Flaky test is discussed here: #12338 and not related to changes in this PR.

gbbafna added 14 commits March 21, 2024 09:32
@gbbafna
WIP changes for remote primary to remote primary relocation
891edf3 
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@gbbafna
refactor
b0607f9 
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@gbbafna
fixing flakiness
29c1614 
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@gbbafna
Moving isSegRepEnabled to IndexSetting itself
bd23d77 
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@gbbafna
Fixing UTs , some restructuring
bec0de5 
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@gbbafna
Fixing relocation issue of ckp not found when opening engine
9730194 
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@gbbafna
Address PR comments
cb716cf 
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@gbbafna
java doc check fix
24e53a5 
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@gbbafna
removing redundant logs
63574fa 
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@gbbafna
Using isRemoteDataAttributePresent for determining if node is remote
8204388 
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@gbbafna
spotless fix
d075375 
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@gbbafna
Address PR comments
72cefe9 
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@gbbafna
PR Test comments
8b20f1a 
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@gbbafna
Changing wait period for sync to 30 sec from 1 min
5601758 
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 21, 2024

❌ Gradle check result for cf5dc23: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 21, 2024

✅ Gradle check result for 5601758: SUCCESS

@sachinpkale sachinpkale merged commit f3d2bee into opensearch-project:main Mar 21, 2024
@gbbafna gbbafna added the backport 2.x Backport to 2.x branch label Mar 21, 2024
@opensearch-trigger-bot
Copy link
Copy Markdown
Contributor

opensearch-trigger-bot bot commented Mar 21, 2024

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-12494-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 f3d2beee637f63e38c8f26dbcee9f2a82f9c87b6
# Push it to GitHub
git push --set-upstream origin backport/backport-12494-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-12494-to-2.x.

This was referenced Mar 21, 2024
Closed
Closed
gbbafna added a commit to gbbafna/OpenSearch that referenced this pull request Mar 21, 2024
@gbbafna
[Remote Migration] Changes for Primary Relocation during migration (o…
91ee47d 
…pensearch-project#12494)

Changes for Primary Relocation during migration

---------

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
@gbbafna gbbafna mentioned this pull request Mar 21, 2024
Merged
7 tasks
sachinpkale pushed a commit that referenced this pull request Mar 22, 2024
@gbbafna
[Remote Migration] Changes for Primary Relocation during migration (#…
0646e90 
…12494) (#12816)

Changes for Primary Relocation during migration

---------

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
@gbbafna @shiv0408
[Remote Migration] Changes for Primary Relocation during migration (o…
2abfaa3 
…pensearch-project#12494)

Changes for Primary Relocation during migration

---------

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Bukhtawar Bukhtawar Bukhtawar left review comments

@sachinpkale sachinpkale sachinpkale approved these changes

@ashking94 ashking94 ashking94 approved these changes

@abbashus abbashus Awaiting requested review from abbashus

@adnapibar adnapibar Awaiting requested review from adnapibar

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@dreamer-89 dreamer-89 Awaiting requested review from dreamer-89

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

@nknize nknize Awaiting requested review from nknize

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@ryanbogan ryanbogan Awaiting requested review from ryanbogan

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@tlfeng tlfeng Awaiting requested review from tlfeng

@VachaShah VachaShah Awaiting requested review from VachaShah

Assignees

No one assigned

Labels

backport 2.x Backport to 2.x branch backport-failed enhancement Enhancement or improvement to existing feature or request skip-changelog Storage:Durability Issues and PRs related to the durability framework

Projects

Storage Project Board
Status: ✅ Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Remote Store Migration] Relocation of primary shards from doc rep to remote store node

4 participants

@gbbafna @sachinpkale @ashking94 @Bukhtawar