[Segment Replication] Update force segment replication round to be synchronous by dreamer-89 · Pull Request #5898 · opensearch-project/OpenSearch

dreamer-89 · 2023-01-17T06:19:16Z

Description

This change fixes the primary relocation path with segrep. This changes:

Segment replication to happen synchronously. This has side effects of writes happening on older primary and segment replication to timeout due to parallel writes on older primary and round of segment replication from older primary to target.
Remove StepListener listener post above change. This needed corresponding changes in function definitions & unit test changes.
Updates existing integration test to reduce flakyness
Verified newly added tests are not flaky

Issues Resolved

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

dreamer-89 · 2023-01-17T06:20:15Z

Converted to draft as it needs more unit test changes and a stablized integration test which verifies delay operations behavior.

dreamer-89 · 2023-01-19T02:15:28Z

Tested the change on a 3 data-node 1 master cluster for below use-cases where I didn't find any issue.

Large primary shard relocation. With no activity happening during the relocation, the round of segment replication completes in few ms. Steps gist link.
Concurrent ingestion during relocation. Noticed that not all document show up post relocation even though all documents were Ack'ed. Steps gist link. This does not happen with doc rep enabled indices. Cut a separate issue as I found this happening in other scenarios as well. [Segment Replication] [BUG] Missing documents post ingestion during primary failover and relocation #5946

dreamer-89 · 2023-01-24T20:16:10Z

@ashking94 @Bukhtawar @mch2 : Gentle reminder on review.

...er/src/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationIT.java

...ernalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationRelocationIT.java

server/src/main/java/org/opensearch/indices/recovery/RemoteRecoveryTargetHandler.java

dreamer-89 · 2023-01-26T18:34:56Z

@ashking94 @Bukhtawar @mch2 : Gentle reminder on review.

@ashking94 @Bukhtawar : Ping for review ^

mch2

Looks good. Thanks for fixing this!

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Signed-off-by: Suraj Singh <surajrider@gmail.com> pick da8cb72ab4f Update unit test post rebase

Signed-off-by: Suraj Singh <surajrider@gmail.com>

github-actions · 2023-01-27T22:11:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10078/
CommitID: fb0b5f2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-27T22:24:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/10082/
CommitID: 2084a84

opensearch-trigger-bot · 2023-01-27T22:35:27Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-5898-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ebb5813273313d8ebbc4df8576c713651212d3ed
# Push it to GitHub
git push --set-upstream origin backport/backport-5898-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-5898-to-2.x.

…nchronous (opensearch-project#5898) * Update force segment replication sync to be synchronous Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add logs and fix spotlessApply Signed-off-by: Suraj Singh <surajrider@gmail.com> pick da8cb72ab4f Update unit test post rebase * Update unit test post rebase Signed-off-by: Suraj Singh <surajrider@gmail.com> * Update integration tests Signed-off-by: Suraj Singh <surajrider@gmail.com> * Mute testPrimaryRelocationWithSegRepFailure Signed-off-by: Suraj Singh <surajrider@gmail.com> * Remove extra closing bracket after main merge Signed-off-by: Suraj Singh <surajrider@gmail.com> * PR feedback Signed-off-by: Suraj Singh <surajrider@gmail.com> * Spotless fix Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com>

dreamer-89 requested review from Bukhtawar, CEHENKLE, Rishikesh1159, VachaShah, adnapibar, anasalkouz, andrross, dblock, gbbafna, kartg, kotwanikunal, mch2, nknize, owaiskazi19, reta, ryanbogan, saratvemulapalli, setiah, shwetathareja, tlfeng and xuezhou25 as code owners January 17, 2023 06:19

dreamer-89 marked this pull request as draft January 17, 2023 06:19

dreamer-89 added the skip-changelog label Jan 17, 2023

This comment was marked as outdated.

Sign in to view

dreamer-89 force-pushed the Issue_5848 branch from 4745c50 to 2542a56 Compare January 19, 2023 02:18

dreamer-89 marked this pull request as ready for review January 19, 2023 02:19

This comment was marked as outdated.

Sign in to view

mch2 reviewed Jan 25, 2023

View reviewed changes

dreamer-89 mentioned this pull request Jan 26, 2023

[Segment Replication] Revisit network timeout settings to avoid timeout exceptions #6027

Closed

mch2 approved these changes Jan 27, 2023

View reviewed changes

dreamer-89 added 6 commits January 27, 2023 09:17

Update force segment replication sync to be synchronous

8446946

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Add logs and fix spotlessApply

d5e44de

Signed-off-by: Suraj Singh <surajrider@gmail.com> pick da8cb72ab4f Update unit test post rebase

Update unit test post rebase

790627d

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Update integration tests

e2df180

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Mute testPrimaryRelocationWithSegRepFailure

f336cf3

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Remove extra closing bracket after main merge

419c3aa

Signed-off-by: Suraj Singh <surajrider@gmail.com>

dreamer-89 force-pushed the Issue_5848 branch from b6a9b1a to c9dc2e7 Compare January 27, 2023 20:14

This comment was marked as outdated.

Sign in to view

PR feedback

fb0b5f2

Signed-off-by: Suraj Singh <surajrider@gmail.com>

dreamer-89 force-pushed the Issue_5848 branch from c9dc2e7 to fb0b5f2 Compare January 27, 2023 21:48

Spotless fix

2084a84

Signed-off-by: Suraj Singh <surajrider@gmail.com>

dreamer-89 merged commit ebb5813 into opensearch-project:main Jan 27, 2023

dreamer-89 added the backport 2.x Backport to 2.x branch label Jan 27, 2023

dreamer-89 mentioned this pull request Jan 27, 2023

Fix flaky SegmentReplicationITs. #6015

Merged

6 tasks

mch2 mentioned this pull request Jan 27, 2023

[BUG] Flaky Test failure GeoTileGridIT.testGeoShapes #6054

Closed

dreamer-89 mentioned this pull request Jan 27, 2023

[Backport 2.x] Fix flaky SegmentReplicationITs. #6053

Closed

This was referenced Jan 28, 2023

[Segment Replication] Backport PRs from main #6055

Merged

[BUG][Segment Replication] Stuck Initializing Shard w/ Segment Replication enabled #6084

Closed

minalsha mentioned this pull request Jan 31, 2023

[BUG] Flaky Test failure GeoTileGridIT.testGeoShapes opensearch-project/geospatial#212

Closed

dreamer-89 mentioned this pull request Feb 10, 2023

Fix flaky segrep test testPrimaryStopped_ReplicaPromoted #6282

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Segment Replication] Update force segment replication round to be synchronous#5898

[Segment Replication] Update force segment replication round to be synchronous#5898
dreamer-89 merged 8 commits intoopensearch-project:mainfrom
dreamer-89:Issue_5848

dreamer-89 commented Jan 17, 2023 •

edited

Loading

Uh oh!

dreamer-89 commented Jan 17, 2023

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

dreamer-89 commented Jan 19, 2023 •

edited

Loading

Uh oh!

This comment was marked as outdated.

dreamer-89 commented Jan 24, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dreamer-89 commented Jan 26, 2023

Uh oh!

mch2 left a comment

Uh oh!

This comment was marked as outdated.

github-actions bot commented Jan 27, 2023

Uh oh!

github-actions bot commented Jan 27, 2023

Uh oh!

opensearch-trigger-bot bot commented Jan 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

dreamer-89 commented Jan 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Check List

Uh oh!

dreamer-89 commented Jan 17, 2023

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

dreamer-89 commented Jan 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

dreamer-89 commented Jan 24, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dreamer-89 commented Jan 26, 2023

Uh oh!

mch2 left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

github-actions bot commented Jan 27, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 27, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

opensearch-trigger-bot bot commented Jan 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dreamer-89 commented Jan 17, 2023 •

edited

Loading

dreamer-89 commented Jan 19, 2023 •

edited

Loading