Implemented computation of segment replication stats at shard level#17055
Merged
mch2 merged 34 commits intoopensearch-project:mainfrom Feb 27, 2025
Merged
Implemented computation of segment replication stats at shard level#17055mch2 merged 34 commits intoopensearch-project:mainfrom
mch2 merged 34 commits intoopensearch-project:mainfrom
Conversation
The method implemented here computes the segment replication stats at the shard level, instead of relying on the primary shard to compute stats based on reports from its replicas. Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
vinaykpud
requested review from
Bukhtawar,
CEHENKLE,
Rishikesh1159,
VachaShah,
anasalkouz,
andrross,
ashking94,
dblock,
dbwiddis,
gbbafna,
jainankitk,
jed326,
kotwanikunal,
linuxpi,
mch2,
msfroh,
nknize,
owaiskazi19,
reta,
sachinpkale,
saratvemulapalli,
shwetathareja and
sohami
as code owners
github-actions
bot
added
enhancement
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
vinaykpud
force-pushed
the
node-stats
branch
from
7ff35b8 to
d89fdce
Compare
vinaykpud
force-pushed
the
node-stats
branch
from
d89fdce to
fcc57bf
Compare
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
Contributor
|
❌ Gradle check result for 25fd006: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
mch2
approved these changes
Feb 26, 2025
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
Contributor
|
❌ Gradle check result for d8585f7: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Contributor
|
❕ Gradle check result for d8585f7: UNSTABLE Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
mch2
approved these changes
Feb 27, 2025
Contributor
|
The backport to To backport manually, run these commands in your terminal: # Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-17055-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ee7fbbd226b2be81128eaafe19aad0a39244368c
# Push it to GitHub
git push --set-upstream origin backport/backport-17055-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.xThen, create a pull request where the |
Member
|
ignoring backport here, this is a 3.0 only change. |
vinaykpud
added a commit
to vinaykpud/OpenSearch
that referenced
this pull request
Mar 18, 2025
…pensearch-project#17055) * Implemented computation of segment replication stats at shard level The method implemented here computes the segment replication stats at the shard level, instead of relying on the primary shard to compute stats based on reports from its replicas. Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Updated style checks in the test Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Updated changelog Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * fixed style issues Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Fix the failing integration test Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Fix stylecheck Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Fixed the comments for the initial revision Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Updated to use System.nanoTime() for lag calculation Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Fixed the integration test for node stats Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Modified the version in the ReplicationCheckpoint for backward compatibility Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Added precomputation logic for the stats calculation Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Removed unwanted lines Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Clean up the maps when index closed Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Added a null check for the indexshard checkpoint Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * fix style checks Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Updated version and added bwc for RemoteSegmentMetadata Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Upated the javadoc comments Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Address comments PR Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Removed the latestReceivedCheckpoint map from SegmentReplicationTargetService Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Added granular locks for the concurrency of stats methods Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Style check fixes Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Changes to maintain atomicity Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * spotlessApply Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * removed querying the remotestore when replication is in progress Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * spotlessApply Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> --------- Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The method implemented here computes the segment replication stats at the shard level, instead of relying on the primary shard to compute stats based on reports from its replicas.
Method implemented in this PR serves the segment replication stats for following core APIs:
/_nodes/stats)/_cluster/stats)/_statsor/{index}/_stats)Related Issues
Resolves #16801
Related to #15306
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.