Skip to content

[Backport 2.x] Make RemoteStoreReplicationSource#getSegmentFiles asynchronous#10771

Merged
mch2 merged 1 commit into2.xfrom
backport/backport-10725-to-2.x
Oct 20, 2023
Merged

[Backport 2.x] Make RemoteStoreReplicationSource#getSegmentFiles asynchronous#10771
mch2 merged 1 commit into2.xfrom
backport/backport-10725-to-2.x

Conversation

@opensearch-trigger-bot
Copy link
Copy Markdown
Contributor

@opensearch-trigger-bot opensearch-trigger-bot bot commented Oct 20, 2023

Backport 41a12e2 from #10725.

@github-actions @andrross
Make RemoteStoreReplicationSource#getSegmentFiles asynchronous (#10725)
8fad0ea 
* Make RemoteStoreReplicationSource#getSegmentFiles asynchronous

Also make the remote store download process cancellable in case the
replication event is canceled.

Signed-off-by: Andrew Ross <andrross@amazon.com>

* Add ITs ensuring segRep targets are cleaned up on cancellation during metadata and segment fetch steps.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Wrap metadata fetch in cancellableThreads.executeIO

Signed-off-by: Marc Handalian <handalm@amazon.com>

* self review

Signed-off-by: Marc Handalian <handalm@amazon.com>

* spotless

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add missing node settings when bootstrapping nodes in tests.

Signed-off-by: Marc Handalian <handalm@amazon.com>

---------

Signed-off-by: Andrew Ross <andrross@amazon.com>
Signed-off-by: Marc Handalian <handalm@amazon.com>
Co-authored-by: Andrew Ross <andrross@amazon.com>
(cherry picked from commit 41a12e2)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Oct 20, 2023

Compatibility status:

Checks if related components are compatible with change 8fad0ea

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git]

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Oct 20, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Oct 20, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      2 org.opensearch.index.shard.RemoteIndexShardTests.classMethod
      1 org.opensearch.index.shard.RemoteIndexShardTests.testSegRepSucceedsOnPreviousCopiedFiles

@codecov
Copy link
Copy Markdown

codecov bot commented Oct 20, 2023

Codecov Report

Merging #10771 (8fad0ea) into 2.x (822b9d3) will decrease coverage by 0.07%.
Report is 6 commits behind head on 2.x.
The diff coverage is 73.00%.

@@             Coverage Diff              @@
##                2.x   #10771      +/-   ##
============================================
- Coverage     70.98%   70.91%   -0.07%     
+ Complexity    58685    58652      -33     
============================================
  Files          4839     4839              
  Lines        277012   277135     +123     
  Branches      40639    40657      +18     
============================================
- Hits         196629   196540      -89     
- Misses        63664    63851     +187     
- Partials      16719    16744      +25
Files Coverage Δ
...pensearch/common/settings/FeatureFlagSettings.java 50.00% <ø> (ø)
.../java/org/opensearch/gateway/GatewayMetaState.java 67.59% <100.00%> (-1.88%) ⬇️
...arch/gateway/remote/RemoteClusterStateService.java 69.54% <100.00%> (-0.14%) ⬇️
.../org/opensearch/index/mapper/RangeFieldMapper.java 87.25% <100.00%> (ø)
.../org/opensearch/index/mapper/RootObjectMapper.java 82.74% <100.00%> (+0.07%) ⬆️
...search/index/shard/RemoteStoreRefreshListener.java 87.56% <100.00%> (-2.86%) ⬇️
...ces/replication/PrimaryShardReplicationSource.java 96.00% <ø> (ø)
.../indices/replication/common/ReplicationTarget.java 72.50% <100.00%> (+1.44%) ⬆️
...n/java/org/opensearch/script/ScoreScriptUtils.java 1.50% <100.00%> (ø)
...nsearch/search/aggregations/support/ValueType.java 79.59% <100.00%> (+0.42%) ⬆️
... and 14 more

... and 457 files with indirect coverage changes

@mch2 mch2 merged commit eed9d99 into 2.x Oct 20, 2023
@mch2 mch2 deleted the backport/backport-10725-to-2.x branch October 20, 2023 05:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mch2 mch2 mch2 approved these changes

@reta reta Awaiting requested review from reta reta is a code owner

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@setiah setiah Awaiting requested review from setiah

@kartg kartg Awaiting requested review from kartg

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@nknize nknize Awaiting requested review from nknize

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@peternied peternied Awaiting requested review from peternied

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@ryanbogan ryanbogan Awaiting requested review from ryanbogan

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@dreamer-89 dreamer-89 Awaiting requested review from dreamer-89

@VachaShah VachaShah Awaiting requested review from VachaShah

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

Assignees

No one assigned

Labels

skip-changelog

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mch2