Skip to content

[Backport 2.x] Make search pipelines asynchronous#11010

Merged
reta merged 1 commit intoopensearch-project:2.xfrom
msfroh:backport/backport-10598-to-2.x
Oct 31, 2023
Merged

[Backport 2.x] Make search pipelines asynchronous#11010
reta merged 1 commit intoopensearch-project:2.xfrom
msfroh:backport/backport-10598-to-2.x

Conversation

@msfroh
Copy link
Copy Markdown
Contributor

@msfroh msfroh commented Oct 30, 2023

Description

Backport of #10598 to 2.x

(Conflicts on changelog needed to be resolved.)

Related Issues

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@msfroh
Make search pipelines asynchronous (opensearch-project#10598)
9de561c 
* Make search pipelines asynchronous

If a search processor needs to make a call out to another
service, we should not risk blocking on the transport thread. We should
support async execution.

Signed-off-by: Michael Froh <froh@amazon.com>

* Compute pipelineStart before building request callback chain

Also, IntelliJ suggested refactoring creation of the terminal request
callback into a separate method since the existing method was really
big. I liked that suggestion.

Signed-off-by: Michael Froh <froh@amazon.com>

* Rename async methods (put async at end)

Signed-off-by: Michael Froh <froh@amazon.com>

---------

Signed-off-by: Michael Froh <froh@amazon.com>
(cherry picked from commit da011ba)
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Oct 30, 2023

Compatibility status:

Checks if related components are compatible with change 9de561c

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Oct 30, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Oct 31, 2023

Gradle Check (Jenkins) Run Completed with:

@codecov
Copy link
Copy Markdown

codecov bot commented Oct 31, 2023 •
edited
Loading

Codecov Report

Merging #11010 (9de561c) into 2.x (0f7649f) will increase coverage by 0.09%.
Report is 5 commits behind head on 2.x.
The diff coverage is 75.11%.

@@             Coverage Diff              @@
##                2.x   #11010      +/-   ##
============================================
+ Coverage     70.87%   70.97%   +0.09%     
- Complexity    58747    58801      +54     
============================================
  Files          4845     4847       +2     
  Lines        277517   277635     +118     
  Branches      40691    40705      +14     
============================================
+ Hits         196687   197043     +356     
+ Misses        64138    63824     -314     
- Partials      16692    16768      +76
Files Coverage Δ
...ensearch/action/search/SearchQueryCategorizer.java 96.77% <100.00%> (ø)
...rg/opensearch/common/settings/ClusterSettings.java 92.85% <ø> (ø)
...rg/opensearch/index/translog/RemoteFsTranslog.java 76.30% <100.00%> (+0.69%) ⬆️
...g/opensearch/search/pipeline/PipelinedRequest.java 100.00% <100.00%> (ø)
...nsearch/search/pipeline/SearchPipelineService.java 85.51% <100.00%> (-0.07%) ⬇️
...search/search/pipeline/SearchRequestProcessor.java 100.00% <100.00%> (ø)
...earch/search/pipeline/SearchResponseProcessor.java 100.00% <100.00%> (ø)
.../java/org/opensearch/search/pipeline/Pipeline.java 85.81% <96.62%> (+2.98%) ⬆️
...pensearch/action/search/TransportSearchAction.java 68.07% <70.58%> (+1.47%) ⬆️
...search/index/shard/RemoteStoreRefreshListener.java 84.90% <37.50%> (-1.93%) ⬇️
... and 4 more

... and 445 files with indirect coverage changes

@msfroh msfroh marked this pull request as ready for review October 31, 2023 00:17
@reta reta merged commit 3b0b8f0 into opensearch-project:2.x Oct 31, 2023
@reta reta added the v2.12.0 Issues and PRs related to version 2.12.0 label Oct 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@reta reta reta approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@setiah setiah Awaiting requested review from setiah

@kartg kartg Awaiting requested review from kartg

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@nknize nknize Awaiting requested review from nknize

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@peternied peternied Awaiting requested review from peternied

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@ryanbogan ryanbogan Awaiting requested review from ryanbogan

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@dreamer-89 dreamer-89 Awaiting requested review from dreamer-89

@VachaShah VachaShah Awaiting requested review from VachaShah

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

Assignees

No one assigned

Labels

v2.12.0 Issues and PRs related to version 2.12.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@msfroh @reta