Skip to content

Base Flight producer integration with Flight Server#17066

Closed
rishabhmaurya wants to merge 24 commits intoopensearch-project:mainfrom
rishabhmaurya:flight-producer
Closed

Base Flight producer integration with Flight Server#17066
rishabhmaurya wants to merge 24 commits intoopensearch-project:mainfrom
rishabhmaurya:flight-producer

Conversation

@rishabhmaurya
Copy link
Copy Markdown
Contributor

@rishabhmaurya rishabhmaurya commented Jan 21, 2025

Description

FlightProducer changes

Related Issues

Resolves #17065

Meta - #16679

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Search:Performance labels Jan 21, 2025
@rishabhmaurya rishabhmaurya mentioned this pull request Jan 21, 2025
Merged
1 task
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 21, 2025

❌ Gradle check result for b7d4ea5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 22, 2025

❌ Gradle check result for e4427c8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 23, 2025

❌ Gradle check result for 8cc5556: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

harshavamsi
harshavamsi reviewed Jan 23, 2025
View reviewed changes
server/src/main/java/org/opensearch/search/query/StreamSearchPhase.java
Collector collector = QueryCollectorContext.createQueryCollector(collectors);
final ArrowDocIdCollector arrowDocIdCollector = new ArrowDocIdCollector(collector, root, flushSignal, 1000);
try {
searcher.addQueryCancellation(() -> {
Copy link
Copy Markdown
Contributor

@harshavamsi harshavamsi Jan 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, the searcher is coming from QueryPhaseSearcher which already has query cancellation

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 24, 2025

❌ Gradle check result for 2db7f85: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@rishabhmaurya
Arrow Flight Server bootstrap logic
c743d29 
* new plugin for StreamManager implementation
* integration with server module
* support for SslContext in Flight server and client
* ClientManager for creating a pool of flight clients for data nodes
* custom event loop group and thread pool for server and client channel

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
interim changes - integration with Auxiliary Transport
3cc9163 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
changes to use grpc-netty-shaded
ffa51f8 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
Update javadoc
3c68363 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
fix the shaded dependencies
87c8369 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
Move arrow-flight-rpc from module to plugin
be77c68 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
remove unnecessary imports
3ae31e9 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
rebase fixes
3560c25 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
Fix permissions and other runtime issues
c538c57 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
Remove StreamManagerWrapper and Node.java changes from PR
34ae62c 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
Fix permissions for test
ca71801 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
remove testGetFlightClientLocationExecutionError as thread interrupti…
9cba7fa 
…on was causing client close failure

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
Fix the issue with single node ClientManager
7c01930 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
Fix flight server integ test on unix machine
d592451 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
suppress JSM removal warning
6f1f435 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
Fix security policy and FlightClientManagerTests
0643e3c 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
remove StreamManagerWrapper from the PR
5f71494 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
Arrow Flight Server bootstrap logic
f66223d 
* new plugin for StreamManager implementation
* integration with server module
* support for SslContext in Flight server and client
* ClientManager for creating a pool of flight clients for data nodes
* custom event loop group and thread pool for server and client channel

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
fix the shaded dependencies
4b95358 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
Flight Producer changes and integration
1c6fcc2 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya
Fix the issue with single node ClientManager
6b8a6e9 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@rishabhmaurya rishabhmaurya force-pushed the flight-producer branch 2 times, most recently from 86fd6dc to 08c15a5 Compare January 26, 2025 03:00
@rishabhmaurya
Fixes after rebase on 01/25
3ff3dd5 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 26, 2025

❌ Gradle check result for 3ff3dd5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Dec 18, 2024
Open
@rishabhmaurya
Fix concurrency issues in FlightClientManager and FlightStreamManager
2a6590f 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 30, 2025

❌ Gradle check result for 2a6590f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@rishabhmaurya
Add more integration tests for flight producer
60a3586 
Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 31, 2025

❌ Gradle check result for 60a3586: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Mar 14, 2025
Open
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Sep 8, 2025
Open
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Oct 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz will be requested when the pull request is marked ready for review anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross will be requested when the pull request is marked ready for review andrross is a code owner

@ashking94 ashking94 Awaiting requested review from ashking94 ashking94 will be requested when the pull request is marked ready for review ashking94 is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar will be requested when the pull request is marked ready for review Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE will be requested when the pull request is marked ready for review CEHENKLE is a code owner

@cwperks cwperks Awaiting requested review from cwperks cwperks will be requested when the pull request is marked ready for review cwperks is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis will be requested when the pull request is marked ready for review dbwiddis is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna will be requested when the pull request is marked ready for review gbbafna is a code owner

@jed326 jed326 Awaiting requested review from jed326 jed326 will be requested when the pull request is marked ready for review jed326 is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal will be requested when the pull request is marked ready for review kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 will be requested when the pull request is marked ready for review mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh will be requested when the pull request is marked ready for review msfroh is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 will be requested when the pull request is marked ready for review owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta will be requested when the pull request is marked ready for review reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 will be requested when the pull request is marked ready for review Rishikesh1159 is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale will be requested when the pull request is marked ready for review sachinpkale is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli will be requested when the pull request is marked ready for review saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja will be requested when the pull request is marked ready for review shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami will be requested when the pull request is marked ready for review sohami is a code owner

1 more reviewer

@harshavamsi harshavamsi harshavamsi left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

enhancement Enhancement or improvement to existing feature or request Search:Performance

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature Request] Base Flight producer for FlightServer

2 participants

@rishabhmaurya @harshavamsi