Stream agg#18930

Closed

harshavamsi wants to merge 76 commits intoopensearch-project:mainfrom

harshavamsi:stream-agg

Contributor

harshavamsi commented Aug 5, 2025

Description

[Describe what this change achieves]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

rishabhmaurya added 30 commits

August 1, 2025 11:27


          vectorized version of StreamInput and StreamOutput

f3d35d1

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Fix for the fetch phase optimization

80aa54c

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Fix issues at flight transport layer; Add middleware for header manag…

0c17227

…ement

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Fix race condition with header in flight transport

eff27e9

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Refactor; gradle check fixes

f66c735

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Add stats API

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Stats API refactor; Cancellation of stream through StreamTransportRes…

764b8ab

…ponse

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Added base test class for stream transport and tests for FlightClient…

09b994d

…Channel

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Fix tests due to null stream transport passed to StubbableTransport

1c500b2

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Fix the failing tests due to connection profile missing STREAM type

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          cancellation and timeout fixes; fixes for resource cleanup; more test…

46e6992

…s; documentation update

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Increase latch await time for early cancellation test to fix flakiness

14c3646

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          improve javadocs; code refactor

74b8a49

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          fix issues in flight client channel; added docs on usage; standardize…

97c76aa

… the exceptions

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          pass along request Id from OutboundHandler to TcpChannel; refactor Fl…

04d1437

…ightTransportResponse for header management; more tests; update docs

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          code coverage

138d35f

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          API changes for stream transport

263ec94

* extensibility for transport classes
* StreamTransport and StreamTransportService implementation
* streaming based search action

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          update docs

8a40862

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Standardize error handling

c18ba77

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          stream transport metrics and integration

815c09a

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          unit tests for metrics

bccccb3

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Fixes related to security and FGAC

d1738dd

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Chaos IT and fixes on resource leaks like reader context cleanup afte…

2bd02df

…r search

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          register stream default timeout setting

ac5512a

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          test stability and latch timeout settings

f2acdc9

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          pr comment: nitpick

558ddcf

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          aggregation ser/de changes not required anymore

18db622

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Add changelog

ff125ff

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          Allow flight server to bind to multiple addresses

5a7b90a

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>


          example plugin to demonstrate defining stream based transport action

04dbe86

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>

harshavamsi requested review from a team, Rishikesh1159, VachaShah, kotwanikunal, mch2, msfroh, owaiskazi19, reta, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners

August 5, 2025 18:37


          clean up

a759dc5

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

Contributor

github-actions bot commented Aug 5, 2025

❌ Gradle check result for ccf9df6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?


          Add tests for StreamingStringTermsAggregator and SendBatch

66ddce7

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>
Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

This was referenced Aug 5, 2025

[AUTOCUT] Gradle Check Flaky Test Report for SmokeTestMultiNodeClientYamlTestSuiteIT #14408

Open

[AUTOCUT] Gradle Check Flaky Test Report for RemoteRestoreSnapshotIT #14324

Open

bowenlan-amzn added 3 commits

August 5, 2025 14:57


          Clean up and address comments

43bce78

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>


          spotless

6052ff5

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>


          Merge branch 'main' into stream-agg

044385f

harshavamsi force-pushed the stream-agg branch from ccf9df6 to 5432891 Compare

August 5, 2025 22:46

bowenlan-amzn and others added 3 commits

August 5, 2025 16:01


          add comment

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>


          Refactor StreamStringTermsAggregator

75bb6d2

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>


          Unblock prepareStreamSearch in NodeClient

0a738ee

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

harshavamsi force-pushed the stream-agg branch from 424347f to 0a738ee Compare

August 5, 2025 23:34

harshavamsi requested a review from peternied as a code owner

August 5, 2025 23:34

Contributor

github-actions bot commented Aug 6, 2025

✅ Gradle check result for 0a738ee: SUCCESS

codecov bot commented Aug 6, 2025 •

edited

Loading

Codecov Report

❌ Patch coverage is 74.00612% with 85 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.85%. Comparing base (3a5da00) to head (0a738ee).
⚠️ Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
.../search/StreamSearchQueryThenFetchAsyncAction.java	53.84%	17 Missing and 7 partials ⚠️
...ions/bucket/terms/StreamStringTermsAggregator.java	81.11%	11 Missing and 6 partials ⚠️
.../search/aggregations/BucketCollectorProcessor.java	60.00%	5 Missing and 3 partials ⚠️
...pensearch/rest/action/search/RestSearchAction.java	12.50%	6 Missing and 1 partial ⚠️
...rch/action/search/StreamTransportSearchAction.java	54.54%	3 Missing and 2 partials ⚠️
...ket/terms/GlobalOrdinalsStringTermsAggregator.java	61.53%	5 Missing ⚠️
...ch/action/search/StreamSearchTransportService.java	69.23%	3 Missing and 1 partial ⚠️
...regations/bucket/terms/TermsAggregatorFactory.java	72.72%	1 Missing and 2 partials ⚠️
...arch/action/search/StreamSearchActionListener.java	83.33%	1 Missing and 1 partial ⚠️
...va/org/opensearch/search/DefaultSearchContext.java	77.77%	0 Missing and 2 partials ⚠️
... and 7 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18930      +/-   ##
============================================
+ Coverage     72.83%   72.85%   +0.02%     
- Complexity    69116    69254     +138     
============================================
  Files          5633     5637       +4     
  Lines        317959   318249     +290     
  Branches      45986    46033      +47     
============================================
+ Hits         231590   231876     +286     
- Misses        67549    67623      +74     
+ Partials      18820    18750      -70

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

harshavamsi closed this

harshavamsi deleted the stream-agg branch

November 25, 2025 20:50

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

andrross Awaiting requested review from andrross andrross is a code owner

ashking94 Awaiting requested review from ashking94 ashking94 is a code owner

Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

cwperks Awaiting requested review from cwperks cwperks is a code owner

dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

jed326 Awaiting requested review from jed326 jed326 is a code owner

kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

mch2 Awaiting requested review from mch2 mch2 is a code owner

msfroh Awaiting requested review from msfroh msfroh is a code owner

owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

reta Awaiting requested review from reta reta is a code owner

Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

sohami Awaiting requested review from sohami sohami is a code owner

VachaShah Awaiting requested review from VachaShah

peternied Awaiting requested review from peternied peternied is a code owner

Labels

None yet