Introduce Template query by mingshl · Pull Request #16818 · opensearch-project/OpenSearch

mingshl · 2024-12-09T22:24:42Z

Description

This Pull Request introduces the following key enhancements:

1. New 'Template' Query Type:

Implements a new query type that holds an object of query content.
Supports placeholders within the query content.
Passes the query content to search request processing.

2. Query Rewrite Context Refactoring:

Query Rewrite Context now carries the pipelineContext variables after all search request processors processed.
Template queries resolve placeholders during query rewrites.
Inner queries are constructed dynamically based on resolved placeholders.

Related Issues

#16823
opensearch-project/ml-commons#3054

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

mingshl · 2024-12-09T22:25:15Z

@reta this PR addressed the issue #16823, can you please help move the issue to OpenSearch Project?

github-actions · 2024-12-09T22:29:18Z

❌ Gradle check result for ff1acc0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

mingshl · 2024-12-10T00:43:24Z

updated git rebase HEAD~3 --signoff

github-actions · 2024-12-10T00:49:51Z

❌ Gradle check result for cf97416: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

reta · 2024-12-10T14:52:27Z

@reta this PR addressed the issue opensearch-project/OpenSearch#16823, can you please help move the issue to OpenSearch Project?

@mingshl I sadly cannot do it (not a maintainers on ml-commons), @dblock @andrross could you?

mingshl · 2024-12-10T17:49:17Z

@reta this PR addressed the issue opensearch-project/OpenSearch#16823, can you please help move the issue to OpenSearch Project?

@mingshl I sadly cannot do it (not a maintainers on ml-commons), @dblock @andrross could you?

Thanks @reta. I will reach out.

github-actions · 2025-01-15T03:20:10Z

✅ Gradle check result for e5784fd: SUCCESS

codecov · 2025-01-15T03:20:34Z

Codecov Report

Attention: Patch coverage is 82.64463% with 21 lines in your changes missing coverage. Please review.

Project coverage is 72.25%. Comparing base (c6dfc65) to head (fcc44d5).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...pensearch/index/query/QueryCoordinatorContext.java	58.82%	7 Missing ⚠️
...g/opensearch/index/query/TemplateQueryBuilder.java	90.16%	3 Missing and 3 partials ⚠️
...pensearch/index/query/BaseQueryRewriteContext.java	88.57%	2 Missing and 2 partials ⚠️
...java/org/opensearch/index/query/QueryBuilders.java	0.00%	1 Missing ⚠️
...rg/opensearch/index/query/QueryRewriteContext.java	0.00%	1 Missing ⚠️
...rch/search/pipeline/PipelineProcessingContext.java	0.00%	1 Missing ⚠️
...g/opensearch/search/pipeline/PipelinedRequest.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #16818   +/-   ##
=========================================
  Coverage     72.24%   72.25%           
+ Complexity    65421    65403   -18     
=========================================
  Files          5306     5309    +3     
  Lines        304215   304300   +85     
  Branches      44118    44124    +6     
=========================================
+ Hits         219788   219866   +78     
+ Misses        66414    66377   -37     
- Partials      18013    18057   +44

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

dbwiddis

Some concerns here:

breaking API
synchronization issues
I'm not sure your substitution is doing what you think it's doing (or it's doing something not expected by a casual reader)

server/src/main/java/org/opensearch/search/SearchService.java

server/src/main/java/org/opensearch/index/query/QueryRewriteContext.java

server/src/main/java/org/opensearch/index/query/BaseQueryRewriteContext.java

server/src/main/java/org/opensearch/index/query/TemplateQueryBuilder.java

server/src/main/java/org/opensearch/index/query/BaseQueryRewriteContext.java

Make QueryRewriteContext an interface. Rename existing impl to BaseQueryRewriteContext. Create coordinator-level context called QueryCoordinatorContext. It can expose pipelined search request, including PipelineProcessingContext to rewrite methods. Signed-off-by: Mingshi Liu <mingshl@amazon.com>

Introduce template query that holds the content of query which can contain placeholders and can be filled by the variables from PipelineProcessingContext produced by search processors. This allows query rewrite by the search processors. Signed-off-by: Mingshi Liu <mingshl@amazon.com>

Signed-off-by: Mingshi Liu <mingshl@amazon.com> Add changelog Signed-off-by: Mingshi Liu <mingshl@amazon.com>

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

github-actions · 2025-01-23T20:17:21Z

❌ Gradle check result for fcc44d5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-01-23T21:54:22Z

❕ Gradle check result for fcc44d5: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

dbwiddis

LGTM thanks for your patience!

Introduce template query that holds the content of query which can contain placeholders and can be filled by the variables from PipelineProcessingContext produced by search processors. This allows query rewrite by the search processors. --------- Signed-off-by: Mingshi Liu <mingshl@amazon.com> Co-authored-by: Michael Froh <froh@amazon.com> (cherry picked from commit 13ab4ec) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Introduce template query that holds the content of query which can contain placeholders and can be filled by the variables from PipelineProcessingContext produced by search processors. This allows query rewrite by the search processors. --------- Signed-off-by: Mingshi Liu <mingshl@amazon.com> Co-authored-by: Michael Froh <froh@amazon.com> Signed-off-by: expani <anijainc@amazon.com>

Introduce template query that holds the content of query which can contain placeholders and can be filled by the variables from PipelineProcessingContext produced by search processors. This allows query rewrite by the search processors. Backport 2.x note: Instead of extracting an interface from QueryRewriteContext, we can keep it as a base class and subclass it for QueryCoordinatorContext. This avoids a breaking API change on the 2.x line. --------- Signed-off-by: Mingshi Liu <mingshl@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Michael Froh <froh@amazon.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Introduce template query that holds the content of query which can contain placeholders and can be filled by the variables from PipelineProcessingContext produced by search processors. This allows query rewrite by the search processors. --------- Signed-off-by: Mingshi Liu <mingshl@amazon.com> Co-authored-by: Michael Froh <froh@amazon.com> Signed-off-by: Eugene Tolbakov <ev.tolbakov@gmail.com>

mingshl force-pushed the template-query branch from a159416 to cf97416 Compare December 10, 2024 00:42

mingshl requested review from martin-gaievski, msfroh and rishabhmaurya January 15, 2025 18:04

dbwiddis reviewed Jan 16, 2025

View reviewed changes

msfroh and others added 7 commits January 23, 2025 11:18

Uploading SHAs files for licenses

b4aef04

Signed-off-by: Mingshi Liu <mingshl@amazon.com> Add changelog Signed-off-by: Mingshi Liu <mingshl@amazon.com>

update exception

a33560c

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

change exception handling

ada0528

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

write replaceVariables without lang3

6b6a6de

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

add empty or null check

fcc44d5

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

mingshl force-pushed the template-query branch from e5784fd to fcc44d5 Compare January 23, 2025 19:21

mingshl requested a review from cwperks as a code owner January 23, 2025 19:21

mingshl mentioned this pull request Jan 23, 2025

[BUG] Racing condition in executeAsyncActions #17103

Closed

dbwiddis approved these changes Jan 23, 2025

View reviewed changes

msfroh merged commit 13ab4ec into opensearch-project:main Jan 23, 2025

opensearch-trigger-bot bot mentioned this pull request Jan 23, 2025

[Backport 2.x] Introduce Template query #17105

Closed

ohltyler mentioned this pull request Jan 27, 2025

[ENHANCEMENT] Integrate with template query and ext improvements in ML commons opensearch-project/dashboards-flow-framework#548

Closed

opensearch-ci-bot mentioned this pull request Jan 28, 2025

[AUTOCUT] Gradle Check Flaky Test Report for WorkloadManagementIT #17154

Open

andrross mentioned this pull request Feb 27, 2025

Add @bugmakerrrrrr as maintainerrrrrr #17466

Merged

3 tasks

This was referenced Jan 13, 2026

docs: add template-query report for v2.19.0 tkykenmt/opensearch-feature-explorer#2109

Merged

[feature] Template Query tkykenmt/opensearch-feature-explorer#2034

Closed

Conversation

mingshl commented Dec 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

mingshl commented Dec 9, 2024

Uh oh!

github-actions bot commented Dec 9, 2024

Uh oh!

mingshl commented Dec 10, 2024

Uh oh!

github-actions bot commented Dec 10, 2024

Uh oh!

reta commented Dec 10, 2024

Uh oh!

mingshl commented Dec 10, 2024

Uh oh!

github-actions bot commented Jan 15, 2025

Uh oh!

codecov bot commented Jan 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dbwiddis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 23, 2025

Uh oh!

github-actions bot commented Jan 23, 2025

Uh oh!

dbwiddis left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

mingshl commented Dec 9, 2024 •

edited

Loading

codecov bot commented Jan 15, 2025 •

edited

Loading