Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion _config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ opensearch_collection:
name: Aggregations
nav_fold: true
search-plugins:
name: Search features
name: Search
nav_fold: true
vector-search:
name: Vector search
Expand Down
2 changes: 1 addition & 1 deletion _search-plugins/async/index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
layout: default
title: Asynchronous search
nav_order: 51
nav_order: 40
parent: Improving search performance
has_children: true
redirect_from:
Expand Down
6 changes: 3 additions & 3 deletions _search-plugins/caching/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ layout: default
title: Caching
parent: Improving search performance
has_children: true
nav_order: 100
nav_order: 10
redirect_from:
- /search-plugins/caching/
---
Expand All @@ -18,9 +18,9 @@ Understanding how your data uses the cache can help improve your cluster's perfo

OpenSearch supports the following on-heap cache types:

- [**Index request cache**]({{site.url}}{{site.baseurl}}/search-plugins/caching/request-cache/): Caches the local results on each shard. This allows frequently used and potentially resource-heavy search requests to return results almost instantaneously.
- [**Index request cache**]({{site.url}}{{site.baseurl}}/search-plugins/caching/request-cache/): Caches the local results on each shard. This allows frequently used and potentially resource-heavy search requests to return results almost instantaneously.
- **Query cache**: Caches common data from similar queries at the shard level. The query cache is more granular than the request cache and can cache data to be reused in different queries.
- **Field data cache**: Caches field data and global ordinals, which are both used to support aggregations on certain field types.
- [**Field data cache**]({{site.url}}{{site.baseurl}}/search-plugins/caching/field-data-cache/): Caches field data and global ordinals, which are both used to support aggregations on certain field types.

## Additional cache stores

Expand Down
2 changes: 1 addition & 1 deletion _search-plugins/concurrent-segment-search.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
layout: default
title: Concurrent segment search
parent: Improving search performance
nav_order: 53
nav_order: 20
---

# Concurrent segment search
Expand Down
2 changes: 1 addition & 1 deletion _search-plugins/cross-cluster-search.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
layout: default
title: Cross-cluster search
nav_order: 65
nav_order: 230
redirect_from:
- /security/access-control/cross-cluster-search/
- /security-plugin/access-control/cross-cluster-search/
Expand Down
2 changes: 1 addition & 1 deletion _search-plugins/filter-search.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
layout: default
title: Filter results
parent: Search options
parent: Customizing search results
nav_order: 36
---

Expand Down
13 changes: 9 additions & 4 deletions _search-plugins/improving-search-performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,19 @@ layout: default
title: Improving search performance
nav_order: 220
has_children: true
has_toc: false
---

# Improving search performance

OpenSearch offers several ways to improve search performance:
OpenSearch offers several ways to improve search performance, from foundational optimizations to specialized techniques:

- Run resource-intensive queries asynchronously with [asynchronous search]({{site.url}}{{site.baseurl}}/search-plugins/async/).
- Store frequently accessed data in memory with [caching]({{site.url}}{{site.baseurl}}/search-plugins/caching/) for faster retrieval.

- Search segments concurrently using [concurrent segment search]({{site.url}}{{site.baseurl}}/search-plugins/concurrent-segment-search/).
- Search segments concurrently using [concurrent segment search]({{site.url}}{{site.baseurl}}/search-plugins/concurrent-segment-search/) for better resource utilization.

- Improve aggregation performance using a [star-tree index]({{site.url}}{{site.baseurl}}/search-plugins/star-tree-index/).
- Control shard selection for optimized query routing with [search shard routing]({{site.url}}{{site.baseurl}}/search-plugins/searching-data/search-shard-routing/).

- Run resource-intensive queries asynchronously with [asynchronous search]({{site.url}}{{site.baseurl}}/search-plugins/async/) to avoid timeouts.

- Improve aggregation performance using a [star-tree index]({{site.url}}{{site.baseurl}}/search-plugins/star-tree-index/) for analytical workloads.
86 changes: 26 additions & 60 deletions _search-plugins/index.md
Original file line number Diff line number Diff line change
@@ -1,98 +1,64 @@
---
layout: default
title: Search features
title: Search
nav_order: 1
has_children: false
has_toc: false
nav_exclude: true
permalink: /search-plugins/
redirect_from:
- /search-plugins/index/
keyword:
search_methods:
- heading: "Keyword (BM25) search"
description: "Find exact and close matches using traditional text search"
description: "Lexical text search using the BM25 algorithm to match and rank documents based on term frequency and document length. Find exact and close matches using traditional text search."
link: "/search-plugins/keyword-search/"
vector:
- heading: "Vector search"
description: "Search by similarity using dense or sparse vector embeddings"
description: "Similarity (k-nearest neighbor) search using dense and sparse vector embeddings to power semantic search, retrieval-augmented generation, and multimodal image search."
link: "/vector-search/"
ai:
- heading: "AI search"
description: "Build intelligent search applications using AI models"
description: "AI-powered search capabilities beyond vector embeddings. Enrich search and ingestion flows with any AI service to power the full range of AI-enhanced search use cases."
link: "/vector-search/ai-search/"
---

# Search features
# Search

OpenSearch provides many features for customizing your search use cases and improving search relevance.

## Search methods

OpenSearch supports the following search methods.
OpenSearch supports multiple search methods to meet different use cases and requirements.

### Exact matching and keywords

OpenSearch implements lexical (keyword) text search using the BM25 algorithm to match and rank documents based on term frequency and document length.

{% include cards.html cards=page.keyword %}

### Similarity and meaning

OpenSearch supports similarity (k-nearest neighbor) search using dense and sparse vector embeddings to power use cases such as semantic search, retrieval-augmented generation, and multimodal image search.

{% include cards.html cards=page.vector %}

### AI-powered search

OpenSearch supports AI-powered search capabilities beyond vector embeddings. OpenSearch's AI search enables search and ingestion flows to be enriched by any AI service to power the full range of AI-enhanced search use cases.

{% include cards.html cards=page.ai %}
{% include cards.html cards=page.search_methods %}

## Query languages

In OpenSearch, you can use the following query languages to search your data:

- [Query domain-specific language (DSL)]({{site.url}}{{site.baseurl}}/query-dsl/index/): The primary OpenSearch query language that supports creating complex, fully customizable queries.
In OpenSearch, you can use the following query languages to search your data.

- [Query string query language]({{site.url}}{{site.baseurl}}/query-dsl/full-text/query-string/): A scaled-down query language that you can use in a query parameter of a search request or in OpenSearch Dashboards.
Language | Where you can use it | Description
:--- | :--- | :---
[Query domain-specific language (DSL)]({{site.url}}{{site.baseurl}}/query-dsl/index/) | Search API, Dev Tools | The primary OpenSearch query language that supports creating complex, fully customizable queries.
[Query string query language]({{site.url}}{{site.baseurl}}/query-dsl/full-text/query-string/) | Search API `q` parameter, Discover and Dashboard search bar | A scaled-down query language whose syntax is based on Apache Lucene.
[SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/sql/index/) | SQL API, Query Workbench | A traditional query language that bridges the gap between relational database concepts and the flexibility of OpenSearch's document-oriented data storage.
[Piped Processing Language (PPL)]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index/) | PPL API, Query Workbench | The primary language used with observability in OpenSearch. PPL uses a pipe syntax that chains commands into a query.

- [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/sql/index/): A traditional query language that bridges the gap between traditional relational database concepts and the flexibility of OpenSearch’s document-oriented data storage.
For information about querying data in OpenSearch Dashboards, including the Dashboards Query Language (DQL), see [OpenSearch Dashboards query languages]({{site.url}}{{site.baseurl}}/dashboards/#query-languages).

- [Piped Processing Language (PPL)]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index/): The primary language used with observability in OpenSearch. PPL uses a pipe syntax that chains commands into a query.
## Customizing search results

- [Dashboards Query Language (DQL)]({{site.url}}{{site.baseurl}}/dashboards/dql/): A simple text-based query language for filtering data in OpenSearch Dashboards.
OpenSearch provides fundamental result handling capabilities that work with all search types. You can customize result navigation (pagination, sorting), result formatting (highlighting, field selection), query enhancement (autocomplete, did-you-mean), and result filtering. For more information, see [Customizing search results]({{site.url}}{{site.baseurl}}/search-plugins/searching-data/).

Check failure on line 48 in _search-plugins/index.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _search-plugins/index.md#L48

[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'types. You'.
Raw output
{"message": "[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'types. You'.", "location": {"path": "_search-plugins/index.md", "range": {"start": {"line": 48, "column": 88}}}, "severity": "ERROR"}

## Search performance
## Optimizing search quality

OpenSearch offers several ways to improve search performance:
OpenSearch provides comprehensive tools and features to help you measure, analyze, and improve the quality of your search results. These integrated features work together to optimize search relevance based on user behavior and machine learning. For more information, see [Optimizing search quality]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/).

- [Asynchronous search]({{site.url}}{{site.baseurl}}/search-plugins/async/): Runs resource-intensive queries asynchronously.

- [Concurrent segment search]({{site.url}}{{site.baseurl}}/search-plugins/concurrent-segment-search/): Searches segments concurrently.

## Search relevance

*Search relevance* is a measure of how well a document matches a search query. When you run a search query, OpenSearch compares the words in your query to the words in each document and assigns a relevance score based on factors such as how frequently the words appear and how closely they match. For more information, see [Relevance]({{site.url}}{{site.baseurl}}/getting-started/intro/#relevance).

To help you fine-tune and improve search relevance, OpenSearch provides several specialized features:

- [Search Relevance Workbench]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/using-search-relevance-workbench/): A suite of tools that support search quality improvements through experimentation.

- [Querqy]({{site.url}}{{site.baseurl}}/search-plugins/querqy/): Offers query rewriting capability.
## Search pipelines

- [User Behavior Insights]({{site.url}}{{site.baseurl}}/search-plugins/ubi/): Links user behavior to user queries to improve search quality.

## Search results
Search pipelines are the foundational infrastructure that enables OpenSearch's AI and vector search capabilities. They provide modular processors that can transform queries (text-to-vector conversion, ML inference, query rewriting), enhance results (reranking, RAG, field manipulation), and orchestrate complex AI workflows. For more information, see [Search pipelines]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/).

OpenSearch supports the following commonly used operations on search results:
## Improving search performance

- [Paginate]({{site.url}}{{site.baseurl}}/search-plugins/searching-data/paginate/)
- [Paginate with Point in Time]({{site.url}}{{site.baseurl}}/search-plugins/point-in-time/)
- [Sort]({{site.url}}{{site.baseurl}}/search-plugins/searching-data/sort/)
- [Highlight search terms]({{site.url}}{{site.baseurl}}/search-plugins/searching-data/highlight/)
- [Autocomplete]({{site.url}}{{site.baseurl}}/search-plugins/searching-data/autocomplete/)
- [Did-you-mean]({{site.url}}{{site.baseurl}}/search-plugins/searching-data/did-you-mean/)
OpenSearch provides various features to optimize search performance from foundational caching techniques to specialized optimizations. For more information, see [Improving search performance]({{site.url}}{{site.baseurl}}/search-plugins/improving-search-performance/).

## Search pipelines
## Cross-cluster search

You can process search queries and search results with [search pipelines]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/index/).
OpenSearch supports searching across multiple clusters to scale your search infrastructure for large deployments. For more information, see [Cross-cluster search]({{site.url}}{{site.baseurl}}/search-plugins/cross-cluster-search/).
1 change: 1 addition & 0 deletions _search-plugins/ltr/advanced-functionality.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ layout: default
title: Advanced functionality
nav_order: 80
parent: Learning to Rank
grand_parent: Optimizing search quality
has_children: false
---

Expand Down
1 change: 1 addition & 0 deletions _search-plugins/ltr/core-concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ layout: default
title: ML ranking core concepts
nav_order: 10
parent: Learning to Rank
grand_parent: Optimizing search quality
has_children: false
---

Expand Down
1 change: 1 addition & 0 deletions _search-plugins/ltr/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ layout: default
title: Common issues
nav_order: 1000
parent: Learning to Rank
grand_parent: Optimizing search quality
has_children: false
---

Expand Down
1 change: 1 addition & 0 deletions _search-plugins/ltr/feature-engineering.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ layout: default
title: Feature engineering
nav_order: 40
parent: Learning to Rank
grand_parent: Optimizing search quality
has_children: false
---

Expand Down
1 change: 1 addition & 0 deletions _search-plugins/ltr/fits-in.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ layout: default
title: Scope of the plugin
nav_order: 20
parent: Learning to Rank
grand_parent: Optimizing search quality
has_children: false
---

Expand Down
20 changes: 19 additions & 1 deletion _search-plugins/ltr/index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
---
layout: default
title: Learning to Rank
nav_order: 20
parent: Optimizing search quality
nav_order: 40
has_children: true
has_toc: false
redirect_from:
Expand All @@ -22,3 +23,20 @@
- If you are new to LTR, start with the [ML ranking core concepts]({{site.url}}{{site.baseurl}}/search-plugins/ltr/core-concepts/) documentation.
- For a quick introduction, see the demo in [hello-ltr](https://github.com/o19s/hello-ltr).
- If you are familiar with LTR, start with the [Integrating the plugin]({{site.url}}{{site.baseurl}}/search-plugins/ltr/fits-in/) documentation.

## Documentation

### Core concepts and setup

Check failure on line 29 in _search-plugins/ltr/index.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _search-plugins/ltr/index.md#L29

[OpenSearch.StackedHeadings] Do not stack headings. Insert an introductory sentence between headings.
Raw output
{"message": "[OpenSearch.StackedHeadings] Do not stack headings. Insert an introductory sentence between headings.", "location": {"path": "_search-plugins/ltr/index.md", "range": {"start": {"line": 29, "column": 1}}}, "severity": "ERROR"}
- [Core concepts]({{site.url}}{{site.baseurl}}/search-plugins/ltr/core-concepts/): Understand the fundamental concepts behind Learning to Rank.
- [Fits in]({{site.url}}{{site.baseurl}}/search-plugins/ltr/fits-in/): Learn how LTR integrates with your OpenSearch infrastructure.

### Feature engineering and model development
- [Feature engineering]({{site.url}}{{site.baseurl}}/search-plugins/ltr/feature-engineering/): Design effective features for your ranking models.
- [Working with features]({{site.url}}{{site.baseurl}}/search-plugins/ltr/working-with-features/): Create and manage feature sets.
- [Logging features]({{site.url}}{{site.baseurl}}/search-plugins/ltr/logging-features/): Collect feature data for model training.
- [Training models]({{site.url}}{{site.baseurl}}/search-plugins/ltr/training-models/): Build and train your ranking models.

### Deployment and advanced topics
- [Searching with your model]({{site.url}}{{site.baseurl}}/search-plugins/ltr/searching-with-your-model/): Deploy models in production search.
- [Advanced functionality]({{site.url}}{{site.baseurl}}/search-plugins/ltr/advanced-functionality/): Explore advanced LTR features and techniques.
- [FAQ]({{site.url}}{{site.baseurl}}/search-plugins/ltr/faq/): Common questions and troubleshooting.
1 change: 1 addition & 0 deletions _search-plugins/ltr/logging-features.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ layout: default
title: Logging feature scores
nav_order: 50
parent: Learning to Rank
grand_parent: Optimizing search quality
has_children: false
---

Expand Down
1 change: 1 addition & 0 deletions _search-plugins/ltr/searching-with-your-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ layout: default
title: Optimizing search with LTR
nav_order: 70
parent: Learning to Rank
grand_parent: Optimizing search quality
has_children: false
---

Expand Down
1 change: 1 addition & 0 deletions _search-plugins/ltr/training-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ layout: default
title: Uploading trained models
nav_order: 60
parent: Learning to Rank
grand_parent: Optimizing search quality
has_children: false
---

Expand Down
1 change: 1 addition & 0 deletions _search-plugins/ltr/working-with-features.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ layout: default
title: Working with features
nav_order: 30
parent: Learning to Rank
grand_parent: Optimizing search quality
has_children: false
---

Expand Down
4 changes: 2 additions & 2 deletions _search-plugins/querqy/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@
layout: default
title: Querqy
parent: Query rewriting
grand_parent: Search relevance
grand_parent: Optimizing search quality
has_children: false
redirect_from:
- /search-plugins/querqy/
nav_order: 210
nav_order: 60
---

# Querqy
Expand Down
4 changes: 4 additions & 0 deletions _search-plugins/search-pipelines/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,10 @@ Alternatively, you can use a temporary pipeline with a request or set a default

To learn about retrieving details for an existing search pipeline, see [Retrieving search pipelines]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/retrieving-search-pipeline/).

For information about troubleshooting search pipelines, see [Debugging search pipelines]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/debugging-search-pipeline/).

To delete an existing search pipeline, see [Deleting search pipelines]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/deleting-search-pipeline/).

## Manual and automatic processor creation

Search processors can be created manually or automatically:
Expand Down
2 changes: 1 addition & 1 deletion _search-plugins/search-relevance/compare-query-sets.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ layout: default
title: Comparing query sets
nav_order: 12
parent: Search Relevance Workbench
grand_parent: Search relevance
grand_parent: Optimizing search quality
has_children: false
---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ layout: default
title: Comparing single queries
nav_order: 10
parent: Search Relevance Workbench
grand_parent: Search relevance
grand_parent: Optimizing search quality
has_children: false
---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ layout: default
title: Comparing search results
nav_order: 11
parent: Search Relevance Workbench
grand_parent: Search relevance
grand_parent: Optimizing search quality
---

# Comparing search results
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ layout: default
title: Evaluating search quality
nav_order: 50
parent: Search Relevance Workbench
grand_parent: Search relevance
grand_parent: Optimizing search quality
has_children: false
---

Expand Down
2 changes: 1 addition & 1 deletion _search-plugins/search-relevance/experiments.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ layout: default
title: Experiments
nav_order: 9
parent: Search Relevance Workbench
grand_parent: Search relevance
grand_parent: Optimizing search quality
has_children: false
has_toc: false
---
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ layout: default
title: Exploring search evaluation results
nav_order: 65
parent: Search Relevance Workbench
grand_parent: Search relevance
grand_parent: Optimizing search quality
has_children: false
---

Expand Down
Loading