Document new Split and Sort SearchResponseProcessors

[dbwiddis]

Description

Documents the new Sort and Split SearchResponseProcessors.

Fixes an incorrect field name in the Ingest Split Processor docs.

Issues Resolved

Fixes #7743
Fixes #7744
Fixes #7754

Feature PRs:

• Add SortResponseProcessor to Search Pipelines OpenSearch#14785
• Add SplitResponseProcessor to Search Pipelines OpenSearch#14800

Version

2.16.0

Frontend features

N/A

Checklist

• By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
  For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

[dbwiddis]

@kolchfa-aws Ready for doc review!

[kolchfa-aws]

Thank you, @dbwiddis! I edited in place. Moving to editorial review.

[natebower]

@dbwiddis @kolchfa-aws Please see my changes and let me know if you have any questions. Thanks!

[dbwiddis]

@dbwiddis @kolchfa-aws Please see my changes and let me know if you have any questions. Thanks!

Accepted all the recommendations!

[dbwiddis]

Note I'm still waiting for the PRs to get merged (hopefully by today) :)

[kolchfa-aws]

No problem. If they are not merged, we can revert this PR 😄

[dbwiddis]

Well one is now merged, other should be shortly, so we should be gtg!

[dbwiddis]

And, the other one is merged. 🎉

[mingshl]

@dbwiddis thanks for contributing the sort processor. I saw the functionality is very similar to the sort ingest processor.

But there is a difference between ingest and response: in the search response, there are multiple documents. Can we achieve sorting the documents in the response based on the value in sort_field?

for example: a sample search response:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 4,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "diary_index",
        "_id": "1",
        "_score": 1,
        "_source": {
          "diary": "how are you",
          "similarity_score": -11.055182
        }
      },
      {
        "_index": "diary_index",
        "_id": "2",
        "_score": 1,
        "_source": {
          "diary": "today is sunny",
          "similarity_score": 8.969885
        }
      },
      {
        "_index": "diary_index",
        "_id": "3",
        "_score": 1,
        "_source": {
          "diary": "today is july fifth",
          "similarity_score": -5.736348
        }
      },
      {
        "_index": "diary_index",
        "_id": "4",
        "_score": 1,
        "_source": {
          "diary": "it is winter",
          "similarity_score": -10.045217
        }
      }
    ]
  }
}

can we sort by the similarity_score field using sort processor?
so the target response is sorting by similarity_score in descending order?

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 4,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [

{
        "_index": "diary_index",
        "_id": "2",
        "_score": 1,
        "_source": {
          "diary": "today is sunny",
          "similarity_score": 8.969885
        }
      },
      {
        "_index": "diary_index",
        "_id": "3",
        "_score": 1,
        "_source": {
          "diary": "today is july fifth",
          "similarity_score": -5.736348
        }
      },
      {
        "_index": "diary_index",
        "_id": "4",
        "_score": 1,
        "_source": {
          "diary": "it is winter",
          "similarity_score": -10.045217
        }
      }
      , {
        "_index": "diary_index",
        "_id": "1",
        "_score": 1,
        "_source": {
          "diary": "how are you",
          "similarity_score": -11.055182
        }
      }
    ]
  }
}

[dbwiddis]

@mingshl that's not the spec I was asked to write to, it seems you'd want a "sort hits by field" functionality.

But aren't hits supposed to be sorted by score?

If you need a different functionality, please open a new issue to discuss it. Unfortunately we're within 24 hours of code freeze so there's no way it will get into 2.17.0.

[mingshl]

@mingshl that's not the spec I was asked to write to, it seems you'd want a "sort hits by field" functionality.

But aren't hits supposed to be sorted by score?

If you need a different functionality, please open a new issue to discuss it. Unfortunately we're within 24 hours of code freeze so there's no way it will get into 2.17.0.

We will need a new RFC to track the rank processor to sort the search hits based on the field in the hit.

[mingshl]

@mingshl that's not the spec I was asked to write to, it seems you'd want a "sort hits by field" functionality.
But aren't hits supposed to be sorted by score?
If you need a different functionality, please open a new issue to discuss it. Unfortunately we're within 24 hours of code freeze so there's no way it will get into 2.17.0.

We will need a new RFC to track the rank processor to sort the search hits based on the field in the hit.

Raised new RFC opensearch-project/OpenSearch#15631