[Feature Request] Introduce a rescore search response processor

[mingshl]

Is your feature request related to a problem? Please describe

When a field with score is added by a response processor, we try to sort the response by the added field.

in the search response, there are multiple documents in hits. a rescore search response processor can take the score from a field from the document, and sort the search hits based on the field value. Users can opt to remove the field in the documents.

Describe the solution you'd like

The proposed solution as follows:

PUT /_search/pipeline/my_pipeline
{
  "response_processors": [
    {
      "rescore": {
        "field": "similarity_score",
        "sort": "desc",
        "replace_score": true 
        "remove_sort_field" : true
      }
    }
  ]
}

for example:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 4,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "diary_index",
        "_id": "1",
        "_score": 1,
        "_source": {
          "diary": "how are you",
          "similarity_score": -11.055182
        }
      },
      {
        "_index": "diary_index",
        "_id": "2",
        "_score": 1,
        "_source": {
          "diary": "today is sunny",
          "similarity_score": 8.969885
        }
      },
      {
        "_index": "diary_index",
        "_id": "3",
        "_score": 1,
        "_source": {
          "diary": "today is july fifth",
          "similarity_score": -5.736348
        }
      },
      {
        "_index": "diary_index",
        "_id": "4",
        "_score": 1,
        "_source": {
          "diary": "it is winter",
          "similarity_score": -10.045217
        }
      }
    ]
  }
}

the response after processing:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 4,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [

{
        "_index": "diary_index",
        "_id": "2",
        "_score": 8.969885,
        "_source": {
          "diary": "today is sunny"
        }
      },
      {
        "_index": "diary_index",
        "_id": "3",
        "_score": -5.736348,
        "_source": {
          "diary": "today is july fifth"
        }
      },
      {
        "_index": "diary_index",
        "_id": "4",
        "_score": -10.045217,
        "_source": {
          "diary": "it is winter"
        }
      }
      , {
        "_index": "diary_index",
        "_id": "1",
        "_score": -11.055182,
        "_source": {
          "diary": "how are you"
        }
      }
    ]
  }
}

Related component

Search

Describe alternatives you've considered

No response

Additional context

No response

[mch2]

tagging @msfroh @martin-gaievski

[msfroh]

Makes sense -- if a field was added by response processor, the only way that you can really do something with it (before sending back to the client) is with another response processor.

With the ever-present "naming is hard" challenge, I'm not sure I like the name rescore. Maybe sort? Or sort_by_field?

[msfroh]

In theory, if we really wanted to get into the "do one thing" philosophy for search processors, then we could use the remove_field response processor to remove the field after sorting, rather than including it in this processor.

Similarly/alternatively, we could add a copy response processor that copies the field value into _score. (That could also be done with a shared copy document processor, if we ever get those.) Of course, the copy ingest processor already does two things by having the remove_source parameter.

[mingshl]

Makes sense -- if a field was added by response processor, the only way that you can really do something with it (before sending back to the client) is with another response processor.

With the ever-present "naming is hard" challenge, I'm not sure I like the name rescore. Maybe sort? Or sort_by_field?

there is a sort response processor already, but it's sorting a array field in the document.

yep, naming is hard. sort_by_field is straight forward but a bit long.

[brianf-aws]

Hey y'all, I would like to work on this feature! I like Rescore since its taking a already scored value e.g. _score": 1 and changing it to something more relevant. The closest thing I can think of is refineScore as a name.

I have some questions I want to bring to the table:

1. What do we do with Tiebreakers? Should we use First item in with same score to be shown earlier?
2. How do we want to handle failure? Should we just return the values that were given before hand?
   • One thing I was thinking is when the field provided doesn't exist Do we return the previous response or stop the pipeline
3. What are some ways that users can mishandle the processor? _I'm new to crafting processors but would like some context
4. What do we do when the field to sort by field is not a singleton number?

[brianf-aws]

Adding this META GH enhancement for ML Inference