[BUG] function score query returned an invalid (negative) score with multi match cross fields query

[snikoyo]

Describe the bug

The cross_fields scoring type can produce negative scores when some documents are missing fields.

This bug was already reported for Elasticsearch and fixed with this MR for Elasticsearch >= 8.4.

We encounter it randomly on production for certain queries when we search a field that does not exist in all documents.
It appears deterministically for these search queries.

Error message:

{ "error": { "root_cause": [ { "type": "exception", "reason": "function score query returned an invalid score: -0.05202394 for doc: 1123896" } ], "type": "search_phase_execution_exception", "reason": "all shards failed", "phase": "query", "grouped": true, "failed_shards": [ { "shard": 0, "index": "my_index", "node": "my_node", "reason": { "type": "exception", "reason": "function score query returned an invalid score: -0.05202394 for doc: 1123896" } } ] }, "status": 500 }

The document number mentioned in the error message does not exist.

To Reproduce

The steps to reproduce are in the above mentioned bug report for Elasticsearch.

Expected behavior

The function score should never return a negative value.

Plugins

None.

Host/Environment (please complete the following information):

OpenSearch version: 2.5

[macohen]

In this case, a term is in every field in all the documents: "red" and it also appears in the field "shape" which only exists in one doc.

given n = 4 (number of documents containing the term) and N = 3 (total number of documents with the field), the IDF calculation: log(1 + (N - n + 0.5) / (n + 0.5)) where n = 4, documents containing term and N = 3: log(1 + (3 - 4 + 0.5)/( 4 + 0.5)) which produces -0.11778303. I also plugged this into another calculator and got -0.0511525224. We have two things to investigate:

• How can we correct the negative value and still maintain the right relative scores?
• is the IDF calculation described in the explanation actually what's happening in the code?

DELETE shape_index 

PUT shape_index
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  },
  "mappings": {
    "dynamic": false,
    "properties": {
      "thing_id": {
        "type": "integer"
      },
      "shape": {
        "type": "text"
      },
      "color": {
        "type": "text", 
        "fields": {
          "standard": { "type": "text" },  
          "raw": { "type": "keyword" }     
        }
      }
    }
  }
}


PUT /shape_index/_doc/1
{
  "thing_id": 1,
  "color": "orange red yellow"
}



PUT /shape_index/_doc/2
{
  "thing_id": 2,
  "shape": "red square",
  "color": "orange red yellow"
}


PUT /shape_index/_doc/3
{
  "thing_id": 3,
  "color": "orange red yellow"
}


PUT /shape_index/_doc/4
{
  "thing_id": 4,
  "color": "orange red yellow"
}


PUT /shape_index/_doc/5
{
  "thing_id": 5,
  "color": "orange red yellow"
}


PUT /shape_index/_doc/6
{
  "thing_id": 6,
  "color": "orange red yellow"
}


PUT /shape_index/_doc/7
{
  "thing_id": 7,
  "color": "orange red yellow"
}


PUT /shape_index/_doc/8
{
  "thing_id": 8,
  "color": "orange red yellow"
}


PUT /shape_index/_doc/9
{
  "thing_id": 9,
  "color": "orange red yellow"
}


PUT /shape_index/_doc/10
{
  "thing_id": 10,
  "color": "orange red yellow"
}


PUT /shape_index/_doc/11
{
  "thing_id": 11,
  "color": "orange red yellow"
}


GET /shape_index/_search
{
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "must": [
            {
              "multi_match": {
                "query": "red",
                "type": "cross_fields",
                "fields": [
                  "color^5",
                  "shape^2"                  
                ],
                "slop": 10
              }
            }
          ]
        }
      },
      "functions": []
    }
  },
  "from": 0,
  "size": 10,
  "explain": true
}

extract from results with explain:

{
              "value": -0.4462871,
              "description": "*weight(shape:red in 0) [PerFieldSimilarity], result of:*",
              "details": [
                {
                  "value": -0.4462871,
                  "description": "score(freq=1.0), computed as boost * idf * tf from:",
                  "details": [
                    {
                      "value": 4.4,
                      "description": "boost",
                      "details": []
                    },
                    {
                      "value": -0.22314355,
                      "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                      "details": [
                        {
                          "value": 2,
                          "description": "n, number of documents containing term",
                          "details": []
                        },
                        {
                          "value": 1,
                          "description": "N, total number of documents with field",
                          "details": []
                        }
                      ]
                    },
                    {
                      "value": 0.45454544,
                      "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                      "details": [
                        {
                          "value": 1,
                          "description": "freq, occurrences of term within document",
                          "details": []
                        },
                        {
                          "value": 1.2,
                          "description": "k1, term saturation parameter",
                          "details": []
                        },
                        {
                          "value": 0.75,
                          "description": "b, length normalization parameter",
                          "details": []
                        },
                        {
                          "value": 2,
                          "description": "dl, length of field",
                          "details": []
                        },
                        {
                          "value": 2,
                          "description": "avgdl, average length of field",
                          "details": []
                        }
                      ]
                    }
                  ]
                }
              ]
            }

[dblock]

@snikoyo please do help us with a fix if you have time, making sure we're not copying any non-APLv2 code of course

[msfroh]

{
"value": 2,
"description": "n, number of documents containing term",
"details": []
},
{
"value": 1,
"description": "N, total number of documents with field",
"details": []
}

How do more documents have the term than have the field?! head-asplode