[BUG] Flattened Field exists Query Issue with Dotted Sub-Field Names

[IAmFQQ]

Describe the bug

We have encountered a discrepancy between Elasticsearch and OpenSearch when querying exists on flattened fields with dotted sub-field names.

related topic #11425

Related component

Search:Query Capabilities

To Reproduce

1. Create an index with the following mapping:

PUT qiafan_test2
{
  "mappings": {
    "properties": {
      "earth": {
        "type": "flattened/flat_object" (ES/OS)
      }
    }
  }
}

2. Index a document:

POST qiafan_test2/_doc/1
{
  "earth": {
    "american.north": "what an amazing day"
  }
}

3. Run the following query:

GET qiafan_test2/_search
{
  "query": {
    "bool": {
      "boost": 1,
      "must": [
        {
          "bool": {
            "should": [
              {
                "exists": {
                  "boost": 1,
                  "field": "earth.american.north"
                }
              },
              {
                "exists": {
                  "boost": 1,
                  "field": "american.north"
                }
              }
            ]
          }
        }
      ]
    }
  }
}
 

Expected behavior

The query should return the indexed document, as the field earth.american.north exists in the document.

Observed Behavior

In Elasticsearch (7.10): The query returns the expected document.
In OpenSearch (2.18): The query does not return any results.

Additional Details

Analysis

The issue appears to be related to how OpenSearch handles exists queries on flattened fields with dotted sub-field names. Specifically:

1. exists on earth.american.north does not behave as expected.
2. The behavior differs from term queries, where such fields work correctly, as it supported in this PR Fix parsing of flat object fields with dots in keys #11425
3. american.north is not supported.

Impact

We are migrating our service from Elasticsearch 7.10 to Opensearch. This behavior prevents certain queries from functioning correctly, potentially affecting applications relying on exists queries for flattened fields.

Request

We kindly request the community to investigate this issue. If this behavior is by design, clarification in the documentation would be helpful. If it is a bug, we hope for a fix or a recommended workaround.

Thank you for your assistance!

[bugmakerrrrrr]

There are a lot of problems with the current flat_object implementation, but for compatibility reasons we need to address them in a new version, which is currently under development, you can refer to #16061.

As for your exists query, this is because the current implementation uses an inverted index under field name to perform the query, and this inverted index will only resolve the last level of subfields, e.g. for the example you gave, there will only be a mapping of earth -> earth.north, so it will give the wrong result, if we use doc values to implement exists query, we should be able to solve this problem. If we implement an exists query using doc values, we should be able to solve this problem.

[IAmFQQ]

hi @bugmakerrrrrr ,
Thanks for your suggestions, really appreciated.

If we implement an exists query using doc values, we should be able to solve this problem.

Is this referring to the implementation here, #16802
It seems this hasn’t been released in 2.18 yet, as the issue still exists in our testing on 2.18.
shall we know when it’s expected to be released?

[bugmakerrrrrr]

Is this referring to the implementation here, #16802

@IAmFQQ No, This is currently unfixed. If you don't mind, I can open a PR in the next couple days to fix it.

[IAmFQQ]

@bugmakerrrrrr
Thank you so much for your help! If you’re able to fix this in the next few days, I’ll cherry-pick and apply it. This will allow us to migrate to OpenSearch smoothly.

[sandeshkr419]

@bugmakerrrrrr Are you planning out the fix for 2.19? In that case, please take into account 2.19 code freeze on 1/28/2025.