Performance degradation in OpenSearch 1.1.0 due to Lucene 8.9.0

[rtiruveedulaarkin]

Is your feature request related to a problem? Please describe.
We are planning to upgrade from Elasticsearch 7.7.1 to OpenSearch 1.2.4 release. We have compared the performance of OpenSearch 1.2.4 with Elasticsearch 7.7.1. For cardinality queries (with keyword fields), the performance is degraded by 50%. So we couldn't upgrade to OpenSearch.

The performance degradation is observed from in OpenSearch 1.1.0 release onwards.

Below is the code snippet which is running slow in OpenSearch 1.1.0.

    // with opensearch 1.0.1: 240 requests/second
    // with opensearch 1.1.0:  97 requests/second
    public static SearchRequestBuilder getSearchRequest1(TransportClient client, String index, String randomValue) {
        QueryBuilder qb = QueryBuilders.boolQuery().mustNot(QueryBuilders.termQuery("__id.keyword", randomValue));
        CardinalityAggregationBuilder agg = AggregationBuilders
                .cardinality("somename")
                .field("__id.keyword");
        return client.prepareSearch(index).setQuery(qb).addAggregation(agg);
    }

This degradation is caused due to lucene upgrade from 8.8.2 to 8.9.0. The commit is e153629

Lucene developer said this is caused due to an enhancement in Lucene and it has to be fixed in OpenSearch.

Lucene ticket that I have filed: https://issues.apache.org/jira/browse/LUCENE-10509

Describe the solution you'd like

As per Lucene developer (Adrien Grand):

The cardinality aggregation performs value lookups on each document. OpenSearch should change the way cardinality aggregations run to collect matching ordinals into a bitset first, and only look up values once the entire segment has been collected. This should address the performance problem and will likely make the cardinality aggregation faster than it was before Lucene 8.9.

Describe alternatives you've considered
None

Additional context

[andrross]

@rtiruveedulaarkin Have you tested against OpenSearch 1.2 that uses Lucene 8.10.1? We had a similar issue recently (#1647) that was mitigated in Lucene 8.10.1. The Lucene issue you linked doesn't show any tests against 8.10.1, so I'm curious if LUCENE-9917 also mitigates the regression seen here.

[rtiruveedulaarkin]

@andrross I have tested against OpenSearch 1.2.4 and 1.3.1. Both are using lucene 8.10.1. The performance issue is present in both these releases.

[andrross]

@rtiruveedulaarkin And can you confirm that you're using the default BEST_SPEED codec in your tests? It is expected that BEST_COMPRESSION codec can be slower with the newer versions of Lucene.

[rtiruveedulaarkin]

@andrross I haven't specified codec while creating index. So the index must be using the default best_speed.

[andrross]

Thanks @rtiruveedulaarkin

We'll dig into this soon to reproduce it. The mitigation suggested in the Lucene ticket seems like a good optimization to implement in any case.

[rishabhmaurya]

@rtiruveedulaarkin can you set profile as true and post the following debug info for CardinalityAggregator -

public void collectDebugInfo(BiConsumer<String, Object> add) {
    super.collectDebugInfo(add);
    add.accept("empty_collectors_used", emptyCollectorsUsed);
    add.accept("numeric_collectors_used", numericCollectorsUsed);
    add.accept("ordinals_collectors_used", ordinalsCollectorsUsed);
    add.accept("ordinals_collectors_overhead_too_high", ordinalsCollectorsOverheadTooHigh);
    add.accept("string_hashing_collectors_used", stringHashingCollectorsUsed);
}

[rtiruveedulaarkin]

@rishabhmaurya This is the debug info for CardinalityAggregator.

  "debug": {
      "empty_collectors_used": 0,
      "numeric_collectors_used": 0,
      "ordinals_collectors_overhead_too_high": 1,
      "ordinals_collectors_used": 0,
      "string_hashing_collectors_used": 1
  },