[Concurrent Segment Search] Doc count error needs to be computed at the slice level

[jed326]

Is your feature request related to a problem? Please describe

For #9246 we forced doc count error to be 0 during the shard level reduce phase as we were not eliminating any buckets at that stage. However, this logic was changed to use a slice_size = shard_size * 1.5 + 10 heuristic as a part of #11585. This means that it's now possible to eliminate bucket candidates during the shard level reduce so the doc count error needs to be calculated accordingly in those cases.

As an example, take this agg from the noaa OSB workload:

{
  "size": 0,
  "aggs": {
    "station": {
      "terms": {
        "field": "station.elevation",
        "size": 50
      },
      "aggs": {
        "date": {
          "terms": {
            "field": "date",
            "size": 1
          },
          "aggs": {
            "max": {
              "max": {
                "field": "TMAX"
              }
            }
          }
        }
      }
    }
  }
}

The "date" aggregation uses size = 1, so the computed slice_size heuristic will be 26 which is fairly small compared to the cardinality of the "date" field.

Attaching the aggregation outputs with concurrent search enabled/disabled:
cs-disabled.txt
cs-enabled.txt

Describe the solution you'd like

Doc count error needs to be calculated in a way that includes the buckets eliminated at the slice level reduce.

Related component

Search:Performance

Describe alternatives you've considered

No response

Additional context

No response

[jed326]

Copying over the explanation of how the slice_size heuristic can affect doc_count and doc_count_error from #11732 (comment)

With the changes in this PR, there are basically 3 different cases when comparing the results from non-concurrent search vs. concurrent search. I'll illustrate this with the ShardSizeTermsIT::testShardSizeEqualsSizeString test. First, the baseline is non-concurrent search, where the response will always look like so:

{
    "keys": {
        "doc_count_error_upper_bound": 5,
        "sum_other_doc_count": 7,
        "buckets": [{
            "key": "1",
            "doc_count": 8,
            "doc_count_error_upper_bound": 0
        }, {
            "key": "3",
            "doc_count": 8,
            "doc_count_error_upper_bound": 0
        }, {
            "key": "2",
            "doc_count": 4,
            "doc_count_error_upper_bound": 2
        }]
    }
}

Now, there are basically 3 ways that using concurrent search with the slice_size heuristic can change the results.

1. Some buckets may have a lower doc_count with higher doc_count_error.

This is the most intuitive outcome -- because we apply a slice_size we eliminate more buckets at the slice level which will contribute to a greater doc count error.

{
    "keys": {
        "doc_count_error_upper_bound": 5,
        "sum_other_doc_count": 8,
        "buckets": [{
            "key": "1",
            "doc_count": 8,
            "doc_count_error_upper_bound": 4
        }, {
            "key": "3",
            "doc_count": 8,
            "doc_count_error_upper_bound": 1
        }, {
            "key": "2",
            "doc_count": 3,
            "doc_count_error_upper_bound": 4
        }]
    }
}

2. Buckets may have the same doc_count but with higher doc_count_error.

This case is confusing at first because if we collect all of the existing documents for a bucket key, we would initially expect the doc_count_error to be 0. However, that's not the case -- take this example:

{
    "keys": {
        "doc_count_error_upper_bound": 6,
        "sum_other_doc_count": 7,
        "buckets": [{
            "key": "1",
            "doc_count": 8,
            "doc_count_error_upper_bound": 5
        }, {
            "key": "3",
            "doc_count": 8,
            "doc_count_error_upper_bound": 5
        }, {
            "key": "2",
            "doc_count": 4,
            "doc_count_error_upper_bound": 3
        }]
    }
}

If bucket key "1" has 8 docs across 2 shards, and but in each shard all of the key "1" docs are in the same slice, then we will need to calculate the doc_count_error from the slices where bucket key "1" is not in the top slice_size buckets.

3. Buckets may have a greater doc_count

For example:

{
    "keys": {
        "doc_count_error_upper_bound": 4,
        "sum_other_doc_count": 6,
        "buckets": [{
            "key": "1",
            "doc_count": 8,
            "doc_count_error_upper_bound": 1
        }, {
            "key": "3",
            "doc_count": 8,
            "doc_count_error_upper_bound": 3
        }, {
            "key": "2",
            "doc_count": 5,
            "doc_count_error_upper_bound": 3
        }]
    }
}

This case fundamentally has to do with how "ties" are broken when terms have the same doc count. The tie breaker (in cases where doc count error is relevant), is the bucket key itself. So take a case where bucket key "2" has 1 document on a shard, and bucket key "4" has 2 documents on a shard.

shard [4, 4, 2]
slice_1 [4, 2]
slice_2 [4]

Now, let's imagine a term agg with size == shard_size == 1. In non-concurrent search bucket "4" will always be returned ahead of bucket "2", however in concurrent search if the 2 documents for bucket "4" are split across different slices, then bucket "2" will actually be return ahead of bucket "4"!

[sohami]

I think its worth mentioning that same behavior exists with non-concurrent path at shard level, depending on how data is distributed across the shards. Probably in most of the cases, if a shard has x,y,z as top terms, then each segment may also have (not guaranteed) those as its top terms and should get reflected in the response.