[Concurrent Segment Search][Meta] GA readiness items

[sohami]

This is a place holder issue for tracking all the GA readiness work items needed for concurrent search:

1. Remove the experimental feature flag opensearch.experimental.feature.concurrent_segment_search.enabled
2. Update the default value of cluster setting search.concurrent_segment_search.enabled to false after above is done
3. Update tests using feature flag to use cluster setting instead. This will get addressed to large extent by [Concurrent Segment Search] Enable Search ITs with Concurrent Segment Search #7440
4. Doc updates: a) Profile APIs, b) Cluster/Index Settings, c) New stats that will be added
   [DOC] Concurrent Segment Search support in OpenSearch documentation-website#3662
5. Remove the log in QueryPhaseSearcherWrapper
6. Nightly benchmark setup

[sohami]

@Pallavi-AWS Fyi

[sandervandegeijn]

Maybe a dumb question, why not enable it by default? We have enabled the feature and never saw any disadvantages, even in the experimental stage. Just a decent performance improvement. 👍

[jed326]

@sandervandegeijn We definitely want to explore making concurrent segment search the default case but in the first release we want to minimize any potential disruptions to users. We already know that in cases of very high resource usage concurrent search may see some performance regressions. I'm thinking after the GA release of concurrent search we can start a more detailed discussion on the pros/cons of making concurrent search default.

[sandervandegeijn]

Okay cool :) Caution is almost always a good thing!

[reta]

@sohami @jed326 this is targeting at least 2.13.0 right? (I believe it is super risky for 2.12.0)