[Backport 2.x] [Tiered caching] Introducing cache plugins and exposing Ehcache as one of the pluggable disk cache option

[sgup432]

Backport 57f6f46 from #11874.

[sgup432]

@msfroh @peternied Need a look from your side.

[github-actions]

Compatibility status:

Checks if related components are compatible with change ebdac1e

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git]

[github-actions]

❕ Gradle check result for ebdac1e: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled
      1 org.opensearch.index.shard.RemoteIndexShardTests.testNoFailuresOnFileReads
      1 org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT.testConcurrentDecommissionAction

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 82.41525% with 83 lines in your changes are missing coverage. Please review.

Project coverage is 71.20%. Comparing base (e252875) to head (ebdac1e).
Report is 1 commits behind head on 2.x.

Files	Patch %	Lines
.../opensearch/cache/store/disk/EhcacheDiskCache.java	75.53%	36 Missing and 10 partials ⚠️
...search/cache/common/tier/TieredSpilloverCache.java	92.23%	7 Missing and 1 partial ⚠️
.../java/org/opensearch/cache/EhcacheCachePlugin.java	20.00%	8 Missing ⚠️
.../cache/common/tier/TieredSpilloverCachePlugin.java	30.00%	7 Missing ⚠️
...org/opensearch/cache/EhcacheDiskCacheSettings.java	90.90%	2 Missing and 1 partial ⚠️
.../store/settings/OpenSearchOnHeapCacheSettings.java	82.35%	2 Missing and 1 partial ⚠️
.../opensearch/common/cache/service/CacheService.java	88.23%	1 Missing and 1 partial ⚠️
...arch/common/cache/store/OpenSearchOnHeapCache.java	83.33%	2 Missing ⚠️
...n/java/org/opensearch/common/settings/Setting.java	75.00%	1 Missing and 1 partial ⚠️
...ache/common/tier/TieredSpilloverCacheSettings.java	80.00%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x   #12514      +/-   ##
============================================
- Coverage     71.29%   71.20%   -0.09%     
- Complexity    60070    60074       +4     
============================================
  Files          4945     4953       +8     
  Lines        282349   282683     +334     
  Branches      41369    41393      +24     
============================================
- Hits         201298   201297       -1     
- Misses        64035    64364     +329     
- Partials      17016    17022       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.