[Tiered Caching] [Bug Fix] Use concurrentMap instead of HashMap to fix Concurrent Modification Exception

[kiranprakash154]

Description

#14032 reported msearches failing with a Concurrent Modification Exception, though they were not able to repro it consistently they were able to attach a debugger and find out the line throwing the exception, which was

OpenSearch/server/src/main/java/org/opensearch/indices/IndicesRequestCache.java

Line 571 in 5b93f2e

cleanupKeyToCountMap.computeIfAbsent(shardId, k -> new HashMap<>()).merge(cleanupKey.readerCacheKeyId, 1, Integer::sum);

We are using a HashMap (not thread safe) for the inner map of the cleanupKeyToCountMap and hence it throws a Concurrent Modification Exception when the map is getting updated by multiple threads concurrently.

The fix is to use a thread safe Concurrent Map instead.

Related Issues

Resolves #14032

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 58aa93a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for aabf139: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 39642ab: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for cc8b6b7: SUCCESS

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.70%. Comparing base (b15cb0c) to head (cc8b6b7).
Report is 413 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14134      +/-   ##
============================================
+ Coverage     71.42%   71.70%   +0.28%     
- Complexity    59978    62051    +2073     
============================================
  Files          4985     5117     +132     
  Lines        282275   291687    +9412     
  Branches      40946    42166    +1220     
============================================
+ Hits         201603   209156    +7553     
- Misses        63999    65278    +1279     
- Partials      16673    17253     +580     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dblock]

Looks good, or at least harmless, ready to merge?

[kiranprakash154]

Looks good, or at least harmless, ready to merge?

Sorry @dblock , closed this since i have a duplicate PR open - #14221