[BUG] Missing privileges in FileCacheFacroty and TransferManager on search that includes searchable snapshots

[asor-gl]

Describe the bug

When a search is preformed that includes searchable snapshots it fails with runtime error. Added trace can be examined to see more detail about missing permissions to perform file interaction.

Related component

Search:Searchable Snapshots

To Reproduce

1. Perform a search on indices that include searchable snapshots.
2. Search results in error returning runtime error.

Expected behavior

Search is performed successfully.

Additional Details

Plugins

Screenshots
Error traces where missing permissions are found:

opensearch-master-data-1 opensearch [2024-10-26T22:23:45,806][TRACE][o.o.s.SearchService      ] [opensearch-master-data-1] Query phase failed
opensearch-master-data-1 opensearch org.opensearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase.executeInternal(QueryPhase.java:307) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase.execute(QueryPhase.java:155) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1850) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1911) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:345) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:328) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.cache.store.OpenSearchOnHeapCache.lambda$computeIfAbsent$0(OpenSearchOnHeapCache.java:98) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.cache.Cache.compute(Cache.java:483) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.cache.Cache.computeIfAbsent(Cache.java:434) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.cache.store.OpenSearchOnHeapCache.computeIfAbsent(OpenSearchOnHeapCache.java:98) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:288) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1917) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.indices.IndicesService.loadIntoContext(IndicesService.java:1848) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:641) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:707) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:676) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1005) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
opensearch-master-data-1 opensearch 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
opensearch-master-data-1 opensearch 	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
opensearch-master-data-1 opensearch Caused by: org.opensearch.script.ScriptException: runtime error
opensearch-master-data-1 opensearch 	at org.opensearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:109) ~[?:?]
opensearch-master-data-1 opensearch 	at org.opensearch.painless.PainlessScript$Script.execute((doc.containsKey('streams') && doc['streams'].size() > 0 ...:1) ~[?:?]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.support.values.ScriptBytesValues.advanceExact(ScriptBytesValues.java:76) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ValuesSourceCollectorSource$1.collect(MapStringTermsAggregator.java:213) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:129) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectBucket(BucketsAggregator.java:115) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.filter.FiltersAggregator$1.collect(FiltersAggregator.java:186) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BestBucketsDeferringCollector.prepareSelectedBuckets(BestBucketsDeferringCollector.java:221) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.DeferableBucketAggregator.beforeBuildingBuckets(DeferableBucketAggregator.java:119) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:217) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForVariableBuckets(BucketsAggregator.java:406) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator.buildAggregations(AutoDateHistogramAggregator.java:257) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator$FromSingle.buildAggregations(AutoDateHistogramAggregator.java:439) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:205) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.BucketCollectorProcessor.processPostCollection(BucketCollectorProcessor.java:78) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:292) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:560) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:355) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:462) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:450) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWith(QueryPhase.java:432) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhaseSearcherWrapper.searchWith(QueryPhaseSearcherWrapper.java:60) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.neuralsearch.search.query.HybridQueryPhaseSearcher.searchWith(HybridQueryPhaseSearcher.java:61) ~[?:?]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase.executeInternal(QueryPhase.java:282) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	... 26 more
opensearch-master-data-1 opensearch Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/share/opensearch/data/nodes/0/cache/otDA0XeQT4uO9HBczv6Vpw/0/RemoteLocalStore/_4p_Lucene90_0.dvd_block_1" "delete")
opensearch-master-data-1 opensearch 	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
opensearch-master-data-1 opensearch 	at java.base/java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
opensearch-master-data-1 opensearch 	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
opensearch-master-data-1 opensearch 	at java.base/java.lang.SecurityManager.checkDelete(SecurityManager.java:861) ~[?:?]
opensearch-master-data-1 opensearch 	at java.base/sun.nio.fs.UnixPath.checkDelete(UnixPath.java:803) ~[?:?]
opensearch-master-data-1 opensearch 	at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:269) ~[?:?]
opensearch-master-data-1 opensearch 	at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:109) ~[?:?]
opensearch-master-data-1 opensearch 	at java.base/java.nio.file.Files.deleteIfExists(Files.java:1191) ~[?:?]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.filecache.FileCacheFactory.lambda$createDefaultBuilder$0(FileCacheFactory.java:58) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.ExceptionsHelper.catchAsRuntimeException(ExceptionsHelper.java:352) ~[opensearch-core-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.filecache.FileCacheFactory.lambda$createDefaultBuilder$1(FileCacheFactory.java:58) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.utils.cache.LRUCache.removeNode(LRUCache.java:366) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.utils.cache.LRUCache.remove(LRUCache.java:186) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.utils.cache.SegmentedCache.remove(SegmentedCache.java:105) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.filecache.FileCache.remove(FileCache.java:105) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.utils.TransferManager$DelayedCreationCachedIndexInput.getIndexInput(TransferManager.java:169) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.utils.TransferManager.fetchBlob(TransferManager.java:89) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput.fetchBlock(OnDemandBlockSnapshotIndexInput.java:160) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.demandBlock(OnDemandBlockIndexInput.java:340) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.seekInternal(OnDemandBlockIndexInput.java:311) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.seek(OnDemandBlockIndexInput.java:209) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput.seek(OnDemandBlockSnapshotIndexInput.java:30) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1122) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-1 opensearch 	at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-1 opensearch 	at org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-1 opensearch 	at org.opensearch.index.fielddata.FieldData$12.nextValue(FieldData.java:456) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.fielddata.ScriptDocValues$BinaryScriptDocValues.setNextDocId(ScriptDocValues.java:514) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.fielddata.ScriptDocValues$Strings.setNextDocId(ScriptDocValues.java:547) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:103) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:56) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.painless.PainlessScript$Script.execute((doc.containsKey('streams') && doc['streams'].size() > 0 ...:36) ~[?:?]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.support.values.ScriptBytesValues.advanceExact(ScriptBytesValues.java:76) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ValuesSourceCollectorSource$1.collect(MapStringTermsAggregator.java:213) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:129) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectBucket(BucketsAggregator.java:115) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.filter.FiltersAggregator$1.collect(FiltersAggregator.java:186) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BestBucketsDeferringCollector.prepareSelectedBuckets(BestBucketsDeferringCollector.java:221) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.DeferableBucketAggregator.beforeBuildingBuckets(DeferableBucketAggregator.java:119) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:217) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForVariableBuckets(BucketsAggregator.java:406) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator.buildAggregations(AutoDateHistogramAggregator.java:257) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator$FromSingle.buildAggregations(AutoDateHistogramAggregator.java:439) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:205) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.BucketCollectorProcessor.processPostCollection(BucketCollectorProcessor.java:78) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:292) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:560) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:355) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:462) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:450) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWith(QueryPhase.java:432) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhaseSearcherWrapper.searchWith(QueryPhaseSearcherWrapper.java:60) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.neuralsearch.search.query.HybridQueryPhaseSearcher.searchWith(HybridQueryPhaseSearcher.java:61) ~[?:?]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase.executeInternal(QueryPhase.java:282) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	... 26 more

After following exception was fixed another one was present:

[2024-10-28T16:36:55,861][TRACE][o.o.s.SearchService      ] [opensearch-master-data-0] Query phase failed
opensearch-master-data-0 opensearch org.opensearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase.executeInternal(QueryPhase.java:307) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase.execute(QueryPhase.java:155) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1850) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1911) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:345) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:328) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.cache.store.OpenSearchOnHeapCache.lambda$computeIfAbsent$0(OpenSearchOnHeapCache.java:98) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.cache.Cache.compute(Cache.java:483) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.cache.Cache.computeIfAbsent(Cache.java:434) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.cache.store.OpenSearchOnHeapCache.computeIfAbsent(OpenSearchOnHeapCache.java:98) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:288) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1917) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.indices.IndicesService.loadIntoContext(IndicesService.java:1848) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:641) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:707) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:676) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1005) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
opensearch-master-data-0 opensearch 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
opensearch-master-data-0 opensearch 	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
opensearch-master-data-0 opensearch Caused by: org.opensearch.script.ScriptException: runtime error
opensearch-master-data-0 opensearch 	at org.opensearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:109) ~[?:?]
opensearch-master-data-0 opensearch 	at org.opensearch.painless.PainlessScript$Script.execute((doc.containsKey('gl2_source_node') && doc['gl2_source_node'].size() > 0 ...:1) ~[?:?]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.support.values.ScriptBytesValues.advanceExact(ScriptBytesValues.java:76) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ValuesSourceCollectorSource$1.collect(MapStringTermsAggregator.java:213) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:129) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectBucket(BucketsAggregator.java:115) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.filter.FiltersAggregator$1.collect(FiltersAggregator.java:186) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BestBucketsDeferringCollector.prepareSelectedBuckets(BestBucketsDeferringCollector.java:221) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.DeferableBucketAggregator.beforeBuildingBuckets(DeferableBucketAggregator.java:119) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:217) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForVariableBuckets(BucketsAggregator.java:406) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator.buildAggregations(AutoDateHistogramAggregator.java:257) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator$FromSingle.buildAggregations(AutoDateHistogramAggregator.java:439) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:205) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.BucketCollectorProcessor.processPostCollection(BucketCollectorProcessor.java:78) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:292) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:560) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:355) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:462) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:450) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWith(QueryPhase.java:432) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhaseSearcherWrapper.searchWith(QueryPhaseSearcherWrapper.java:60) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.neuralsearch.search.query.HybridQueryPhaseSearcher.searchWith(HybridQueryPhaseSearcher.java:61) ~[?:?]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase.executeInternal(QueryPhase.java:282) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	... 26 more
opensearch-master-data-0 opensearch Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/share/opensearch/data/nodes/0/cache/GGfzun4mSpSiqhVfEnhizA/0/RemoteLocalStore/_qs_Lucene90_0.dvd_block_5" "read")
opensearch-master-data-0 opensearch 	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
opensearch-master-data-0 opensearch 	at java.base/java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
opensearch-master-data-0 opensearch 	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
opensearch-master-data-0 opensearch 	at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:742) ~[?:?]
opensearch-master-data-0 opensearch 	at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:789) ~[?:?]
opensearch-master-data-0 opensearch 	at java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:563) ~[?:?]
opensearch-master-data-0 opensearch 	at java.base/java.nio.file.Files.exists(Files.java:2514) ~[?:?]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.utils.TransferManager.createIndexInput(TransferManager.java:111) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.utils.TransferManager$DelayedCreationCachedIndexInput.getIndexInput(TransferManager.java:166) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.utils.TransferManager.fetchBlob(TransferManager.java:89) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput.fetchBlock(OnDemandBlockSnapshotIndexInput.java:160) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.demandBlock(OnDemandBlockIndexInput.java:340) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.seekInternal(OnDemandBlockIndexInput.java:311) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.seek(OnDemandBlockIndexInput.java:209) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput.seek(OnDemandBlockSnapshotIndexInput.java:30) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1122) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-0 opensearch 	at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-0 opensearch 	at org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-0 opensearch 	at org.opensearch.index.fielddata.FieldData$12.nextValue(FieldData.java:456) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.fielddata.ScriptDocValues$BinaryScriptDocValues.setNextDocId(ScriptDocValues.java:514) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.fielddata.ScriptDocValues$Strings.setNextDocId(ScriptDocValues.java:547) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:103) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:56) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.painless.PainlessScript$Script.execute((doc.containsKey('gl2_source_node') && doc['gl2_source_node'].size() > 0 ...:44) ~[?:?]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.support.values.ScriptBytesValues.advanceExact(ScriptBytesValues.java:76) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ValuesSourceCollectorSource$1.collect(MapStringTermsAggregator.java:213) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:129) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectBucket(BucketsAggregator.java:115) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.filter.FiltersAggregator$1.collect(FiltersAggregator.java:186) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BestBucketsDeferringCollector.prepareSelectedBuckets(BestBucketsDeferringCollector.java:221) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.DeferableBucketAggregator.beforeBuildingBuckets(DeferableBucketAggregator.java:119) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:217) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForVariableBuckets(BucketsAggregator.java:406) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator.buildAggregations(AutoDateHistogramAggregator.java:257) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator$FromSingle.buildAggregations(AutoDateHistogramAggregator.java:439) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:205) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.BucketCollectorProcessor.processPostCollection(BucketCollectorProcessor.java:78) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:292) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:560) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:355) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:462) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:450) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWith(QueryPhase.java:432) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhaseSearcherWrapper.searchWith(QueryPhaseSearcherWrapper.java:60) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.neuralsearch.search.query.HybridQueryPhaseSearcher.searchWith(HybridQueryPhaseSearcher.java:61) ~[?:?]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase.executeInternal(QueryPhase.java:282) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	... 26 more

Host/Environment (please complete the following information):

• OS: docker image: opensearchproject/opensearch:2.17.1
• Version 2.17.1

Additional context
We tested the build with internal patch adding needed permissions and it fixes the issue, please let me know if i should create a PR for it.

[finnegancarroll]

Hi @asor-gl, thanks for opening this issue.
This looks a lot like #14268 and i'm thinking the fix for that did not go far enough. The previous issue elevated permissions for fileCache.compute since inserting into the cache can trigger evictions.

In this case it seems script execution is trying to access the underlying file associated with a cacheEntry and failing since cacheEntry.getIndexInput() is missing these elevated permissions.

[asor-gl]

@finnegancarroll Thanks for the tip. I'll test if adding elevated permissions higher up will work as well, as in the other issue.

Update: can confirm that #16544 fixes the issue with permissions.

[sandeshkr419]

Closing as #16544 fixes up this issue.