Fix DerivedFieldQuery to support concurrent search.

[mch2]

Description

This change updates DerivedFieldQuery to create a separate ValueFetcher instance per thread. The DerivedFieldValueFetcher is not thread safe in that it holds a ref to a compiled DerivedFieldScript that is created per thread. Each script also holds a SourceLookup object that is not thread safe.

Note there are no changes here specific to aggs with derived fields, that test already passes because we create a new ValueFetcher per leaf here.

Without this change these ITs fail with:

WARNING: Uncaught exception in thread: Thread[#170,opensearch[node_s1][search][T#1],5,TGRP-SimplePainlessIT]
java.lang.AssertionError: StoredFieldsReader are only supposed to be consumed in the thread in which they have been acquired. But was acquired in Thread[#171,opensearch[node_s1][index_searcher][T#1],5,TGRP-SimplePainlessIT] and consumed in Thread[#172,opensearch[node_s1][index_searcher][T#2],5,TGRP-SimplePainlessIT].
	at __randomizedtesting.SeedInfo.seed([77FDD1CE9C5EAC30]:0)
	at org.apache.lucene.tests.codecs.asserting.AssertingCodec.assertThread(AssertingCodec.java:44)
	at org.apache.lucene.tests.codecs.asserting.AssertingStoredFieldsFormat$AssertingStoredFieldsReader.document(AssertingStoredFieldsFormat.java:75)
	at org.opensearch.search.lookup.SourceLookup.loadSourceIfNeeded(SourceLookup.java:104)
	at org.opensearch.script.DerivedFieldScript.lambda$static$1(DerivedFieldScript.java:42)
	at org.opensearch.script.DynamicMap.get(DynamicMap.java:84)
	at org.opensearch.painless.PainlessScript$Script.execute(emit(params._source["field"]):12)
	at org.opensearch.index.mapper.DerivedFieldValueFetcher.fetchValuesInternal(DerivedFieldValueFetcher.java:54)
	at org.opensearch.index.mapper.DerivedFieldValueFetcher.getIndexableField(DerivedFieldValueFetcher.java:59)
	at org.opensearch.index.query.DerivedFieldQuery$1$1.matches(DerivedFieldQuery.java:99)

Related Issues

Resolves #15007

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for e0aa323: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for e0aa323: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 2e0d474: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 7cc1a6b: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

❌ Gradle check result for 917ebd5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 1520b44: SUCCESS

[github-actions]

❕ Gradle check result for 2373d0f: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.