[BUG] BWC failures for aggregations in MixedClusterClientYamlTestSuiteIT

[mattweber]

Lots of BWC failures for aggregations in MixedClusterClientYamlTestSuiteIT found in #2103. Here are just a few:

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v1.3.0#mixedClusterTest' --tests "org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=search.aggregation/230_composite/Composite aggregation with unmapped field}" -Dtests.seed=D4386AC6DA14EDFC -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sv-SE -Dtests.timezone=America/Paramaribo -Druntime.java=17

org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=search.aggregation/230_composite/Composite aggregation with unmapped field} FAILED
    java.lang.AssertionError: Failure at [search.aggregation/230_composite:578]: hits.total didn't match expected value:
                        hits.total: expected Integer [7] but was Integer [6]
  
REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v1.3.0#mixedClusterTest' --tests "org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=search.aggregation/230_composite/Terms source from part of sorted}" -Dtests.seed=D4386AC6DA14EDFC -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sv-SE -Dtests.timezone=America/Paramaribo -Druntime.java=17

org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=search.aggregation/230_composite/Terms source from part of sorted} FAILED
    java.lang.AssertionError: Failure at [search.aggregation/230_composite:1017]: expected [2xx] status code but api [search] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"i_o_exception","reason":"Unknown SortOrder ordinal [2]","stack_trace":"[Unknown SortOrder ordinal [2]]; nested: IOException[Unknown SortOrder ordinal [2]];\n\tat 

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v1.3.0#mixedClusterTest' --tests "org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=search.aggregation/230_composite/Composite aggregation with format and calendar_interval}" -Dtests.seed=D4386AC6DA14EDFC -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sv-SE -Dtests.timezone=America/Paramaribo -Druntime.java=17

org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=search.aggregation/230_composite/Composite aggregation with format and calendar_interval} FAILED
    java.lang.AssertionError: Failure at [search.aggregation/230_composite:365]: expected [2xx] status code but api [search] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"i_o_exception","reason":"Unknown SortOrder ordinal [2]","stack_trace":"[Unknown SortOrder ordinal [2]]; nested: IOException[Unknown SortOrder ordinal [2]];\n\tat 

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v1.3.0#mixedClusterTest' --tests "org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=search.aggregation/230_composite/Simple Composite aggregation with GeoTile grid}" -Dtests.seed=D4386AC6DA14EDFC -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sv-SE -Dtests.timezone=America/Paramaribo -Druntime.java=17

org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=search.aggregation/230_composite/Simple Composite aggregation with GeoTile grid} FAILED
    java.lang.AssertionError: Failure at [search.aggregation/230_composite:694]: expected [2xx] status code but api [search] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"i_o_exception","reason":"Unknown SortOrder ordinal [2]","stack_trace":"[Unknown SortOrder ordinal [2]]; nested: IOException[Unknown SortOrder ordinal [2]];\n\tat 

[dblock]

org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=search.aggregation/230_composite/Mixed ip and unmapped fields} FAILED
    java.lang.AssertionError: Failure at [search.aggregation/230_composite:815]: hits.total.value didn't match expected value:
                  hits.total.value: expected Integer [1] but was Integer [0]
        at __randomizedtesting.SeedInfo.seed([5880D97EA4BDAA3E:D0D4E6A40A41C7C6]:0)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:442)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.test(OpenSearchClientYamlSuiteTestCase.java:415)
        at jdk.internal.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:833)

        Caused by:
        java.lang.AssertionError: hits.total.value didn't match expected value:
                      hits.total.value: expected Integer [1] but was Integer [0]
            at org.opensearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:115)
            at org.opensearch.test.rest.yaml.section.Assertion.execute(Assertion.java:89)
            at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:431)
            ... 39 more

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v1.3.0#mixedClusterTest' --tests "org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=search.aggregation/230_composite/date_histogram on date_nanos}" -Dtests.seed=5880D97EA4BDAA3E -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=zh-TW -Dtests.timezone=Europe/Helsinki -Druntime.java=17

From #2103 (comment)

[dblock]

#2096 (comment)

[andrross]

I can replicate the failures going back to a commit over a week old on main, so that suggests to me the problem is a commit to the 1.x branch, though I don't know how to make my local build run the bwc tests against anything other than the latest commit on that branch to test my theory...

[andrross]

On my dev machine, this command succeeds:

./gradlew ':qa:mixed-cluster:v1.3.0#mixedClusterTest' --tests "org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=search.aggregation/230_composite/Composite aggregation with unmapped field}" -Dtests.seed=D4386AC6DA14EDFC -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sv-SE -Dtests.timezone=America/Paramaribo -Druntime.java=17 -Dbwc.refspec.1.x=b0d94cba13fd9d7b2a37aa0610ef9215ee054374

while this command fails:

./gradlew ':qa:mixed-cluster:v1.3.0#mixedClusterTest' --tests "org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=search.aggregation/230_composite/Composite aggregation with unmapped field}" -Dtests.seed=D4386AC6DA14EDFC -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sv-SE -Dtests.timezone=America/Paramaribo -Druntime.java=17 -Dbwc.refspec.1.x=5b2713661ea14323754f4083de23888797ab0882

I believe this is the problematic commit on the 1.x branch

[andrross]

@penghuo has potentially identified the fix in main, which is to change the compatibility check here now that the feature has been backported to 1.3:

OpenSearch/server/src/main/java/org/opensearch/search/aggregations/bucket/composite/CompositeValuesSourceBuilder.java

Line 84 in d47725d

if (in.getVersion().onOrAfter(Version.V_2_0_0)) {

He's working on a PR. That is the better fix if it works out, otherwise the revert to the 1.x PR can be merged.

[mattweber]

I think this can be closed now that #2147 is merged. @andrross ?

[dblock]

Closing