Show only intersecting buckets to the Adjacency matrix aggregation

[brusic]

Re-release of a previous PR that did not correctly sign the commit

Description

Show only intersecting buckets to the Adjacency matrix aggregation

The Adjacency matrix aggregation will compute and return all buckets related to any of the combinations of the filters specified, including hits on a single filter. However, sometimes a user would want to return only the buckets related to an intersection and not for a single filter. An optional show_only_intersecting parameter would stop buckets resulting from a single filter being hit to be created.

Related Issues

#8832

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change dc5897b

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

[github-actions]

❌ Gradle check result for 68d1746: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[brusic]

Apparently failed with > Task :qa:wildfly:spotlessMisc FAILED, but it is working locally using jdk11. No issues mentioning a flaky test.

Jenkins error

* What went wrong:
A problem was found with the configuration of task ':qa:wildfly:spotlessMisc' (type 'SpotlessTaskImpl').
  - Gradle detected a problem with the following location: '/var/jenkins/workspace/gradle-check/search/qa/wildfly'.
    
    Reason: Task ':qa:wildfly:spotlessMisc' uses this output of task ':qa:wildfly:preProcessFixture' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.
    
    Possible solutions:
      1. Declare task ':qa:wildfly:preProcessFixture' as an input of ':qa:wildfly:spotlessMisc'.
      2. Declare an explicit dependency on ':qa:wildfly:preProcessFixture' from ':qa:wildfly:spotlessMisc' using Task#dependsOn.
      3. Declare an explicit dependency on ':qa:wildfly:preProcessFixture' from ':qa:wildfly:spotlessMisc' using Task#mustRunAfter.

Local execution

% ./gradlew :qa:wildfly:spotlessMisc      

> Configure project :
========================= WARNING =========================
         Backwards compatibility tests are disabled!
See https://github.com/opensearch-project/OpenSearch/issues/4173
===========================================================
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.5
  OS Info               : Mac OS X 13.4.1 (x86_64)
  JDK Version           : 11 (Homebrew JDK)
  JAVA_HOME             : /usr/local/Cellar/openjdk@11/11.0.21/libexec/openjdk.jdk/Contents/Home
  Random Testing Seed   : 4202A040C72920CD
  In FIPS 140 mode      : false
=======================================

BUILD SUCCESSFUL in 7s
10 actionable tasks: 1 executed, 9 up-to-date

% ./gradlew :qa:wildfly:preProcessFixture

> Configure project :
========================= WARNING =========================
         Backwards compatibility tests are disabled!
See https://github.com/opensearch-project/OpenSearch/issues/4173
===========================================================
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.5
  OS Info               : Mac OS X 13.4.1 (x86_64)
  JDK Version           : 11 (Homebrew JDK)
  JAVA_HOME             : /usr/local/Cellar/openjdk@11/11.0.21/libexec/openjdk.jdk/Contents/Home
  Random Testing Seed   : E8272F8D6651F238
  In FIPS 140 mode      : false
=======================================

BUILD SUCCESSFUL in 6s
189 actionable tasks: 3 executed, 186 up-to-date

[github-actions]

✅ Gradle check result for 0d7fd10: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 57.57576% with 14 lines in your changes missing coverage. Please review.

Project coverage is 72.19%. Comparing base (e7e19f7) to head (4eb5173).
Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
...t/adjacency/AdjacencyMatrixAggregationBuilder.java	69.23%	3 Missing and 5 partials ⚠️
...ns/bucket/adjacency/AdjacencyMatrixAggregator.java	0.00%	5 Missing ⚠️
...et/adjacency/AdjacencyMatrixAggregatorFactory.java	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11733      +/-   ##
============================================
- Coverage     72.20%   72.19%   -0.01%     
+ Complexity    65289    65254      -35     
============================================
  Files          5299     5299              
  Lines        303536   303560      +24     
  Branches      43941    43946       +5     
============================================
- Hits         219180   219167      -13     
- Misses        66441    66452      +11     
- Partials      17915    17941      +26     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[msfroh]

Thanks @brusic! There are a couple of commits that are not signed off (with git commit -s). Your easiest option is probably to do a git rebase -i, squash all your commits down into a single commit, then you can hit that with git commit --amend -s.

Also, could you please add a changelog entry to https://github.com/opensearch-project/OpenSearch/blob/main/CHANGELOG.md?

[github-actions]

❌ Gradle check result for cb96ab7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 5610ad1: SUCCESS

[github-actions]

✅ Gradle check result for 02f135f: SUCCESS

[github-actions]

✅ Gradle check result for 4eb5173: SUCCESS

[reta]

@brusic FYI, there are flaky tests very luckily caused by this change:

https://build.ci.opensearch.org/job/gradle-check/52247/testReport/junit/org.opensearch.backwards/MixedClusterClientYamlTestSuiteIT/test__p0_search_aggregation_70_adjacency_matrix_Terms_lookup_/

java.lang.AssertionError: Failure at [search.aggregation/70_adjacency_matrix:85]: expected [2xx] status code but api [search] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Unknown NamedWriteable [org.opensearch.index.query.QueryBuilder][1]","stack_trace":"OpenSearchException[Unknown NamedWriteable [org.opensearch.index.query.QueryBuilder][1]]; nested: IllegalArgumentException[Unknown NamedWriteable [org.opensearch.index.query.QueryBuilder][1]];\n\tat org.opensearch.OpenSearchException.guessRootCauses(OpenSearchException.java:710)\n\tat org.opensearch.action.search.SearchPhaseExecutionException.guessRootCauses(SearchPhaseExecutionException.java:188)\n\tat org.opensearch.OpenSearchException.guessRootCauses(OpenSearchException.java:692)\n\tat org.opensearch.OpenSearchException.generateFailureXContent(OpenSearchException.java:637)\n\tat org.opensearch.rest.BytesRestResponse.build(BytesRestResponse.java:172)\n\tat org.opensearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:132)\n\tat org.opensearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:109)\n\tat org.opensearch.rest.action.RestActionListener.from(RestActionListener.java:74)\n\tat 

I think it should be fixed by #17030, but would be great to double check. Thank you.

[1] https://build.ci.opensearch.org/job/gradle-check/52247/testReport/junit/org.opensearch.backwards/MixedClusterClientYamlTestSuiteIT/test__p0_search_aggregation_70_adjacency_matrix_Terms_lookup_/

[brusic]

I will check before and after the fix. Is there a branch to check before the backport version check?

[reta]

I will check before and after the fix. Is there a branch to check before the backport version check?

I think we don't need to check before the backport, just main is stable and this tests are not flaky, thank you