Fix KNN build due to Lucene 10.3 upgrade by navneet1v · Pull Request #2878 · opensearch-project/k-NN

navneet1v · 2025-09-17T06:51:11Z

Description

This change allows us to fix the failing builds which was happening to Lucene 10.3 and other changes in core.

Changes done:

Created a new codec KNN1030Codec class for Lucene 10.3 version.
Fixed the interface for search for kNN queries to use AcceptedDocs rather than BitSets
Fix and cleaned up the guava dependencies for grpc.

Known Issue

MacOS CI will fail and it is unrelated to this PR. Will raise another PR for the same. GH issue: #2876

Related Issues

#2888

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

navneet1v · 2025-09-18T22:21:18Z

Need to fix Lucene codec too since OpenSearch core is moved to 10.3 version of Lucene

finnegancarroll · 2025-09-19T00:11:25Z

I see dependency conflict fixed on this branch but ITs still seem to fail to load failure access at runtime when i run locally.

./gradlew ':integTest' --tests 'org.opensearch.knn.recall.RecallTestsIT'
...

=== Standard error of node `node{::integTest-0}` ===
»   ↓ last 40 non error or warning messages from /Users/carrofin/fdev/repos/k-NN/build/testclusters/integTest-0/logs/opensearch.stderr.log ↓
» WARNING: Using incubator modules: jdk.incubator.vector
»  WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
»  fatal error in thread [opensearch[integTest-0][search][T#1]], exiting
»  java.lang.NoClassDefFoundError: com/google/common/util/concurrent/internal/InternalFutureFailureAccess
»       at java.base/java.lang.ClassLoader.defineClass1(Native Method)
»       at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027)
»       at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
»       at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:524)
»       at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:427)
»       at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:421)

It's confusing to me this class is not loaded as I see it in the bundle:
#2885 (comment)

I think @karenyrx has an upstream change to resolve this:
opensearch-project/OpenSearch#19339

My only thought would be it's kind of awkward that grpc transport SPI is supplying failure access when KNN uses it natively for com.google.common.cache.LocalCache in org.opensearch.knn.index.memory.NativeMemoryCacheManager.

navneet1v · 2025-09-20T18:00:26Z

waiting for this PR to be available. opensearch-project/OpenSearch#19339 via nightly snapshots.

Tested locally that with this change build is working.

cd OpenSearch
bash ./scripts/build.sh -v 3.3.0 -s true
cd ../k-NN
./gradlew integTest -PcustomDistributionUrl="/Users/navneev/workplace/OpenSearch/artifacts/dist/opensearch-min-3.3.0-SNAPSHOT-darwin-arm64.tar.gz"

src/main/java/org/opensearch/knn/index/codec/KNN990Codec/NativeEngines990KnnVectorsReader.java

src/main/java/org/opensearch/knn/index/query/TopApproxKnnCollector.java

Vikasht34 · 2025-09-22T02:49:06Z

src/main/java/org/opensearch/knn/index/query/memoryoptsearch/MemoryOptimizedKNNWeight.java

+            final Bits liveDocs = reader.getLiveDocs();
+            acceptDocs = AcceptDocs.fromLiveDocs(liveDocs, reader.maxDoc());
+        } else {
+            acceptDocs = new AcceptDocs() {


This is too verbose to put in else statement, Please find a better way to write this.

Can you please elaborate more your comment. What is the meaning of this is too verbose to put in else comment?

I think we can factor this out as a small function in this class with a small comment. e.g. when cardinality is 0, it's indicating that there's no filtering which is combining user provided filter + liveDocs required..

It was not clear from the comment. I can move this to a private function.

NIT : If cardinality == 0, then can we just pass null?

acceptDocs = AcceptDocs.fromLiveDocs(null, reader.maxDoc());

@navneet1v just want to confirm we can't use fromIteratorSupplier right?

Another thought would be, should we create acceptdocs when we create filterbit set and pass it in?

Synced with Navneet, we can just pass liveDocs. If no filtering + no deleted docs, liveDocs will always be null.e

@navneet1v just want to confirm we can't use fromIteratorSupplier right?

Another thought would be, should we create acceptdocs when we create filterbit set and pass it in?

@shatejas to use the fromIteratorSupplier, we have to pass the iterator right from the base KNNWeight.class . I thought about doing it first and then realized it will break the whole code and will lead to changes across many files, we are already on 25 files in this PR. I think with new interface we should move our code to completely using AcceptedDocs. Will take that change in upcoming PRs.

So, my strategy was to first make this change get the build working, and as a next steps see if we can use fromIteratorSupplier. Based on my deep-dive I can see AcceptedDocs was just a fancy wrapper over bits and DISI. For HNSW the usage of bits remain same, its just when Lucene starts to do exact search it uses iterator() to get the doc Ids. https://github.com/apache/lucene/blob/586dc103361c99fefa2f67946855691a073e805c/lucene/core/src/java/org/apache/lucene/search/AbstractKnnVectorQuery.java#L234

@navneet1v I think I am aligned, as long as we do it as a fast follow up

Sure, I will create a GH issue on this. and take it next.

src/test/java/org/opensearch/knn/memoryoptsearch/AbstractFaissCagraHnswIndexTests.java

0ctopus13prime

LGTM! Added few nit-pickings, thank you for the hard work!

src/test/java/org/opensearch/knn/memoryoptsearch/FaissCagraHnswIndexTests.java

src/test/java/org/opensearch/knn/memoryoptsearch/FaissMemoryOptimizedSearcherTests.java

src/main/java/org/opensearch/knn/index/query/common/QueryUtils.java

shatejas · 2025-09-22T18:56:07Z

src/main/java/org/opensearch/knn/index/query/memoryoptsearch/MemoryOptimizedKNNWeight.java

+            final Bits liveDocs = reader.getLiveDocs();
+            acceptDocs = AcceptDocs.fromLiveDocs(liveDocs, reader.maxDoc());
+        } else {
+            acceptDocs = new AcceptDocs() {


@navneet1v just want to confirm we can't use fromIteratorSupplier right?

Another thought would be, should we create acceptdocs when we create filterbit set and pass it in?

Signed-off-by: Navneet Verma <navneev@amazon.com>

0ctopus13prime

LGTM, thank you

…#2878) Signed-off-by: Navneet Verma <navneev@amazon.com>

navneet1v added the skip-changelog label Sep 17, 2025

navneet1v force-pushed the main branch 2 times, most recently from d0cca7e to f5242bf Compare September 18, 2025 01:09

navneet1v mentioned this pull request Sep 18, 2025

Force core :server error prone annotations version #2885

Closed

5 tasks

navneet1v force-pushed the main branch from f5242bf to 1bd0004 Compare September 18, 2025 22:18

navneet1v changed the title ~~Fixing the macOS CI~~ Fixing knn build due to guava Sep 18, 2025

finnegancarroll mentioned this pull request Sep 19, 2025

Add failureaccess dependency to transport-grpc module opensearch-project/OpenSearch#19339

Merged

3 tasks

navneet1v changed the title ~~Fixing knn build due to guava~~ Fix KNN build due to Lucene 10.3 upgrade Sep 20, 2025

navneet1v force-pushed the main branch 6 times, most recently from 8e55b69 to 7013e9e Compare September 20, 2025 02:03

karenyrx mentioned this pull request Sep 20, 2025

Fix guava and errorprone dependencies #2883

Closed

5 tasks

navneet1v force-pushed the main branch 2 times, most recently from f1982a7 to 3de0da0 Compare September 21, 2025 01:46

navneet1v added this to Vector Search RoadMap Sep 21, 2025

navneet1v moved this to 3.3.0 in Vector Search RoadMap Sep 21, 2025

navneet1v force-pushed the main branch 2 times, most recently from 795cfe2 to a5d591d Compare September 21, 2025 05:21

navneet1v marked this pull request as ready for review September 21, 2025 06:41

navneet1v requested review from VijayanB, heemin32, jmazanec15, naveentatikonda and vamshin as code owners September 21, 2025 06:41

navneet1v requested review from 0ctopus13prime, Vikasht34, junqiu-lei, luyuncheng, martin-gaievski, ryanbogan and shatejas as code owners September 21, 2025 06:41

navneet1v force-pushed the main branch from dd20a9f to 20f21f1 Compare September 21, 2025 07:32

navneet1v added Maintenance Add support for new versions of OpenSearch/Dashboards from upstream and removed skip-changelog labels Sep 21, 2025

Vikasht34 reviewed Sep 22, 2025

View reviewed changes

0ctopus13prime reviewed Sep 22, 2025

View reviewed changes

src/test/java/org/opensearch/knn/memoryoptsearch/AbstractFaissCagraHnswIndexTests.java Show resolved Hide resolved

0ctopus13prime reviewed Sep 22, 2025

View reviewed changes

src/test/java/org/opensearch/knn/memoryoptsearch/AbstractFaissCagraHnswIndexTests.java Outdated Show resolved Hide resolved

0ctopus13prime reviewed Sep 22, 2025

View reviewed changes

shatejas reviewed Sep 22, 2025

View reviewed changes

Fixing the kNN build after lucene upgrade to 10.3

a67fb74

Signed-off-by: Navneet Verma <navneev@amazon.com>

navneet1v force-pushed the main branch from 20f21f1 to a67fb74 Compare September 22, 2025 19:38

0ctopus13prime approved these changes Sep 22, 2025

View reviewed changes

0ctopus13prime mentioned this pull request Sep 22, 2025

[BUG] Cosine similarity score range issue when memory optimized search is enabled. #2887

Closed

0ctopus13prime requested a review from Vikasht34 September 22, 2025 20:51

shatejas approved these changes Sep 22, 2025

View reviewed changes

navneet1v merged commit b285075 into opensearch-project:main Sep 22, 2025
36 of 38 checks passed

github-project-automation bot moved this from 3.3.0 to ✅ Done in Vector Search RoadMap Sep 22, 2025

tkykenmt mentioned this pull request Jan 11, 2026

[bugfix] Lucene 10.3 Update tkykenmt/opensearch-feature-explorer#1370

Closed

3 tasks

jingqimao77-spec pushed a commit to jingqimao77-spec/k-NN that referenced this pull request Mar 15, 2026

Fixing the kNN build after lucene upgrade to 10.3 (opensearch-project…

a656d63

…#2878) Signed-off-by: Navneet Verma <navneev@amazon.com>

Conversation

navneet1v commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes done:

Known Issue

Related Issues

Check List

Uh oh!

navneet1v commented Sep 18, 2025

Uh oh!

finnegancarroll commented Sep 19, 2025

Uh oh!

navneet1v commented Sep 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shatejas Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

0ctopus13prime left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shatejas Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0ctopus13prime left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

navneet1v commented Sep 17, 2025 •

edited

Loading

navneet1v commented Sep 20, 2025 •

edited

Loading

shatejas Sep 22, 2025 •

edited

Loading

shatejas Sep 22, 2025 •

edited

Loading