[LuceneOnFaiss] Partial loading support (memory optimized search) for FAISS index.#2630
Merged
0ctopus13prime merged 11 commits intoopensearch-project:mainfrom Mar 28, 2025
Merged
Conversation
0ctopus13prime
requested review from
VijayanB,
Vikasht34,
heemin32,
jmazanec15,
junqiu-lei,
luyuncheng,
martin-gaievski,
naveentatikonda,
navneet1v,
ryanbogan,
shatejas and
vamshin
as code owners
0ctopus13prime
force-pushed
the
final-lucene-on-faiss
branch
from
e2e8763 to
115f0d8
Compare
0ctopus13prime
force-pushed
the
final-lucene-on-faiss
branch
from
115f0d8 to
2c23673
Compare
jmazanec15
previously approved these changes
Mar 27, 2025
Member
jmazanec15
left a comment
jmazanec15
left a comment
There was a problem hiding this comment.
LGTM - thanks @0ctopus13prime !
shatejas
previously approved these changes
Mar 27, 2025
0ctopus13prime
dismissed stale reviews from shatejas and jmazanec15
via
5293dd7
0ctopus13prime
force-pushed
the
final-lucene-on-faiss
branch
3 times, most recently
from
5293dd7 to
c2c2edd
Compare
…, only FAISS is supporing this. (opensearch-project#2581) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
…t#2590) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
…2594) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
…pensearch-project#2604) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
…pensearch-project#2608) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
…arch-project#2616) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
…dex. (opensearch-project#2609) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
0ctopus13prime
force-pushed
the
final-lucene-on-faiss
branch
from
c2c2edd to
8c2bd58
Compare
0ctopus13prime
added a commit
to 0ctopus13prime/k-NN
that referenced
this pull request
Mar 27, 2025
Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
0ctopus13prime
added a commit
to 0ctopus13prime/k-NN
that referenced
this pull request
Mar 27, 2025
Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
0ctopus13prime
force-pushed
the
final-lucene-on-faiss
branch
from
8c2bd58 to
8f8d6ee
Compare
Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
0ctopus13prime
force-pushed
the
final-lucene-on-faiss
branch
from
8f8d6ee to
d11a143
Compare
jmazanec15
approved these changes
Mar 28, 2025
shatejas
approved these changes
Mar 28, 2025
epalaz
pushed a commit
to epalaz/k-NN
that referenced
this pull request
Mar 28, 2025
… FAISS index. (opensearch-project#2630) * Adding basic building blocks for MemoryOptimizedSearch. At the moment, only FAISS is supporing this. (opensearch-project#2581) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added IxMp section loading logic from FAISS index. (opensearch-project#2590) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added FaissHNSW and bridge to Lucene HNSW graph. (opensearch-project#2594) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added FAISS float flat index. (opensearch-project#2598) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added FAISS byte index deserializer - FaissIndexScalarQuantizedFlat (opensearch-project#2604) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Enable memory optimized searching in VectorReader for FAISS engine. (opensearch-project#2608) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added FAISS CAGRA index. (opensearch-project#2621) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added index scope setting 'index.knn.memory_optimized_search' (opensearch-project#2616) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Apply monotonic increasing integer encoding to FAISS HNSW and IdMapIndex. (opensearch-project#2609) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Adding byte index, FP16 index decoding. (opensearch-project#2618) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added integration tests for LuceneOnFaiss (opensearch-project#2630) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> --------- Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
luyuncheng
pushed a commit
to luyuncheng/k-NN-1
that referenced
this pull request
Jun 18, 2025
… FAISS index. (opensearch-project#2630) * Adding basic building blocks for MemoryOptimizedSearch. At the moment, only FAISS is supporing this. (opensearch-project#2581) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added IxMp section loading logic from FAISS index. (opensearch-project#2590) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added FaissHNSW and bridge to Lucene HNSW graph. (opensearch-project#2594) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added FAISS float flat index. (opensearch-project#2598) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added FAISS byte index deserializer - FaissIndexScalarQuantizedFlat (opensearch-project#2604) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Enable memory optimized searching in VectorReader for FAISS engine. (opensearch-project#2608) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added FAISS CAGRA index. (opensearch-project#2621) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added index scope setting 'index.knn.memory_optimized_search' (opensearch-project#2616) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Apply monotonic increasing integer encoding to FAISS HNSW and IdMapIndex. (opensearch-project#2609) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Adding byte index, FP16 index decoding. (opensearch-project#2618) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added integration tests for LuceneOnFaiss (opensearch-project#2630) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> --------- Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
jingqimao77-spec
pushed a commit
to jingqimao77-spec/k-NN
that referenced
this pull request
Mar 15, 2026
… FAISS index. (opensearch-project#2630) * Adding basic building blocks for MemoryOptimizedSearch. At the moment, only FAISS is supporing this. (opensearch-project#2581) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added IxMp section loading logic from FAISS index. (opensearch-project#2590) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added FaissHNSW and bridge to Lucene HNSW graph. (opensearch-project#2594) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added FAISS float flat index. (opensearch-project#2598) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added FAISS byte index deserializer - FaissIndexScalarQuantizedFlat (opensearch-project#2604) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Enable memory optimized searching in VectorReader for FAISS engine. (opensearch-project#2608) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added FAISS CAGRA index. (opensearch-project#2621) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added index scope setting 'index.knn.memory_optimized_search' (opensearch-project#2616) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Apply monotonic increasing integer encoding to FAISS HNSW and IdMapIndex. (opensearch-project#2609) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Adding byte index, FP16 index decoding. (opensearch-project#2618) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> * Added integration tests for LuceneOnFaiss (opensearch-project#2630) Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com> --------- Signed-off-by: Dooyong Kim <kdooyong@amazon.com> Co-authored-by: Dooyong Kim <kdooyong@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This will add a new index setting
index.knn.memory_optimized_searchto empower user to run vector search on FAISS index under memory constrained environment.For more details, please refer to
RFC: #2401
This PR has ten sub PRs combined with integration testings.
Related Issues
RFC: #2401
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.