Store vector L2 norm to restore original vectors for Faiss cosine with derived source by krocky-cooky · Pull Request #3216 · opensearch-project/k-NN

krocky-cooky · 2026-03-27T00:24:22Z

Description

Problem

When Faiss engine with cosine similarity and derived source are both enabled, vectors are L2-normalized in-place
during indexing. Since derived source reconstructs _source from doc values, it returns normalized vectors instead
of the user's original input.

Solution

Store the pre-normalization L2 norm as a FloatDocValuesField (_knn_norm_<field_name>) alongside the vector. On _source reconstruction, read the norm and multiply it back to recover the original vector.

Key changes

Write path:

VectorTransformer / NormalizeVectorTransformer: Added computeL2Norm() to calculate norm before normalization
KNNVectorFieldMapper.parseCreateField: Computes norm, passes it to getFieldsForFloatVector which adds a
FloatDocValuesField when derived source is enabled and norm ≠ 1.0
DerivedSourceIndexOperationListener: Denormalizes vectors before injecting into translog source
KNN10010DerivedSourceStoredFieldsWriter: Records norm field names in segment attributes

Read path:

DerivedSourceNormSupplier: Functional interface with UNIT (no-op) and fromDocValues (reads from NumericDocValues) implementations
PerFieldDerivedVectorTransformerFactory: Looks up norm FieldInfo and creates the appropriate DerivedSourceNormSupplier
AbstractPerFieldDerivedVectorTransformer.formatVector: Lazily reads norm and applies denormalization on both
byte[] (legacy BinaryDocValues) and float[] (Lucene-based vector format) paths

Utilities:

KNNVectorUtil: Added denormalize(vector, norm, inplace) and getNormFieldName(fieldName)

Backward compatibility

Segments without norm attributes fall back to norm = 1.0 (no denormalization)
Only Faiss + cosinesimil + derived source triggers norm storage; all other configurations are unaffected

Related Issues

Resolves #3083

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request [created](https://github.com/opensearch-project/opensearch-api-specification/blob/main/DEVELOPER_GUIDE.md).
Commits are signed per the DCO using --signoff.
Public documentation issue/PR [created](https://github.com/opensearch-project/documentation-website/issues/new/choose).

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here

krocky-cooky · 2026-03-27T00:32:33Z

Among several implementation approaches, I believe this implementation is the best option, though there are a few
considerations:

Due to floating-point rounding errors, there will be minor differences between the original vector and the vector returned in _source.
Since the difference is less than 1e-4, this should not be a significant issue in practice.
The norm is stored as doc values under _knn_norm_<vector_field>, which incurs additional storage usage.
Since it is a single floating-point value and is only stored under the cosinesimil + faiss condition, the impact is
minimal.
During the read phase, there is additional overhead from reading the norm from disk.
Since the norm is stored in
the same file as the vectors, OS-level caching can be leveraged, keeping the impact small.

Happy to address any feedback!

…iss cosine with derived source When Faiss engine with cosine similarity and derived source are both enabled, vectors are L2-normalized at index time, causing _source to return normalized vectors instead of the originals. This change stores the pre-normalization L2 norm as a FloatDocValuesField and uses it to denormalize vectors when reconstructing _source from doc values or translog. Signed-off-by: Takuo Kuroki <kurotaku9679.sub@gmail.com>

Signed-off-by: Takuo Kuroki <kurotaku9679.sub@gmail.com>

navneet1v · 2026-03-27T18:03:36Z

@krocky-cooky thanks for raising the PR. Will start reviewing the PR.

cc: @Vikasht34 , @shatejas , @kotwanikunal

kotwanikunal

Thanks for making the changes. Took an initial pass through the changes.
I'll revisit it again.

kotwanikunal · 2026-03-30T22:10:20Z

src/main/java/org/opensearch/knn/common/KNNVectorUtil.java

+     * @return the denormalized vector
+     */
+    public static float[] denormalize(float[] vector, float norm, boolean inplace) {
+        float[] result = inplace ? vector : Arrays.copyOf(vector, vector.length);


nit: add in checks here to prevent NPEs

public static float[] denormalize(float[] vector, float norm, boolean inplace) { Objects.requireNonNull(vector, "vector must not be null"); if (norm <= 0 || !Float.isFinite(norm)) { throw new IllegalArgumentException("norm must be a positive finite value, got: " + norm); } ... }

kotwanikunal · 2026-03-30T22:14:48Z

src/main/java/org/opensearch/knn/index/codec/derivedsource/DerivedSourceNormSupplier.java

+    static DerivedSourceNormSupplier fromDocValues(CheckedSupplier<NumericDocValues, IOException> supplier) {
+        return (docId) -> {
+            NumericDocValues dv = supplier.get();
+            dv.advance(docId);


This could go past to NO_MORE_DOCS if we are missing the norm in doc values (compatibility or otherwise).
Can you add in checks to ensure it returns the correct value? Or better use advanceExact - https://lucene.apache.org/core/10_4_0/core/org/apache/lucene/index/NumericDocValues.html#advanceExact(int)

kotwanikunal · 2026-03-30T22:20:38Z

src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java

            getVectorValidator().validateVector(array);
-            getVectorTransformer().transform(array, true);
-            context.doc().addAll(getFieldsForFloatVector(array, isDerivedEnabled(context)));
+            VectorTransformer transformer = getVectorTransformer();


Shouldn't this be limited just to the Cosine space? Given that we have access to VectorDataType?

kotwanikunal · 2026-03-30T22:24:15Z

...a/org/opensearch/knn/index/codec/derivedsource/AbstractPerFieldDerivedVectorTransformer.java

-            return KNNVectorFieldMapperUtil.deserializeStoredVector(vectorBytesRef, vectorDataType);
+            Object deserialized = KNNVectorFieldMapperUtil.deserializeStoredVector(vectorBytesRef, vectorDataType);
+            if (deserialized instanceof float[] floatVector) {
+                float norm = normSupplier.get();


nit: This code seems duplicated. Consider moving it into a method

float[] vector = ... float norm = normSupplier.get(); if (norm != 1.0f) { KNNVectorUtil.denormalize(vector, norm, true); }

kotwanikunal · 2026-03-30T22:32:07Z

src/test/java/org/opensearch/knn/integ/DerivedSourceIT.java

+        );
+
+        // Index documents with non-unit vectors
+        Random random = new Random(42);


Can we also test with [0, 0,...] values? Just to ensure we cover the multiplicative factors correctly?

kotwanikunal · 2026-03-30T22:35:18Z

src/main/java/org/opensearch/knn/index/DerivedKnnFloatVectorField.java

     * @param isDerivedEnabled boolean to indicate if derived source is enabled
     */
-    public DerivedKnnFloatVectorField(String name, float[] vector, boolean isDerivedEnabled) {
+    public DerivedKnnFloatVectorField(String name, float[] vector, boolean isDerivedEnabled, float norm) {


Can we maintain the original signature with a 1.0 value?

public DerivedKnnFloatVectorField(String name, float[] vector, boolean isDerivedEnabled) { ... this.vectorNorm = 1.0f; }

krocky-cooky requested review from 0ctopus13prime, VijayanB, Vikasht34, heemin32, jmazanec15, junqiu-lei, luyuncheng, martin-gaievski, naveentatikonda, navneet1v, ryanbogan, shatejas and vamshin as code owners March 27, 2026 00:24

krocky-cooky force-pushed the feat/return-original-vec-3083 branch from e29f585 to 0fcdcaa Compare March 27, 2026 01:12

krocky-cooky added 5 commits March 27, 2026 10:17

Refactor norm calculation logic

9ac9b5c

Signed-off-by: Takuo Kuroki <kurotaku9679.sub@gmail.com>

changed to detailed commentout

e6291af

Signed-off-by: Takuo Kuroki <kurotaku9679.sub@gmail.com>

add changelog

3957786

Signed-off-by: Takuo Kuroki <kurotaku9679.sub@gmail.com>

Reset jni/external submodules to match main

b452d43

Signed-off-by: Takuo Kuroki <kurotaku9679.sub@gmail.com>

krocky-cooky force-pushed the feat/return-original-vec-3083 branch from 0fcdcaa to b452d43 Compare March 27, 2026 01:18

Merge branch 'main' into feat/return-original-vec-3083

601cb6a

kotwanikunal reviewed Mar 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store vector L2 norm to restore original vectors for Faiss cosine with derived source#3216

Store vector L2 norm to restore original vectors for Faiss cosine with derived source#3216
krocky-cooky wants to merge 6 commits intoopensearch-project:mainfrom
krocky-cooky:feat/return-original-vec-3083

krocky-cooky commented Mar 27, 2026

Uh oh!

krocky-cooky commented Mar 27, 2026

Uh oh!

navneet1v commented Mar 27, 2026

Uh oh!

kotwanikunal left a comment

Uh oh!

kotwanikunal Mar 30, 2026

Uh oh!

kotwanikunal Mar 30, 2026

Uh oh!

kotwanikunal Mar 30, 2026

Uh oh!

kotwanikunal Mar 30, 2026

Uh oh!

kotwanikunal Mar 30, 2026

Uh oh!

kotwanikunal Mar 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

krocky-cooky commented Mar 27, 2026

Description

Problem

Solution

Key changes

Backward compatibility

Related Issues

Check List

Uh oh!

krocky-cooky commented Mar 27, 2026

Uh oh!

navneet1v commented Mar 27, 2026

Uh oh!

kotwanikunal left a comment

Choose a reason for hiding this comment

Uh oh!

kotwanikunal Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

kotwanikunal Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

kotwanikunal Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

kotwanikunal Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

kotwanikunal Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

kotwanikunal Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants