[BugFix] Fix IOContext, Scorer, and max dimension for Lucene SQ 1 bit

[naveentatikonda]

Description

This PR includes the following bug fixes for Lucene SQ 1 bit (both HNSW and Flat):

• Fix IOContext for .veq file to use SEQUENTIAL instead of RANDOM
• Set max dimension to 16000 instead of 1024
• Fix the scorer to use Bulk SIMD Scorer instead of Lucene scorer

Changes

• Add KNN1040ScalarQuantizedVectorsFormat and KNN1040HnswScalarQuantizedVectorsFormat to use custom bulk SIMD scorer for the FLAT and HNSW index type

• Wire KNN1040ScalarQuantizedVectorsFormat and KNN1040HnswScalarQuantizedVectorsFormat into KNN1040PerFieldKnnVectorsFormat for the FLAT resolver

• Add unit tests for the new KNN1040ScalarQuantizedVectorsFormat and KNN1040HnswScalarQuantizedVectorsFormat and codec resolver

• Rename Faiss104ScalarQuantizedVectorScorer.java => KNN1040ScalarQuantizedVectorScorer.java and Faiss1040ScalarQuantizedUtils.java => KNN1040ScalarQuantizedUtils.java

• Add KNN1040ReadAdviceOverridingDirectory that wraps the segment directory and overrides IOContext to use DataAccessHint.SEQUENTIAL for .veq (quantized vector data) files. Lucene's default reader opens these with DataAccessHint.RANDOM, which disables OS read-ahead prefetching. Since quantized vector data is read sequentially during search, switching to sequential access improves I/O throughput. Raw vector data (.vec) retains random access for rescoring.

• Override fieldsReader in KNN1040ScalarQuantizedVectorsFormat to wrap state.directory with KNN1040ReadAdviceOverridingDirectory before passing it to Lucene104ScalarQuantizedVectorsReader. The HNSW path (KNN1040HnswScalarQuantizedVectorsFormat) inherits this behavior via delegation.

• Add unit tests for KNN1040ReadAdviceOverridingDirectory, KNN1040ScalarQuantizedVectorsFormatTests, and KNN1040HnswScalarQuantizedVectorsFormatTests to verify .veq files are opened with SEQUENTIAL hint

Related Issues

#3178

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

❌ Patch coverage is 98.43750% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 83.16%. Comparing base (6822da0) to head (41c2dc0).
⚠️ Report is 1 commits behind head on 3.6.

Files with missing lines	Patch %	Lines
...N1040Codec/KNN1040ScalarQuantizedVectorScorer.java	83.33%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##                3.6    #3203      +/-   ##
============================================
- Coverage     83.18%   83.16%   -0.03%     
- Complexity     4215     4234      +19     
============================================
  Files           453      456       +3     
  Lines         15455    15507      +52     
  Branches       1972     1974       +2     
============================================
+ Hits          12857    12897      +40     
- Misses         1805     1814       +9     
- Partials        793      796       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[naveentatikonda]

This is the test that is failing

[o.o.k.r.RecallTestsIT    ] [testRecall_when1bitScalarQuantizer_thenRecallAbove60percent] before test
][o.o.k.r.RecallTestsIT    ] [testRecall_when1bitScalarQuantizer_thenRecallAbove60percent] Recall value for SpaceType L2 = 0.8819001770019531
][o.o.k.r.RecallTestsIT    ] [testRecall_when1bitScalarQuantizer_thenRecallAbove60percent] Recall value for SpaceType COSINESIMIL = 0.5164001846313476
][o.o.k.r.RecallTestsIT    ] [testRecall_when1bitScalarQuantizer_thenRecallAbove60percent] after test
  2> REPRODUCE WITH: ./gradlew ':integTest' --tests 'org.opensearch.knn.recall.RecallTestsIT.testRecall_when1bitScalarQuantizer_thenRecallAbove60percent' -Dtests.seed=E6972E43A86BA1B9 -Dtests.security.manager=false -Dtests.locale=csw-CA -Dtests.timezone=Africa/Brazzaville -Druntime.java=25
  2> java.lang.AssertionError: expected:<1.0> but was:<0.5164001846313476>
        at __randomizedtesting.SeedInfo.seed([E6972E43A86BA1B9:2583729F1C9C374]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:555)
        at org.junit.Assert.assertEquals(Assert.java:685)
        at org.opensearch.knn.recall.RecallTestsIT.assertRecall(RecallTestsIT.java:738)
        at org.opensearch.knn.recall.RecallTestsIT.testRecall_when1bitScalarQuantizer_thenRecallAbove60percent(RecallTestsIT.java:729)

Thanks @jack-hung-lgtm for adding this IT which helped to catch a bug in cosine similarity space type with Bulk SIMD. Will keep this PR on hold until the issue is resolved.

This logic was missed in our Bulk SIMD implementation of cosine similarity. We are working on resolving the issue. Thanks!

[0ctopus13prime]

If we end up choosing normalizing query vector within flat vector scorer, then can we avoid checking whether it's an unit vector?
We can just normalize targetCopy, as it's idempotent.
Normalize(Normalize(Normalize(vector))) == Normalize(vector)

[0ctopus13prime]

Also curious, why we are not normalizing query vector in QueryBuilder side likewise we're doing for other cosine index type?

[navneet1v]

This is not needed because in OpenSearch version > 3.1 and less than equal to 3.6 readAdvise is not honored

[navneet1v]

Ref: opensearch-project/OpenSearch#21012

[0ctopus13prime]

NIT : .veb -> .veq