Upgrade lucene to version 10.2.1 by expani · Pull Request #17961 · opensearch-project/OpenSearch

expani · 2025-04-16T04:32:59Z

Description

Upgrading to Lucene 10.2.1
https://lucene.apache.org/core/10_2_1/changes/Changes.html

Performance Testing Areas

Snapshot generation and testing search heavy workloads like Big5 via multiple runs
Ensuring no new regressions seen in indexing like force merge time seen with Lucene 10.1.0

github-actions · 2025-04-16T04:38:05Z

❌ Gradle check result for 5e74113: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-04-16T04:49:17Z

❌ Gradle check result for 93c4e0c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-04-16T05:00:14Z

❌ Gradle check result for 9a34fb8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-04-16T05:25:52Z

❌ Gradle check result for 84276e4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

harshavamsi · 2025-04-16T05:28:37Z

@expani see https://github.com/opensearch-project/OpenSearch/pull/17960/files#diff-53173a30404a65ce7f35073d65258143cfa2d47947ff8e0817337ff3d37e01f3L368 -- let's make use of the ConstantScoreScorerSupplier

expani · 2025-04-16T05:35:09Z

@harshavamsi I was thinking if you can merge the constant scorer change with some context as to why it helps. I can rebase it once merged and fix Lucene 10.2.0 upgrade stuff.

I want to focus on test failures in this PR. Like this one

REPRODUCE WITH: ./gradlew ':plugins:analysis-icu:test' --tests "org.opensearch.index.analysis.IcuTokenizerFactoryTests.testIcuCustomizeRuleFile" -Dtests.seed=5147BC3990E890F4 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=kw -Dtests.timezone=Pacific/Gambier -Druntime.java=21

IcuTokenizerFactoryTests > testIcuCustomizeRuleFile FAILED
    java.lang.ExceptionInInitializerError
        at __randomizedtesting.SeedInfo.seed([5147BC3990E890F4:91E71A7FBEA22BE7]:0)
        at org.opensearch.index.analysis.AnalysisRegistry.buildMapping(AnalysisRegistry.java:541)
        at org.opensearch.index.analysis.AnalysisRegistry.buildTokenFilterFactories(AnalysisRegistry.java:338)
        at org.opensearch.index.analysis.AnalysisRegistry.build(AnalysisRegistry.java:241)
        at org.opensearch.test.OpenSearchTestCase.createTestAnalysis(OpenSearchTestCase.java:1767)
        at org.opensearch.test.OpenSearchTestCase.createTestAnalysis(OpenSearchTestCase.java:1755)
        at org.opensearch.index.analysis.IcuTokenizerFactoryTests.createTestAnalysis(IcuTokenizerFactoryTests.java:129)
        at org.opensearch.index.analysis.IcuTokenizerFactoryTests.testIcuCustomizeRuleFile(IcuTokenizerFactoryTests.java:67)

        Caused by:
        com.ibm.icu.util.ICUUncheckedIOException: java.io.IOException: ICU data file error: Header authentication failed, please check if you have a valid ICU data file; data format 4e726d32, format version 5.0.0.0
            at app//com.ibm.icu.impl.Normalizer2Impl.load(Normalizer2Impl.java:506)
            at app//com.ibm.icu.impl.Norm2AllModes$1.createInstance(Norm2AllModes.java:354)
            at app//com.ibm.icu.impl.Norm2AllModes$1.createInstance(Norm2AllModes.java:347)
            at app//com.ibm.icu.impl.SoftCache.getInstance(SoftCache.java:69)
            at app//com.ibm.icu.impl.Norm2AllModes.getInstance(Norm2AllModes.java:344)
            at app//com.ibm.icu.text.Normalizer2.getInstance(Normalizer2.java:219)
            at app//org.opensearch.index.analysis.IcuFoldingTokenFilterFactory.<clinit>(IcuFoldingTokenFilterFactory.java:57)
            ... 7 more

            Caused by:
            java.io.IOException: ICU data file error: Header authentication failed, please check if you have a valid ICU data file; data format 4e726d32, format version 5.0.0.0
                at com.ibm.icu.impl.ICUBinary.readHeader(ICUBinary.java:606)
                at com.ibm.icu.impl.ICUBinary.readHeaderAndDataVersion(ICUBinary.java:557)
                at com.ibm.icu.impl.Normalizer2Impl.load(Normalizer2Impl.java:453)
                ... 13 more

harshavamsi · 2025-04-16T05:56:11Z

@harshavamsi I was thinking if you can merge the constant scorer change with some context as to why it helps. I can rebase it once merged and fix Lucene 10.2.0 upgrade stuff.

I want to focus on test failures in this PR. Like this one

REPRODUCE WITH: ./gradlew ':plugins:analysis-icu:test' --tests "org.opensearch.index.analysis.IcuTokenizerFactoryTests.testIcuCustomizeRuleFile" -Dtests.seed=5147BC3990E890F4 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=kw -Dtests.timezone=Pacific/Gambier -Druntime.java=21

IcuTokenizerFactoryTests > testIcuCustomizeRuleFile FAILED
    java.lang.ExceptionInInitializerError
        at __randomizedtesting.SeedInfo.seed([5147BC3990E890F4:91E71A7FBEA22BE7]:0)
        at org.opensearch.index.analysis.AnalysisRegistry.buildMapping(AnalysisRegistry.java:541)
        at org.opensearch.index.analysis.AnalysisRegistry.buildTokenFilterFactories(AnalysisRegistry.java:338)
        at org.opensearch.index.analysis.AnalysisRegistry.build(AnalysisRegistry.java:241)
        at org.opensearch.test.OpenSearchTestCase.createTestAnalysis(OpenSearchTestCase.java:1767)
        at org.opensearch.test.OpenSearchTestCase.createTestAnalysis(OpenSearchTestCase.java:1755)
        at org.opensearch.index.analysis.IcuTokenizerFactoryTests.createTestAnalysis(IcuTokenizerFactoryTests.java:129)
        at org.opensearch.index.analysis.IcuTokenizerFactoryTests.testIcuCustomizeRuleFile(IcuTokenizerFactoryTests.java:67)

        Caused by:
        com.ibm.icu.util.ICUUncheckedIOException: java.io.IOException: ICU data file error: Header authentication failed, please check if you have a valid ICU data file; data format 4e726d32, format version 5.0.0.0
            at app//com.ibm.icu.impl.Normalizer2Impl.load(Normalizer2Impl.java:506)
            at app//com.ibm.icu.impl.Norm2AllModes$1.createInstance(Norm2AllModes.java:354)
            at app//com.ibm.icu.impl.Norm2AllModes$1.createInstance(Norm2AllModes.java:347)
            at app//com.ibm.icu.impl.SoftCache.getInstance(SoftCache.java:69)
            at app//com.ibm.icu.impl.Norm2AllModes.getInstance(Norm2AllModes.java:344)
            at app//com.ibm.icu.text.Normalizer2.getInstance(Normalizer2.java:219)
            at app//org.opensearch.index.analysis.IcuFoldingTokenFilterFactory.<clinit>(IcuFoldingTokenFilterFactory.java:57)
            ... 7 more

            Caused by:
            java.io.IOException: ICU data file error: Header authentication failed, please check if you have a valid ICU data file; data format 4e726d32, format version 5.0.0.0
                at com.ibm.icu.impl.ICUBinary.readHeader(ICUBinary.java:606)
                at com.ibm.icu.impl.ICUBinary.readHeaderAndDataVersion(ICUBinary.java:557)
                at com.ibm.icu.impl.Normalizer2Impl.load(Normalizer2Impl.java:453)
                ... 13 more

sounds good, i'll add some context

github-actions · 2025-04-16T07:18:15Z

❌ Gradle check result for 48d60bc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

expani · 2025-04-16T07:39:24Z

Failure due to known flaky test #15806 with different seeds

./gradlew ':server:internalClusterTest' --tests "org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testSnapshotWithStuckNode" -Dtests.seed=A77C55DF82AC524C -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ann -Dtests.timezone=W-SU -Druntime.java=21

Tried the same with current mainline and it fails as well. So, don't think it's related to Lucene 10.2.0 upgrade.

Signed-off-by: expani <anijainc@amazon.com>

Signed-off-by: Andrew Ross <andrross@amazon.com>

Signed-off-by: expani <anijainc@amazon.com>

…h current ConstantScoreSupplier Signed-off-by: expani <anijainc@amazon.com>

Signed-off-by: expani <anijainc@amazon.com>

github-actions · 2025-05-15T20:51:10Z

✅ Gradle check result for 9d8ce0b: SUCCESS

andrross · 2025-05-15T20:59:40Z

@expani What do you think? Should we merge this?

getsaurabh02 · 2025-05-19T17:53:31Z

@andrross @expani @harshavamsi Can we please merge this and follwup on the regressions separately, with a plan for 3.1

asimmahmood1 · 2025-05-19T19:21:20Z

@andrross @expani is OOO, I'm ok the merge this. I can help track any of the performance regression with goal to get fixed by 3.1 release.

andrross · 2025-05-19T19:52:16Z

I've got some commits in this PR, so I'd like to get another maintainer review. @mch2 @msfroh Can one of you take a look?

* Upgrade lucene to version 10.2.0 Signed-off-by: expani <anijainc@amazon.com> * Removed usage of non public constructor for DocIdSetBuilder Signed-off-by: expani <anijainc@amazon.com> * Increment version and fixed another compilation error Signed-off-by: expani <anijainc@amazon.com> * Updating license sha for lucene 10.2.0 Signed-off-by: expani <anijainc@amazon.com> * Upgraded icu4j in conjunction with Lucene 10.2.0 Signed-off-by: expani <anijainc@amazon.com> * update sha for icu4j Signed-off-by: expani <anijainc@amazon.com> * Update to 10.2.1 Signed-off-by: Andrew Ross <andrross@amazon.com> * Add changelog entry Signed-off-by: Andrew Ross <andrross@amazon.com> * Updated test based on Lucene-opensearch-project#14561 Signed-off-by: expani <anijainc@amazon.com> * Updated test based on Lucene-14561 Signed-off-by: expani <anijainc@amazon.com> * Updated test based on Lucene-14561 Signed-off-by: expani <anijainc@amazon.com> * Updated test based on Lucene-14561 Signed-off-by: expani <anijainc@amazon.com> * Delegating nextDoc to advance as previous assumption doesn't hold with current ConstantScoreSupplier Signed-off-by: expani <anijainc@amazon.com> * Implemented cost function Signed-off-by: expani <anijainc@amazon.com> --------- Signed-off-by: expani <anijainc@amazon.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com>

* Upgrade lucene to version 10.2.0 Signed-off-by: expani <anijainc@amazon.com> * Removed usage of non public constructor for DocIdSetBuilder Signed-off-by: expani <anijainc@amazon.com> * Increment version and fixed another compilation error Signed-off-by: expani <anijainc@amazon.com> * Updating license sha for lucene 10.2.0 Signed-off-by: expani <anijainc@amazon.com> * Upgraded icu4j in conjunction with Lucene 10.2.0 Signed-off-by: expani <anijainc@amazon.com> * update sha for icu4j Signed-off-by: expani <anijainc@amazon.com> * Update to 10.2.1 Signed-off-by: Andrew Ross <andrross@amazon.com> * Add changelog entry Signed-off-by: Andrew Ross <andrross@amazon.com> * Updated test based on Lucene-opensearch-project#14561 Signed-off-by: expani <anijainc@amazon.com> * Updated test based on Lucene-14561 Signed-off-by: expani <anijainc@amazon.com> * Updated test based on Lucene-14561 Signed-off-by: expani <anijainc@amazon.com> * Updated test based on Lucene-14561 Signed-off-by: expani <anijainc@amazon.com> * Delegating nextDoc to advance as previous assumption doesn't hold with current ConstantScoreSupplier Signed-off-by: expani <anijainc@amazon.com> * Implemented cost function Signed-off-by: expani <anijainc@amazon.com> --------- Signed-off-by: expani <anijainc@amazon.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com>Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com>

* Upgrade lucene to version 10.2.0 Signed-off-by: expani <anijainc@amazon.com> * Removed usage of non public constructor for DocIdSetBuilder Signed-off-by: expani <anijainc@amazon.com> * Increment version and fixed another compilation error Signed-off-by: expani <anijainc@amazon.com> * Updating license sha for lucene 10.2.0 Signed-off-by: expani <anijainc@amazon.com> * Upgraded icu4j in conjunction with Lucene 10.2.0 Signed-off-by: expani <anijainc@amazon.com> * update sha for icu4j Signed-off-by: expani <anijainc@amazon.com> * Update to 10.2.1 Signed-off-by: Andrew Ross <andrross@amazon.com> * Add changelog entry Signed-off-by: Andrew Ross <andrross@amazon.com> * Updated test based on Lucene-opensearch-project#14561 Signed-off-by: expani <anijainc@amazon.com> * Updated test based on Lucene-14561 Signed-off-by: expani <anijainc@amazon.com> * Updated test based on Lucene-14561 Signed-off-by: expani <anijainc@amazon.com> * Updated test based on Lucene-14561 Signed-off-by: expani <anijainc@amazon.com> * Delegating nextDoc to advance as previous assumption doesn't hold with current ConstantScoreSupplier Signed-off-by: expani <anijainc@amazon.com> * Implemented cost function Signed-off-by: expani <anijainc@amazon.com> --------- Signed-off-by: expani <anijainc@amazon.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com>

opensearch-ci-bot mentioned this pull request Apr 16, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RangeAggregatorTests #17928

Closed

expani marked this pull request as ready for review April 16, 2025 07:40

expani requested review from Bukhtawar, CEHENKLE, Rishikesh1159, anasalkouz, andrross, ashking94, cwperks, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, sachinpkale, saratvemulapalli and shwetathareja as code owners April 16, 2025 07:40

expani and others added 9 commits May 15, 2025 12:41

update sha for icu4j

ad6b93a

Signed-off-by: expani <anijainc@amazon.com>

Update to 10.2.1

e23623c

Signed-off-by: Andrew Ross <andrross@amazon.com>

Add changelog entry

22fd352

Signed-off-by: Andrew Ross <andrross@amazon.com>

Updated test based on Lucene-opensearch-project#14561

9752705

Signed-off-by: expani <anijainc@amazon.com>

Updated test based on Lucene-14561

12840f8

Signed-off-by: expani <anijainc@amazon.com>

Updated test based on Lucene-14561

0ac1659

Signed-off-by: expani <anijainc@amazon.com>

Updated test based on Lucene-14561

a758b06

Signed-off-by: expani <anijainc@amazon.com>

Delegating nextDoc to advance as previous assumption doesn't hold wit…

87b0e38

…h current ConstantScoreSupplier Signed-off-by: expani <anijainc@amazon.com>

Implemented cost function

9d8ce0b

Signed-off-by: expani <anijainc@amazon.com>

andrross force-pushed the lucene_10_2_0_upgrade branch from d7b39c5 to 9d8ce0b Compare May 15, 2025 19:42

harshavamsi mentioned this pull request May 15, 2025

[BUG] Regression in Range and sort queries with Lucene 10.2.1 #18313

Closed

andrross approved these changes May 19, 2025

View reviewed changes

mch2 approved these changes May 19, 2025

View reviewed changes

mch2 merged commit 370cd8c into opensearch-project:main May 19, 2025
29 of 30 checks passed

This was referenced May 21, 2025

Update big5 (id_4) to test against Lucene 10.2.1 #18353

Merged

Fix performance regression in ApproximatePointRangeQuery with Lucene 10.2.1 change #18358

Merged

asimmahmood1 mentioned this pull request May 28, 2025

[Performance] big5.cardinality-agg-high regressed in OS 3.1 #18385

Open

BrewTestBot mentioned this pull request Jun 24, 2025

opensearch 3.1.0 Homebrew/homebrew-core#228059

Merged

opensearch-ci-bot mentioned this pull request Aug 17, 2025

[AUTOCUT] Gradle Check Flaky Test Report for QueryPhaseTests #14551

Open

opensearch-ci-bot mentioned this pull request Sep 25, 2025

[AUTOCUT] Gradle Check Flaky Test Report for TelemetryMetricsEnabledSanityIT #19422

Open

This was referenced Jan 10, 2026

docs: add lucene-upgrade report for v3.1.0 tkykenmt/opensearch-feature-explorer#936

Merged

[feature] Lucene Upgrade tkykenmt/opensearch-feature-explorer#923

Closed

Conversation

expani commented Apr 16, 2025 • edited by andrross Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Performance Testing Areas

Uh oh!

github-actions bot commented Apr 16, 2025

Uh oh!

github-actions bot commented Apr 16, 2025

Uh oh!

github-actions bot commented Apr 16, 2025

Uh oh!

github-actions bot commented Apr 16, 2025

Uh oh!

harshavamsi commented Apr 16, 2025

Uh oh!

expani commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

harshavamsi commented Apr 16, 2025

Uh oh!

github-actions bot commented Apr 16, 2025

Uh oh!

expani commented Apr 16, 2025

Uh oh!

github-actions bot commented May 15, 2025

Uh oh!

andrross commented May 15, 2025

Uh oh!

getsaurabh02 commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asimmahmood1 commented May 19, 2025

Uh oh!

andrross commented May 19, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

expani commented Apr 16, 2025 •

edited by andrross

Loading

expani commented Apr 16, 2025 •

edited

Loading

getsaurabh02 commented May 19, 2025 •

edited

Loading