Fix bug "synonym_graph filter fails with word_delimiter_graph when using whitespace or classic tokenizer in synonym_analyzer" by laminelam · Pull Request #19248 · opensearch-project/OpenSearch

laminelam · 2025-09-07T17:29:24Z

This PR fixes the "synonym_graph filter fails with word_delimiter_graph when using whitespace or classic tokenizer in synonym_analyzer" bug

Investigated the issue and looks like there are 2 causes:

When building the analyzers, if one fails for some reason, it throws an exception and the process stops. So the customSynonymAnalyzer does not get instantiated.
On the other hand, if the customSynonymAnalyzer depends on another one that hasn't been built (and registred) yet the process fails too.

analysisRegistry.getAnalyzer(synonymAnalyzerName);

This is not enough because it only looks into the built in and pre built in analyzers. The one from settings are not there.

The solution is two-fold:

Fail safe instead of fail fast when building the analyzers.
Build the depending analyzers first.

Fail safe instead of fail fast when building the analyzers
Right now, if an analyzer fails for some reason the whole building process fails with an exception.

Build the depending analyzers first:
Synonym custom analyzers may depend on another analyzer that has to be built first.
The PR adds a logic to:

add option "order" attribute that defines precedence order between analyzers
add 'analyzersBuiltSoFar' to getChainAwareTokenFilterFactory to pass the already built analyzers needed by the one being built

Resolves #18037

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-09-07T18:39:34Z

❌ Gradle check result for 5c8fbe6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

laminelam · 2025-09-07T21:13:30Z

❌ Gradle check result for 5c8fbe6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Only fail in CI, cannot reproduce locally with same seeds

org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/10_histogram/histogram profiler}

org.opensearch.remotemigration.RemoteMigrationIndexMetadataUpdateIT.testIndexSettingsUpdateAfterIndexMovedToRemoteThroughAllocationExclude

org.opensearch.indices.replication.RemoteStoreReplicationSourceTests.testGetMergedSegmentFilesDownloadTimeout

github-actions · 2025-09-07T22:26:54Z

❌ Gradle check result for 267f48e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

gaobinlong

The DCO check is failed, please amend your commit with '-s' to include the sign off info, and change log is needed.

...lysis-common/src/main/java/org/opensearch/analysis/common/MultiplexerTokenFilterFactory.java

CHANGELOG.md

server/src/main/java/org/opensearch/index/analysis/AnalysisRegistry.java

github-actions · 2026-03-16T20:52:55Z

Persistent review updated to latest commit 1e2895a

github-actions · 2026-03-16T21:36:02Z

❌ Gradle check result for 1e2895a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2026-03-16T22:15:29Z

Persistent review updated to latest commit 739a656

github-actions · 2026-03-16T22:17:48Z

Persistent review updated to latest commit e73f993

github-actions · 2026-03-16T22:26:33Z

Persistent review updated to latest commit 3fcb384

github-actions · 2026-03-16T23:35:41Z

❌ Gradle check result for 3fcb384: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2026-03-18T02:33:44Z

❌ Gradle check result for 3fcb384: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

…whitespace or classic tokenizer in synonym_analyzer" bug opensearch-project#18037 add 'analyzersBuiltSoFar' to getChainAwareTokenFilterFactory to build custom analyzers depending on other (already built) analyzers The analyzers are built following the order of precedence specified in the settings Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com>

…rgs) Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com>

Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com>

…g kahn's algorithm topological sort Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com>

Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com>

Signed-off-by: Andrew Ross <andrross@amazon.com>

github-actions · 2026-03-18T23:15:21Z

Persistent review updated to latest commit 03518e0

github-actions · 2026-03-19T00:30:58Z

✅ Gradle check result for 03518e0: SUCCESS

Fix "synonym_graph filter fails with word_delimiter_graph when using whitespace or classic tokenizer in synonym_analyzer" bug. Use automatic dependency detection using kahn's algorithm topological sort. Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Lamine Idjeraoui <lidjeraoui@apple.com> Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: kkewwei <kkewwei@163.com>

laminelam requested review from a team, Bukhtawar, CEHENKLE, Rishikesh1159, anasalkouz, andrross, ashking94, cwperks, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh, owaiskazi19, reta, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners September 7, 2025 17:29

github-actions bot added bug Something isn't working Indexing Indexing, Bulk Indexing and anything related to indexing labels Sep 7, 2025

laminelam mentioned this pull request Sep 7, 2025

[BUG] synonym_graph filter fails with word_delimiter_graph when using whitespace or classic tokenizer in synonym_analyzer – similar to #16263 #18037

Closed

gaobinlong reviewed Sep 17, 2025

View reviewed changes

...lysis-common/src/main/java/org/opensearch/analysis/common/MultiplexerTokenFilterFactory.java Outdated Show resolved Hide resolved

andrross reviewed Mar 16, 2026

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

andrross reviewed Mar 16, 2026

View reviewed changes

server/src/main/java/org/opensearch/index/analysis/AnalysisRegistry.java Outdated Show resolved Hide resolved

opensearch-ci-bot mentioned this pull request Mar 16, 2026

[AUTOCUT] Gradle Check Flaky Test Report for FullClusterRestartIT #15813

Open

andrross force-pushed the feature/synonym_graph_bug_97 branch from 1e2895a to 739a656 Compare March 16, 2026 22:14

andrross force-pushed the feature/synonym_graph_bug_97 branch 2 times, most recently from cebf2d0 to 3fcb384 Compare March 16, 2026 22:24

andrross approved these changes Mar 16, 2026

View reviewed changes

msfroh approved these changes Mar 18, 2026

View reviewed changes

Lamine Idjeraoui and others added 8 commits March 18, 2026 16:13

Maintain the two versions of getChainAwareTokenFilterFactory (4 & 5 a…

c6d1317

…rgs) Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com>

change order's default value to Integer.MAX_VALUE

0a70e94

Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com>

update change log

23fe1fa

Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com>

replace 'order' attribute with an automatic dependency detection usin…

6c281fb

…g kahn's algorithm topological sort Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com>

improve how test exceptions are caught

6cddb65

Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com>

fix merge issue

54ccbb9

Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com>

Fix CHANGELOG

03518e0

Signed-off-by: Andrew Ross <andrross@amazon.com>

andrross force-pushed the feature/synonym_graph_bug_97 branch from 3fcb384 to 03518e0 Compare March 18, 2026 23:13

andrross merged commit 9c29462 into opensearch-project:main Mar 19, 2026
33 of 44 checks passed

opensearch-ci-bot mentioned this pull request Mar 19, 2026

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT #14288

Open

andrross mentioned this pull request Mar 21, 2026

Add release notes for 3.6.0 andrross/OpenSearch#216

Open

1 task

Conversation

laminelam commented Sep 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Check List

Uh oh!

github-actions bot commented Sep 7, 2025

Uh oh!

laminelam commented Sep 7, 2025

Uh oh!

github-actions bot commented Sep 7, 2025

Uh oh!

gaobinlong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 16, 2026

Uh oh!

github-actions bot commented Mar 16, 2026

Uh oh!

github-actions bot commented Mar 16, 2026

Uh oh!

github-actions bot commented Mar 16, 2026

Uh oh!

github-actions bot commented Mar 16, 2026

Uh oh!

github-actions bot commented Mar 16, 2026

Uh oh!

github-actions bot commented Mar 18, 2026

Uh oh!

github-actions bot commented Mar 18, 2026

Uh oh!

github-actions bot commented Mar 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

laminelam commented Sep 7, 2025 •

edited

Loading