Fix bc-fips jar hell with OpenSearch core 3.6.0 by martin-gaievski · Pull Request #4652 · opensearch-project/ml-commons

martin-gaievski · 2026-02-19T22:11:52Z

OpenSearch core PR #18921 added bc-fips to the distribution lib/ directory via fipsOnly configuration. The ml-algorithms module declares bc-fips as an implementation dependency, which bundles it in the plugin zip. This causes jar hell when installing the ML plugin since core already provides bc-fips.

Fix:

compileOnly "org.bouncycastle:bc-fips:${versions.bouncycastle_jce}"
testRuntimeOnly "org.bouncycastle:bc-fips:${versions.bouncycastle_jce}"

compileOnly means:
Available on the compile classpath → code that imports org.bouncycastle.crypto.fips.* compiles fine
not included in the plugin zip → no duplicate jar → no jar hell
At runtime, the plugin uses the bc-fips-2.1.2.jar already provided by OpenSearch core's lib/ directory

testRuntimeOnly means:
Available on the test runtime classpath → unit tests that exercise aws-encryption-sdk-java (which calls bc-fips internally) can find the classes
Does not affect the plugin zip packaging

Without testRuntimeOnly, the bc-fips jar is absent from the test classpath. When unit tests run code paths that trigger aws-encryption-sdk-java → it tries to load BouncyCastle FIPS classes → ClassNotFoundException → test failure. Adding testRuntimeOnly provides the jar specifically for tests, without affecting the packaged plugin.

Error from neural-search CI: https://github.com/opensearch-project/neural-search/actions/runs/22154798712/job/64212973643

java.lang.IllegalStateException: jar hell!
class: META-INF.versions.11.org.bouncycastle.crypto.fips.FipsSecureRandom
jar1: .../distro/3.6.0-ARCHIVE/lib/bc-fips-2.1.2.jar
jar2: .../plugins/.installing-.../bc-fips-2.1.2.jar

…es it Signed-off-by: Martin Gaievski <gaievski@amazon.com>

coderabbitai · 2026-02-19T22:12:05Z

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

📝 Walkthrough

Walkthrough

The bc-fips dependency in ml-algorithms/build.gradle is reconfigured from a single implementation scope to two separate scopes: compileOnly and testRuntimeOnly. This excludes the dependency from the main runtime classpath while maintaining availability during compilation and test runtime.

Changes

Cohort / File(s)	Summary
Build Configuration `ml-algorithms/build.gradle`	Changed `bc-fips` dependency scope from `implementation` to `compileOnly` and `testRuntimeOnly`, removing it from main runtime classpath.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Description check	❓ Inconclusive	The description thoroughly explains the problem, solution, and rationale, but does not follow the required template format with sections like Related Issues and Check List.	Add the missing template sections: explicitly link Related Issues, and complete the Check List with selections for testing, documentation, commits, and DCO confirmation.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: adjusting the bc-fips dependency scope to resolve jar hell with OpenSearch core 3.6.0.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Tip

Issue Planner is now in beta. Read the docs and try it out! Share your feedback on Discord.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Signed-off-by: Martin Gaievski <gaievski@amazon.com>

dhrubo-os · 2026-02-20T01:53:05Z

@martin-gaievski could you please apply spotless? ./gradlew spotlessApply

martin-gaievski · 2026-02-20T02:01:47Z

@martin-gaievski could you please apply spotless? ./gradlew spotlessApply

I've done that before submitting this change, it doesn't do anything. All my change is in the build.gradle

./gradlew spotlessApply
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 9.2.0
  OS Info               : Mac OS X 26.2 (aarch64)
  JDK Version           : 21 (Amazon Corretto JDK 21 (21.0.3+9-LTS))
  JAVA_HOME             : /Users/gaievski/.sdkman/candidates/java/21.0.3-amzn
  Random Testing Seed   : 46355A471867DEB4
  Crypto Standard       : any-supported
=======================================

[Incubating] Problems report is available at: file:///Users/gaievski/dev/opensearch/ml-commons/build/reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 10.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/9.2.0/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 1s
15 actionable tasks: 15 up-to-date
(base) gaievski@80a99731c143 ml-commons % git status
On branch fix-bc-fips-jar-hell
nothing to commit, working tree clean

dhrubo-os · 2026-02-20T02:17:09Z

@martin-gaievski could you please apply spotless? ./gradlew spotlessApply

I've done that before submitting this change, it doesn't do anything. All my change is in the build.gradle

./gradlew spotlessApply
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 9.2.0
  OS Info               : Mac OS X 26.2 (aarch64)
  JDK Version           : 21 (Amazon Corretto JDK 21 (21.0.3+9-LTS))
  JAVA_HOME             : /Users/gaievski/.sdkman/candidates/java/21.0.3-amzn
  Random Testing Seed   : 46355A471867DEB4
  Crypto Standard       : any-supported
=======================================

[Incubating] Problems report is available at: file:///Users/gaievski/dev/opensearch/ml-commons/build/reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 10.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/9.2.0/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 1s
15 actionable tasks: 15 up-to-date
(base) gaievski@80a99731c143 ml-commons % git status
On branch fix-bc-fips-jar-hell
nothing to commit, working tree clean

Hmm, usually it doesn't fail: https://github.com/opensearch-project/ml-commons/actions/runs/22206846541/job/64232588172?pr=4652

if spotlessApply is applied.

martin-gaievski · 2026-02-20T02:27:58Z

@martin-gaievski could you please apply spotless? ./gradlew spotlessApply

I've done that before submitting this change, it doesn't do anything. All my change is in the build.gradle

./gradlew spotlessApply
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 9.2.0
  OS Info               : Mac OS X 26.2 (aarch64)
  JDK Version           : 21 (Amazon Corretto JDK 21 (21.0.3+9-LTS))
  JAVA_HOME             : /Users/gaievski/.sdkman/candidates/java/21.0.3-amzn
  Random Testing Seed   : 46355A471867DEB4
  Crypto Standard       : any-supported
=======================================

[Incubating] Problems report is available at: file:///Users/gaievski/dev/opensearch/ml-commons/build/reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 10.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/9.2.0/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 1s
15 actionable tasks: 15 up-to-date
(base) gaievski@80a99731c143 ml-commons % git status
On branch fix-bc-fips-jar-hell
nothing to commit, working tree clean

Hmm, usually it doesn't fail: https://github.com/opensearch-project/ml-commons/actions/runs/22206846541/job/64232588172?pr=4652

if spotlessApply is applied.

It looks like the failure is not related to spotless rules, checkout the actual error:

https://github.com/opensearch-project/ml-commons/actions/runs/22206846541/job/64232588172?pr=4652#step:4:34

FAILURE: Build failed with an exception.

* Where:
Build file '/home/runner/work/ml-commons/ml-commons/client/build.gradle' line: 85

and this is problematic line:
https://github.com/opensearch-project/ml-commons/blob/main/client/build.gradle#L85

project.shadow.component(publication)

mingshl · 2026-02-20T03:05:43Z

* Where:
Build file '/home/runner/work/ml-commons/ml-commons/client/build.gradle' line: 85

* What went wrong:
A problem occurred evaluating project ':opensearch-ml-client'.
> Could not find method component() for arguments [org.gradle.api.publish.maven.internal.publication.DefaultMavenPublication_Decorated@34c6f4f7] on extension 'shadow' of type java.lang.Object.

mingshl · 2026-02-20T03:27:10Z

@martin-gaievski on client/build.gradle file line 85,
would you try replace project.shadow.component(publication) to from components.shadow

martin-gaievski · 2026-02-20T05:47:00Z

@martin-gaievski on client/build.gradle file line 85, would you try replace project.shadow.component(publication) to from components.shadow

thanks @mingshl, I pushed new commit with suggested edit, this time spotless check has passed. I do see other 4 CI actions are pending, not sure if you can kink them off as a repo maintainer.

mingshl · 2026-02-20T06:23:58Z

@martin-gaievski on client/build.gradle file line 85, would you try replace project.shadow.component(publication) to from components.shadow

thanks @mingshl, I pushed new commit with suggested edit, this time spotless check has passed. I do see other 4 CI actions are pending, not sure if you can kink them off as a repo maintainer.

approved CI. This should work. Let's see!!

Replace deprecated project.shadow.component(publication) with from components.shadow in client, common, and spi build.gradle files to fix compatibility with Gradle 9.x. Signed-off-by: Martin Gaievski <gaievski@amazon.com>

dhrubo-os · 2026-02-20T17:19:59Z

| -> Failed installing file:/__w/ml-commons/ml-commons/plugin/build/distributions/opensearch-ml-3.6.0.0-SNAPSHOT.zip
| -> Rolling back opensearch-job-scheduler
| -> Rolled back opensearch-job-scheduler
| -> Rolling back file:/__w/ml-commons/ml-commons/plugin/build/distributions/opensearch-ml-3.6.0.0-SNAPSHOT.zip
| -> Rolled back file:/__w/ml-commons/ml-commons/plugin/build/distributions/opensearch-ml-3.6.0.0-SNAPSHOT.zip
| Exception in thread "main" java.lang.IllegalStateException: failed to load plugin opensearch-ml due to jar hell
| 	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:779)
| 	at org.opensearch.plugins.PluginsService.checkJarHellForPlugin(PluginsService.java:404)
| 	at org.opensearch.tools.cli.plugin.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:834)
| 	at org.opensearch.tools.cli.plugin.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:811)
| 	at org.opensearch.tools.cli.plugin.InstallPluginCommand.installPlugin(InstallPluginCommand.java:846)
| 	at org.opensearch.tools.cli.plugin.InstallPluginCommand.execute(InstallPluginCommand.java:277)
| 	at org.opensearch.tools.cli.plugin.InstallPluginCommand.execute(InstallPluginCommand.java:251)
| 	at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110)
| 	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
| 	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
| 	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
| 	at org.opensearch.cli.Command.main(Command.java:101)
| 	at org.opensearch.tools.cli.plugin.PluginCli.main(PluginCli.java:66)
| Caused by: java.lang.IllegalStateException: jar hell!
| class: META-INF.versions.11.org.bouncycastle.crypto.fips.FipsSecureRandom$Random11Spi
| jar1: /__w/ml-commons/ml-commons/plugin/build/testclusters/integTest-0/distro/3.6.0-ARCHIVE/plugins/.installing-2484764218645680865/bc-fips-2.1.2.jar
| jar2: /__w/ml-commons/ml-commons/plugin/build/testclusters/integTest-0/distro/3.6.0-ARCHIVE/lib/bc-fips-2.1.2.jar
| 	at org.opensearch.common.bootstrap.JarHell.checkClass(JarHell.java:316)
| 	at org.opensearch.common.bootstrap.JarHell.checkJarHell(JarHell.java:215)
| 	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:777)
| 	... 12 more

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':opensearch-ml-plugin:integTest'.
> Process 'command './bin/opensearch-plugin'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to generate a Build Scan (powered by Develocity).
> Get more help at https://help.gradle.org./

BUILD FAILED in 6m 19s
Error: Process completed with exit code 1.

Seems like it didn't fix the jar hell issue?

martin-gaievski · 2026-02-20T17:35:43Z

| -> Failed installing file:/__w/ml-commons/ml-commons/plugin/build/distributions/opensearch-ml-3.6.0.0-SNAPSHOT.zip
| -> Rolling back opensearch-job-scheduler
| -> Rolled back opensearch-job-scheduler
| -> Rolling back file:/__w/ml-commons/ml-commons/plugin/build/distributions/opensearch-ml-3.6.0.0-SNAPSHOT.zip
| -> Rolled back file:/__w/ml-commons/ml-commons/plugin/build/distributions/opensearch-ml-3.6.0.0-SNAPSHOT.zip
| Exception in thread "main" java.lang.IllegalStateException: failed to load plugin opensearch-ml due to jar hell
| 	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:779)
| 	at org.opensearch.plugins.PluginsService.checkJarHellForPlugin(PluginsService.java:404)
| 	at org.opensearch.tools.cli.plugin.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:834)
| 	at org.opensearch.tools.cli.plugin.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:811)
| 	at org.opensearch.tools.cli.plugin.InstallPluginCommand.installPlugin(InstallPluginCommand.java:846)
| 	at org.opensearch.tools.cli.plugin.InstallPluginCommand.execute(InstallPluginCommand.java:277)
| 	at org.opensearch.tools.cli.plugin.InstallPluginCommand.execute(InstallPluginCommand.java:251)
| 	at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110)
| 	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
| 	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
| 	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
| 	at org.opensearch.cli.Command.main(Command.java:101)
| 	at org.opensearch.tools.cli.plugin.PluginCli.main(PluginCli.java:66)
| Caused by: java.lang.IllegalStateException: jar hell!
| class: META-INF.versions.11.org.bouncycastle.crypto.fips.FipsSecureRandom$Random11Spi
| jar1: /__w/ml-commons/ml-commons/plugin/build/testclusters/integTest-0/distro/3.6.0-ARCHIVE/plugins/.installing-2484764218645680865/bc-fips-2.1.2.jar
| jar2: /__w/ml-commons/ml-commons/plugin/build/testclusters/integTest-0/distro/3.6.0-ARCHIVE/lib/bc-fips-2.1.2.jar
| 	at org.opensearch.common.bootstrap.JarHell.checkClass(JarHell.java:316)
| 	at org.opensearch.common.bootstrap.JarHell.checkJarHell(JarHell.java:215)
| 	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:777)
| 	... 12 more

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':opensearch-ml-plugin:integTest'.
> Process 'command './bin/opensearch-plugin'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to generate a Build Scan (powered by Develocity).
> Get more help at https://help.gradle.org./

BUILD FAILED in 6m 19s
Error: Process completed with exit code 1.

Seems like it didn't fix the jar hell issue?

Looking into this problem more deeply, it may be the issue with the opensearch build system, jar hell check should skip jars that match between lib/ and plugin when they are the same version. And the standard snapshot should not include FIPS jars, due to this discrepancy the error only can be seen in CI, not in local environment.

cwperks · 2026-02-20T17:46:13Z

@martin-gaievski this fixes ML to adapt to the gradle shadow plugin upgrade from v8 to v9, but this repo also needs to become aware of the FIPS build param. I will raise a PR shortly.

cwperks · 2026-02-20T17:52:14Z

@martin-gaievski @dhrubo-os @mingshl This should address the FIPS build param too: #4654

dbwiddis · 2026-02-20T21:34:21Z

Looks like this is superceded by #4654

dhrubo-os · 2026-02-24T19:10:52Z

Closing the PR, as this is already solved.

Fix bc-fips jar hell by changing to compileOnly since core now provid…

c2f397b

…es it Signed-off-by: Martin Gaievski <gaievski@amazon.com>

martin-gaievski closed this Feb 19, 2026

martin-gaievski reopened this Feb 19, 2026

reverting scope of dependency from compile to implementation

cb94c62

Signed-off-by: Martin Gaievski <gaievski@amazon.com>

martin-gaievski force-pushed the fix-bc-fips-jar-hell branch from 78132c0 to 5e5abf6 Compare February 20, 2026 00:59

martin-gaievski marked this pull request as ready for review February 20, 2026 01:07

martin-gaievski requested review from HenryL27, Zhangxunmt, austintlee, b4sjoo, dhrubo-os, jngz-es, mingshl, model-collapse, pyek-bot, rbhavna, sam-herman, xinyual, ylwu-amzn and zane-neo as code owners February 20, 2026 01:07

martin-gaievski force-pushed the fix-bc-fips-jar-hell branch from 5e5abf6 to 966bbe4 Compare February 20, 2026 05:42

martin-gaievski had a problem deploying to ml-commons-cicd-env-require-approval February 20, 2026 05:44 — with GitHub Actions Failure

martin-gaievski had a problem deploying to ml-commons-cicd-env-require-approval February 20, 2026 05:44 — with GitHub Actions Error

martin-gaievski closed this Feb 20, 2026

martin-gaievski reopened this Feb 20, 2026

martin-gaievski had a problem deploying to ml-commons-cicd-env-require-approval February 20, 2026 15:50 — with GitHub Actions Failure

martin-gaievski had a problem deploying to ml-commons-cicd-env-require-approval February 20, 2026 15:50 — with GitHub Actions Error

martin-gaievski had a problem deploying to ml-commons-cicd-env-require-approval February 20, 2026 15:50 — with GitHub Actions Failure

martin-gaievski force-pushed the fix-bc-fips-jar-hell branch from 966bbe4 to 16f1feb Compare February 20, 2026 16:09

Fix deprecated shadow plugin component API

81ef5d0

Replace deprecated project.shadow.component(publication) with from components.shadow in client, common, and spi build.gradle files to fix compatibility with Gradle 9.x. Signed-off-by: Martin Gaievski <gaievski@amazon.com>

martin-gaievski force-pushed the fix-bc-fips-jar-hell branch from 16f1feb to 81ef5d0 Compare February 20, 2026 16:09

martin-gaievski had a problem deploying to ml-commons-cicd-env-require-approval February 20, 2026 16:11 — with GitHub Actions Error

martin-gaievski had a problem deploying to ml-commons-cicd-env-require-approval February 20, 2026 16:11 — with GitHub Actions Failure

martin-gaievski had a problem deploying to ml-commons-cicd-env-require-approval February 20, 2026 16:11 — with GitHub Actions Error

Merge branch 'main' into fix-bc-fips-jar-hell

f8a11be

dhrubo-os had a problem deploying to ml-commons-cicd-env-require-approval February 20, 2026 18:52 — with GitHub Actions Error

dhrubo-os had a problem deploying to ml-commons-cicd-env-require-approval February 20, 2026 18:52 — with GitHub Actions Failure

dhrubo-os closed this Feb 24, 2026

Conversation

martin-gaievski commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai bot commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Walkthrough

Changes

Estimated code review effort

❌ Failed checks (1 inconclusive)

Uh oh!

dhrubo-os commented Feb 20, 2026

Uh oh!

martin-gaievski commented Feb 20, 2026

Uh oh!

dhrubo-os commented Feb 20, 2026

Uh oh!

martin-gaievski commented Feb 20, 2026

Uh oh!

mingshl commented Feb 20, 2026

Uh oh!

mingshl commented Feb 20, 2026

Uh oh!

martin-gaievski commented Feb 20, 2026

Uh oh!

mingshl commented Feb 20, 2026

Uh oh!

dhrubo-os commented Feb 20, 2026

Uh oh!

martin-gaievski commented Feb 20, 2026

Uh oh!

cwperks commented Feb 20, 2026

Uh oh!

cwperks commented Feb 20, 2026

Uh oh!

dbwiddis commented Feb 20, 2026

Uh oh!

dhrubo-os commented Feb 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

martin-gaievski commented Feb 19, 2026 •

edited

Loading

coderabbitai bot commented Feb 19, 2026 •

edited

Loading