[BUG] Plugin installation fails due to jar hell

[stephen-crawford]

What is the bug?
When executing the plugin install action on main, the process fails due to Exception in thread "main" java.lang.IllegalStateException: failed to load plugin opensearch-security due to jar hell

This occurs when attempting to install the security plugin into OpenSearch core and results in the following complete error log:

-> Rolled back file:D:\a\security\security\opensearch-security.zip
	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:681)
	at org.opensearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:862)
	at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:830)
	at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:875)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:275)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:249)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.plugins.PluginCli.main(PluginCli.java:66)
Caused by: java.lang.IllegalStateException: jar hell!
class: com.fasterxml.jackson.databind.AbstractTypeResolver
jar1: D:\a\security\security\opensearch-3.0.0-SNAPSHOT\lib\jackson-databind-2.14.1.jar
jar2: D:\a\security\security\opensearch-3.0.0-SNAPSHOT\plugins\.installing-17316426811803655475\jackson-databind-2.14.1.jar
	at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:314)
	at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:213)
	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:679)
	... 11 more

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Go to the main branch of the security repo
2. Make a trivial change to a forked version of the repo.
3. Wait for the plugin install workflow to fail
4. See error

What is the expected behavior?
This is a new error after the weekend.

Do you have any additional context?
The action worked as expected on Friday.

[peternied]

A couple of things we can try:

• Can we remove the direct dependency on jackson-databind and pick it up translatively, maybe this resolves the issue?
• Let's see about filing an issue in OpenSearch core, this is a breaking change, but I didn't see any clear callouts, maybe they can help us figure out next steps?
• Maybe if either of those don't work, we could hardcode the version that we are downloading from the ci system, eg instead of url: https://artifacts.opensearch.org/snapshots/core/opensearch/${{ inputs.opensearch-version }}-SNAPSHOT/opensearch-min-${{ inputs.opensearch-version }}-SNAPSHOT-windows-x64-latest.zip -> Something else?

[cwperks]

Core has a direct dependency on jackson-databind now? Looking into this more, a lot of components are going to need to react to this change.

[cwperks]

Looks like it was added for extensions: opensearch-project/OpenSearch#5366

We can remove our dependency on jackson-databind since it comes from upstream now.

[cwperks]

Created a PR to remove jackson-databind and jackson-annotations as direct dependencies: #2325

[stephen-crawford]

Thank you @cwperks

[RyanL1997]

I also created this issue in core: opensearch-project/OpenSearch#5526

[reta]

Seems like the jackson-databind and jackson-annotations have to become runtime only dependency, fe https://github.com/opensearch-project/sql/pull/1159/files#diff-520f178549fabd7e74f5b1fedb7275a1434a2ac153fc49e0212455ceeef38e37R110 (they are part of core now)

[RyanL1997]

According to the above fixes, we can close this issue now. Thank you all for the help!