[BUG] Plugin installation on OpenSearch 2.0.0 runs into jar hell

[cliu123]

Describe the bug
Installing security plugin on OpenSearch 2.0.0 fails on jar hell. The following is the error:

-> Installing file:///Users/cgliu/git/cliu123_OpenSearch_Security/build/distributions/opensearch-security-2.0.0.0.zip
-> Downloading file:///Users/cgliu/git/cliu123_OpenSearch_Security/build/distributions/opensearch-security-2.0.0.0.zip
[=================================================] 100%
-> Failed installing file:///Users/cgliu/git/cliu123_OpenSearch_Security/build/distributions/opensearch-security-2.0.0.0.zip
-> Rolling back file:///Users/cgliu/git/cliu123_OpenSearch_Security/build/distributions/opensearch-security-2.0.0.0.zip
-> Rolled back file:///Users/cgliu/git/cliu123_OpenSearch_Security/build/distributions/opensearch-security-2.0.0.0.zip
Exception in thread "main" java.lang.IllegalStateException: failed to load plugin opensearch-security due to jar hell
	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:675)
	at org.opensearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:857)
	at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:825)
	at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:870)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:275)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:249)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)
Caused by: java.lang.IllegalStateException: jar hell!
class: com.fasterxml.jackson.dataformat.yaml.JacksonYAMLParseException
jar1: /Users/cgliu/git/opensearch-2.0.0-SNAPSHOT/lib/jackson-dataformat-yaml-2.13.2.jar
jar2: /Users/cgliu/git/opensearch-2.0.0-SNAPSHOT/plugins/.installing-15161370711895821889/jackson-dataformat-yaml-2.13.2.jar
	at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:317)
	at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:212)
	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:673)

Expected behavior
Plugin is installed smoothly.

[peternied]

[Work around] Can we remove the library from the security plugin since its included transitively in OpenSearch?

[cliu123]

The above workaround doesn't look working. @Rishikesh1159 Would you please help out?
Cc: @CEHENKLE

[Rishikesh1159]

Sure @cliu123 I am working on it. I will provide you an update once I find anything.

[Rishikesh1159]

Due to some breaking changes in opensearch, security plugin build is failing. So, I cannot reproduce the issue. @cliu123 is working on fixeing build failures in security plugin. Once it is fixed I can resume working on this.

[peternied]

@Rishikesh1159

I believe the repro includes using a build where @cliu123 has fixed the breaking changes you can get a copy from here https://github.com/cliu123/security/tree/bump_opensearch_dependency_to_2.0.0

The issue itself can be seen in the plugin install workflow link, here is the PR

Inside the workflow logs there is an error, indicating the service did not start, and there is another task called Get Docker Logs that includes debug information such as:

docker logs ops
  shell: /usr/bin/bash -e {0}
  env:
    JAVA_HOME_11.0.14_x64: /opt/hostedtoolcache/jdk/11.0.14/x64
    JAVA_HOME: /opt/hostedtoolcache/jdk/11.0.14/x64
    JAVA_HOME_11_0_14_X64: /opt/hostedtoolcache/jdk/11.0.14/x64
Error: Exception in thread "main" java.lang.IllegalStateException: failed to load plugin opensearch-security due to jar hell
-> Installing file:///docker-host/security-plugin.zip
	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:678)
-> Downloading file:///docker-host/security-plugin.zip
	at org.opensearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:857)
	at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:825)
	at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:870)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:275)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:249)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:1[3](https://github.com/opensearch-project/security/runs/5726583125?check_suite_focus=true#step:13:3)8)
	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:10[4](https://github.com/opensearch-project/security/runs/5726583125?check_suite_focus=true#step:13:4))
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)
Caused by: java.lang.IllegalStateException: jar hell!
class: com.fasterxml.jackson.dataformat.cbor.CBORConstants
jar1: /opensearch/plugins/.installing-777472399439[5](https://github.com/opensearch-project/security/runs/5726583125?check_suite_focus=true#step:13:5)884235/jackson-dataformat-cbor-2.13.2.jar
jar2: /opensearch/lib/jackson-dataformat-cbor-2.12.[6](https://github.com/opensearch-project/security/runs/5726583125?check_suite_focus=true#step:13:6).jar
	at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:31[7](https://github.com/opensearch-project/security/runs/5726583125?check_suite_focus=true#step:13:7))
	at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:212)
	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:676)
	... [11](https://github.com/opensearch-project/security/runs/5726583125?check_suite_focus=true#step:13:11) more

If you would like to reproduce locally you can by use the workflow file

This is blocking the 2.0.0 release from building, lets please focus attention on this issue and reach out if we are having trouble making progress

[Rishikesh1159]

@mch2 will be working on this issue as top priority. Thanks @peternied @cliu123

[mch2]

taking a look

[mch2]

Hi @cliu123,
I just built your fork with the jackson-databind dependency removed from your gradle scripts - this line and successfully installed & ran the tarball with the demo config with ./bin/opensearch.

I had to use the tarball pulled by the failed workflow, it looks like there are security build failures against the latest version of the 2.0 branch?

What is the desired dependency resolution here? I would suggest unless another version is required security inherits the jackson version from core vs declaring its own.

Also this is failing on the validate step after the server is running. I would expect an error during plugin installation for jarHell vs at runtime, is that ignored/obfuscated in the workflow?

I don't think there is any change required on OpenSearch here. Plugins should either exclude the inherited jar or inherit the jar and not declare its own version. OS also bumped to version 2.13.2.2 today if there was a requirement for 2.13.2 in security.