[BUG] OpenSearch installation failure during discovery-ec2 plugin load

[dreamer-89]

Describe the bug
The OpenSearch installation fails while loading discovery ec2 plugin. This failure is observed while deploying OpenSearch 3.0.0 min-distribution using opensearch cdk. CDK uses https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/core-plugins/discovery-ec2-3.0.0.zip to install the plugin. I suspect this is happening because of jar conflict where jackson-databind is resolved to older version coming from discovery-ec2 plugin which doesn't contain ObjectMapper class.

To Reproduce
Steps to reproduce the behavior:

1. Create min distribution of OpenSearch 3.0.0
2. Synthesize and deploy OpenSearch 3.0.0 min-distribution following steps
3. Post cdk deploy, the cluster is not reachable. The opensearch process does not start because of failure during installation.

Failure stack trace

2023-01-14T02:44:56,831][INFO ][o.o.n.Node               ] [seed] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true,-Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-15050265254909020855, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/home/ec2-user/opensearch, -Dopensearch.path.conf=/home/ec2-user/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2023-01-14T02:44:57,638][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [seed] fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
        at com.amazonaws.util.json.Jackson.<clinit>(Jackson.java:40) ~[?:?]
        at org.opensearch.discovery.ec2.Ec2DiscoveryPlugin.lambda$static$0(Ec2DiscoveryPlugin.java:79) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
        at org.opensearch.discovery.ec2.Ec2DiscoveryPlugin.<clinit>(Ec2DiscoveryPlugin.java:76) ~[?:?]
        at jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method) ~[?:?]
        at jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160) ~[?:?]
        at jdk.internal.reflect.MethodHandleAccessorFactory.ensureClassInitialized(MethodHandleAccessorFactory.java:300) ~[?:?]
        at jdk.internal.reflect.MethodHandleAccessorFactory.newConstructorAccessor(MethodHandleAccessorFactory.java:103) ~[?:?]
        at jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(ReflectionFactory.java:201) ~[?:?]
        at java.lang.reflect.Constructor.acquireConstructorAccessor(Constructor.java:547) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:497) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:484) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:784) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.node.Node.<init>(Node.java:427) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.node.Node.<init>(Node.java:354) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.0.0.jar:3.0.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.0.0.jar:3.0.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-3.0.0.jar:3.0.0]
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ObjectMapper
        at java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:588) ~[?:?]
        at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:521) ~[?:?]
        ... 28 more
fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
        at com.amazonaws.util.json.Jackson.<clinit>(Jackson.java:40)
        at org.opensearch.discovery.ec2.Ec2DiscoveryPlugin.lambda$static$0(Ec2DiscoveryPlugin.java:79)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
        at org.opensearch.discovery.ec2.Ec2DiscoveryPlugin.<clinit>(Ec2DiscoveryPlugin.java:76)
        at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
        at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160)
        at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.ensureClassInitialized(MethodHandleAccessorFactory.java:300)
        at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.newConstructorAccessor(MethodHandleAccessorFactory.java:103)
        at java.base/jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(ReflectionFactory.java:201)
        at java.base/java.lang.reflect.Constructor.acquireConstructorAccessor(Constructor.java:547)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:497)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484)
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:784)
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731)
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533)
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195)
        at org.opensearch.node.Node.<init>(Node.java:427)
        at org.opensearch.node.Node.<init>(Node.java:354)
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.Command.main(Command.java:101)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ObjectMapper
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
        at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        ... 28 more

Expected behavior
The plugin load should not fail

Plugins
discovery-ec2

Host/Environment (please complete the following information):

• OS: linux
• Version 3.0.0

[dreamer-89]

I suspect this is happening because of jar conflict where jackson-databind is resolved to older version coming from discovery-ec2 plugin which doesn't contain ObjectMapper class. @owaiskazi19 @saratvemulapalli : Is it a known issue ?

[dbwiddis]

CC: @ryanbogan -- is this related to #5768?

[dbwiddis]

Looks like #5366 removed the needed dependencies but the #5768 reversion put them back. @dreamer-89 can you check whether your branch is caught up with #5768?

[dreamer-89]

Looks like #5366 removed the needed dependencies but the #5768 reversion put them back. @dreamer-89 can you check whether your branch is caught up with #5768?

Thank you @dbwiddis for looking into it. Yes, I am using latest changes on main. I tried again but fails with exact same error. I noticed post cdk synthesis, command used to install discovery-ec2 plugin uses ci.opensearch.org url rather than typical artifacts url (cdk command below). Not sure how this artifact is generated and maintained. @prudhvigodithi @rishabh6788 : Do you guys know how is this generated ?

"009": {
        "command": "set -ex;cd opensearch; echo \"y\"|sudo -u ec2-user bin/opensearch-plugin install https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/core-plugins/discovery-ec2-3.0.0.zip",
        "cwd": "/home/ec2-user",
        "ignoreErrors": false
       },

Cdk code link

[ryanbogan]

Looking into this to see if the same error occurs with 2.5

[mch2]

I'm not able to repro this with 2.5. I tested without the cdk and manually installed and configured the discovery-ec2 plugin from https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.5.0/6961/linux/x64/tar/builds/opensearch/core-plugins/discovery-ec2-2.5.0.zip which is the plugin associated with the manifest from rc build opensearch-project/opensearch-build#2908 (comment).

[dreamer-89]

Tried this on 2.5 as well where I see jar hell exception. The min-dist I created locally, contains additional jackson-annotations-2.14.1.ja and jackson-databind-2.14.1.jar jars when compared to RC 6961. This probably explains the conflict and points to issue with local setup.

[ec2-user@ip-172-31-51-15 ~]$ diff -qr min-dist/opensearch-2.5.0/ rc-dist/opensearch-2.5.0/ | grep "Only in"
Only in min-dist/opensearch-2.5.0/lib: jackson-annotations-2.14.1.jar
Only in min-dist/opensearch-2.5.0/lib: jackson-databind-2.14.1.jar
Only in rc-dist/opensearch-2.5.0/: manifest.yml
Only in rc-dist/opensearch-2.5.0/modules/ingest-geoip: jackson-annotations-2.14.1.jar
Only in rc-dist/opensearch-2.5.0/modules/ingest-geoip: jackson-databind-2.14.1.jar
Only in rc-dist/opensearch-2.5.0/: opensearch-tar-install.sh
Only in rc-dist/opensearch-2.5.0/: performance-analyzer-rca
...

rc-dist is RC 6961 distribution

min-dist is using latest changes on 2.5. URL: https://d11h2w37tto5s6.cloudfront.net/opensearch-min-2.5.0-linux-x64.tar.gz generated using ./gradlew :distribution:archives:linux-tar:assemble -Dbuild.snapshot=false

[Edit]: Doing a ./gradlew clean before generating the min-dist does the trick and removes the additional jars.

[kaituo]

./gradlew :build --refresh-dependencies solved the issue

[minalsha]

@mch2 @dreamer-89 can we close this issue with no-repro?

[dreamer-89]

@mch2 @dreamer-89 can we close this issue with no-repro?

Thanks @minalsha for checking on this. This issue is already closed. Please let me know if anything else needed here.