[CAMPAIGN] Remove jackson-databind and jackson-annotations from build.gradle since these dependencies are now coming from core

[ryanbogan]

jackson-databind and jackson-annotations were added as api dependencies in server of opensearch core. Therefore, all plugins now have these as transitive dependencies. Plugins that currently consume these dependencies directly will break while building unless the direct dependencies are removed. If your plugin does not consume jackson-databind or jackson-annotations directly, feel free to close the related issue in your plugin's repository.

Example of solution in security plugin: opensearch-project/security#2325

• Remove jackson-databind and jackson-annotations dependencies now coming from core alerting-dashboards-plugin#423
• Remove jackson-databind and jackson-annotations dependencies now coming from core alerting#699
• Remove jackson-databind and jackson-annotations dependencies now coming from core anomaly-detection-dashboards-plugin#364
• Remove jackson-databind and jackson-annotations dependencies now coming from core anomaly-detection#756
• Remove jackson-databind and jackson-annotations dependencies now coming from core asynchronous-search#214
• Remove jackson-databind and jackson-annotations dependencies now coming from core common-utils#336
• [AUTOCUT] Integration Test Failed for reportsDashboards-3.1.0 dashboards-reporting#583
• Remove jackson-databind and jackson-annotations dependencies now coming from core dashboards-visualizations#145
• Remove jackson-databind and jackson-annotations dependencies now coming from core index-management-dashboards-plugin#462
• Remove jackson-databind and jackson-annotations dependencies now coming from core index-management#632
• Remove jackson-databind and jackson-annotations dependencies now coming from core job-scheduler#280
• Remove jackson-databind and jackson-annotations dependencies now coming from core k-NN#677
• Remove jackson-databind and jackson-annotations dependencies now coming from core notifications#579
• Remove jackson-databind and jackson-annotations dependencies now coming from core performance-analyzer#344
• Remove jackson-databind and jackson-annotations dependencies now coming from core performance-analyzer-rca#257
• Remove jackson-databind and jackson-annotations dependencies now coming from core security-dashboards-plugin#1268
• Remove jackson-databind and jackson-annotations dependencies now coming from core security#2339
• Remove jackson-databind and jackson-annotations dependencies now coming from core sql#1167
• Cypress enhancement and fixes for log explorer dashboards-observability#1347
• Remove jackson-databind and jackson-annotations dependencies now coming from core cross-cluster-replication#665
• Remove jackson-databind and jackson-annotations dependencies now coming from core ml-commons#634
• Remove jackson-databind and jackson-annotations dependencies now coming from core geospatial#196
• [BUG] Remove jackson-databind and jackson-annotations implementations from build.gradle opensearch-java#296

[dbwiddis]

Some projects have other Jackson dependencies that are not one of these two. Most extensions may have, for example, jackson-datatype-jsr310 or jackson-datatype-guava. To maximize compatibility, projects should use the same version as OpenSearch core.

Is there any way for a separate project's build.gradle to reference the version number in OpenSearch?

See here for a potential idea.

[dbwiddis]

Is there any way for a separate project's build.gradle to reference the version number in OpenSearch?

Another option for synchronizing versions is dependencyManagement. This could/would declare the dependencies in OpenSeach core with the same version, but not actually load them. Only if dependent projects requested that dependency would it be loaded, but since it is managed, the correct version would be used.

[penghuo]

1. Could you explain more about the reason? SQL plugin using versions defined in OpenSearch. Is there any issue with this solution?

Plugins that currently consume these dependencies directly will break while building unless the direct dependencies are removed.

2. What do you mean break? The proposed solution does not work if plugin has subproject which does not depend on org.opensearch. e.g. https://github.com/opensearch-project/sql/blob/main/core/build.gradle. core is the subpoject include analyzer, planner, expression which depend on jackson, but not OpenSearch.

[ryanbogan]

1. Could you explain more about the reason? SQL plugin using versions defined in OpenSearch. Is there any issue with this solution?

Plugins that currently consume these dependencies directly will break while building unless the direct dependencies are removed.

2. What do you mean break? The proposed solution does not work if plugin has subproject which does not depend on org.opensearch. e.g. https://github.com/opensearch-project/sql/blob/main/core/build.gradle. core is the subpoject include analyzer, planner, expression which depend on jackson, but not OpenSearch.

@penghuo The security plugin and all native plugins/modules that used jackson-databind and jackson-annotations would fail ./gradlew build after these dependencies were moved to server. If your plugin is still functioning properly, then you can close the issue out, but I wanted to draw attention to a potential error that plugins might be facing.

[penghuo]

@ryanbogan got you. thanks.

[ryanbogan]

Is there any way for a separate project's build.gradle to reference the version number in OpenSearch?

@dbwiddis I believe this is already happening in some of the plugins. In the core repo, we have a version.properties file that contains all of the dependency versions for core.

For example, this is the line that security was using before to directly depend on jackson.
implementation "com.fasterxml.jackson.core:jackson-annotations:${versions.jackson}"

The ${versions.jackson} part of the code references the version.properties version number associated with jackson in that file.

[dbwiddis]

I believe this is already happening in some of the plugins.

Indeed it does look very widespread.

For example, the alerting plugin does what I suggested here:

 force "com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:${versions.jackson}"

Would it make sense to expand this campaign to recommend all repos switch to this versioning to prevent future breakage? I commented in a PR elsewhere for slf4j version that would also benefit from this format.

[ryanbogan]

Would it make sense to expand this campaign to recommend all repos switch to this versioning to prevent future breakage?

@dbwiddis Given the 2.5 deadline for this campaign, I think it would be better to do a separate campaign if we want to make that big of a change.

[penghuo]

@ryanbogan correct me if i am wrong.

• I am not sure why security plugin failed, but based on my test on security 2.x branch, security.zip do not include jackson-databind.zip which means it should not have jarhell issue.
• I also test on SQL main and 2.x branch, not issue found so far.
• As i explained below, currently, OpenSearch exclude resolveableCompileOnly depedencies from runtimeClasspath when zip plugin, if your change breaking this, I suggest to add the exclude logic in PluginBuildPlugin instead.

OpenSearch exclude resolveableCompileOnly depedencies from runtimeClasspath when zip plugin

• In OpenSearch PluginBuildPlugin, resolveableCompileOnly depedencies are removed from runtimeClasspath.

            from project.configurations.runtimeClasspath - project.configurations.getByName(
                    CompileOnlyResolvePlugin.RESOLVEABLE_COMPILE_ONLY_CONFIGURATION_NAME
            )

• e.g. for SQL 2.x, jackson-core:2.4.1, jackson-databind:2.4.1 are removed from class path, and excluded from plugin.zip.

➜  sql git:(2.x) ./gradlew opensearch-sql:dependencies

resolveableCompileOnly
+--- org.opensearch:opensearch:2.5.0-SNAPSHOT
|    +--- org.opensearch:opensearch-core:2.5.0-SNAPSHOT
|    +--- org.opensearch:opensearch-secure-sm:2.5.0-SNAPSHOT
|    +--- org.opensearch:opensearch-x-content:2.5.0-SNAPSHOT
|    |    +--- org.opensearch:opensearch-core:2.5.0-SNAPSHOT
|    |    +--- org.yaml:snakeyaml:1.32
|    |    +--- com.fasterxml.jackson.core:jackson-core:2.14.1
|    |    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-smile:2.14.1
|    |    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.14.1
|    |    \--- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.14.1
|    +--- org.opensearch:opensearch-geo:2.5.0-SNAPSHOT
|    +--- org.apache.lucene:lucene-core:9.4.2
|    +--- org.apache.lucene:lucene-analysis-common:9.4.2
|    +--- org.apache.lucene:lucene-backward-codecs:9.4.2
|    +--- org.apache.lucene:lucene-grouping:9.4.2
|    +--- org.apache.lucene:lucene-highlighter:9.4.2
|    +--- org.apache.lucene:lucene-join:9.4.2
|    +--- org.apache.lucene:lucene-memory:9.4.2
|    +--- org.apache.lucene:lucene-misc:9.4.2
|    +--- org.apache.lucene:lucene-queries:9.4.2
|    +--- org.apache.lucene:lucene-queryparser:9.4.2
|    +--- org.apache.lucene:lucene-sandbox:9.4.2
|    +--- org.apache.lucene:lucene-spatial-extras:9.4.2
|    +--- org.apache.lucene:lucene-spatial3d:9.4.2
|    +--- org.apache.lucene:lucene-suggest:9.4.2
|    +--- org.opensearch:opensearch-cli:2.5.0-SNAPSHOT
|    |    +--- net.sf.jopt-simple:jopt-simple:5.0.4
|    |    \--- org.opensearch:opensearch-core:2.5.0-SNAPSHOT
|    +--- com.carrotsearch:hppc:0.8.1
|    +--- joda-time:joda-time:2.10.12
|    +--- com.tdunning:t-digest:3.2
|    +--- org.hdrhistogram:HdrHistogram:2.1.12
|    +--- org.apache.logging.log4j:log4j-api:2.17.1
|    +--- org.apache.logging.log4j:log4j-jul:2.17.1
|    +--- net.java.dev.jna:jna:5.5.0
|    +--- com.fasterxml.jackson.core:jackson-databind:2.14.1
|    \--- com.fasterxml.jackson.core:jackson-annotations:2.14.1
+--- org.locationtech.spatial4j:spatial4j:0.7
+--- org.locationtech.jts:jts-core:1.15.0
+--- org.apache.logging.log4j:log4j-api:2.17.1
+--- org.apache.logging.log4j:log4j-core:2.17.1
+--- net.java.dev.jna:jna:5.5.0
\--- org.projectlombok:lombok:1.18.22