[META] Split and modularize the server monolith

[nknize]

Related modularization discuss issue (more than just JPMS support): #5910
Related JPMS issue: #1588

[WIP] This meta issue tracks the scope for refactoring the monolithic :server module into separable libraries to support JPMS and serverless or cloud native implementation extensions.

Phase 0 - JPMS Support (Eliminate top level split packages, see #1838)

• Initial refactor of o.o.common - refactor classes to :libs:opensearch-common or :libs:opensearch-core where needed
  • o.o.common.io ([Refactor] StreamIO and OpenSearchException foundation to core library #8035, [Refactor] StreamIO from common to core.common namespace in core lib #8157)
• o.o.bootstrap - refactor to o.o.bootstrap in :libs:opensearch-common to new o.o.common.bootstrap package. [JPMS Support] Refactor :libs module bootstrap package to eliminate top level split packages #17117.
• Initial o.apache.lucene - refactor to o.o.core.lucene in :libs:opensearch-core or o.o.lucene in :libs:opensearch-server [JPMS Support] Initial phase-0 Refactor of org.apache.lucene codebase to eliminate top level split packages  #17241
• Remove OneMergeHelper.java (part of o.apache.lucene) and refactor OpenSearchConcurrentMergeScheduler [JPMS Support] Initial phase-0 Refactor of org.apache.lucene codebase to eliminate top level split packages  #17241.
• Remove Operations class from o.apache.lucene and update MinimizationOperations [JPMS Support] Initial phase-0 Refactor of org.apache.lucene codebase to eliminate top level split packages  #17241.
• Update ShuffleForcedMergePolicy removing o.apache.lucene part of :server [JPMS Support] Initial phase-0 Refactor of org.apache.lucene codebase to eliminate top level split packages  #17241.
• o.o.cli - refactor to o.o.core.cli in :libs:opensearch-core or o.o.cli in :libs:opensearch-cli [JPMS Support] Refactor the codebase to eliminate top level split packages #17153
• Cli tests from :server to :libs [JPMS Support] Refactor the codebase to eliminate top level split packages #17153
• :distribution:tools:keystore-cli Issue with org.opensearch.common.settings. Modify to org.opensearch.cli.keystore. [JPMS Support] Refactor the codebase to eliminate top level split packages #17153
• Refactor plugin-cli to org.opensearch.tools.cli.plugin [JPMS Support] Refactor the codebase to eliminate top level split packages #17153
• Refactor upgrade-cli to org.opensearch.tools.cli.upgrade [JPMS Support] Refactor the codebase to eliminate top level split packages #17153
• Refactor the :qa:evil-tests [JPMS Support] Refactor the codebase to eliminate top level split packages #17153
• Spit level problem for :libs:plugin-classholder refactor to org.opensearch.plugin [JPMS Support] Refactor the codebase to eliminate top level split packages #17153
• o.o.client - refactor PR [JPMS Support] Refactoring of org.opensearch.client from :server module #17272.
  • Remove the dependency of o.o.client for rest level client. [JPMS Support] Refactoring of org.opensearch.client from :server module #17272

Phase 1 - Decoupling to support serverless and cloud native implementation extensions

• o.o.env - refactor classes to o.o.core.env in :libs:opensearch-core (dependency of o.o.bootstrap)
• o.o.common.settings - refactor classes to o.o.core.common.settings in :libs:opensearch-core (dependency of o.o.env)
• o.o.cluster.metadata - refactor select classes to a new :libs:opensearch-cluster library
• Remove org.apache.lucene
  • org.apache.lucene.codecs (handle from Star Tree implementation)
  • org.apache.lucene.index (handle from Star Tree implementation)
  • org.apache.lucene.search.grouping.
  • Updated the QueryStringQueryParser from XQueryParser to the more standard QueryParser.

Resulting libraries:

:libs:opensearch-common
:libs:opensearch-core
:libs:opensearch-cluster

relates #1838

[nknize]

/cc @rishabh6788 this work has been largely advancing by me over the last few months and reaching some good breakthrough milestones. The Streamable classes (e.g., StreamInput, StreamOutput, Writeable, NamedWriteable) and OpenSearchException class PR is up to tease out a big part of the tightly coupled classes across the :server module into separable libraries. This unlocks a big portion of :server. The meta issue here seeks to resolve the remaining split packages across server as outlined in the #1838 issue you opened a while ago. A separate goal of #5910 is to ensure as many of the existing extensible classes are in the right library and right package name such that we can thoughtfully expose or restrict extension points to downstream plugin consumers.

If you have some time I'd love for you to jump in and help review the PRs.

[andrross]

Thanks @nknize! I want to start the discussion on expediting the completion of phase 0 (elimination of packages split across jars) AND backporting it the 2.x branch. I think the current state where there is major structural divergence between main and 2.x is undesirable due to causing conflicts on backports. Given that plugins also maintain a branch developing against OpenSearch main, I think they are dealing with the same problem. Also, plugins are not binary-compatible across versions anyway: they must at a minimum update version numbers and re-compile. I think it easier for everyone to complete the structural refactoring quickly and do it across both branches (but also not do it near a release to not add any extra variables as teams are completing integration for that release).

@reta @dblock What do you think?

[reta]

I think it easier for everyone to complete the structural refactoring quickly and do it across both branches (but also not do it near a release to not add any extra variables as teams are completing integration for that release).

@andrross Agree 💯 , the elimination of packages splits would be a huge win towards getting JPMS done, though for 2.x this is going to be 100% breaking change (that goes way beyond @opensearch.internal). Would we selectively keep the structural differences between 2.x and 3.x+ ?

[nknize]

Would we selectively keep the structural differences between 2.x and 3.x+ ?

I think we should look at that on a case by case basis? Let's lean into progress not perfection here so we can iterate quickly.

I also think we shouldn't have FUD around breaking java signatures and package names in minor releases at this juncture (as long as we don't do it too close to a release). The codebase isn't well defined enough for that (hence @opensearch.internal marked on the majority of the codebase). Once we have JPMS and we start defining exports in module-info.java we can use that as the mechanism to collectively invent and simplify on the java surface area we want to expose external to core and guarantee non-breaking across minors on those classes. The only thing we strictly guarantee non breaking at this point is the DSL and index encodings (which is heavily enforced by lucene anyway). #7998 is a nice next step utility for us to quickly and iteratively check what downstream plugins are affected by the change while we're working on the refactor. It doesn't mean we have to open a PR or even fix the breaking change. But it does enable a developer (like me) to quickly check the impact and include that in my PR description w/o reviewers having to do it themselves.

[andrross]

I think we should look at that on a case by case basis?

I agree with a case by case basis. I think the value here of getting rid of a blocker to adopting JPMS is clear, and the "breakages" will be super simple to resolve as it is almost exclusively a matter of fixing imports (and the plugins maintaining a version against main have to resolve these issues anyway), so there's a compelling case for doing the items marked as phase 0 now.

I also agree that it is not practical to be 100% strict on not changing any Java APIs given how tightly coupled the code bases are.

[nknize]

Semi related, I opened an issue for someone to add useful Java annoations we could use to enforce bwc. Not a requirement for this but another mechanism that would be helpful once we start carving out the publicly exposed java classes through module-info.java.