[Feature Request] Move transport-grpc from a core plugin to a module

[karenyrx]

Is your feature request related to a problem? Please describe

Context

In 3.0, GRPC APIs were launched as an experimental feature. Per multiple rounds of benchmarking (through bothh OSB and real-world benchmarks), we have seen it delivers superior results over HTTP/JSON, for Bulk API as well as KNN Search (10-200%+ improvement, depending on payload size). As such, we plan to have GRPC/Protobuf be a generally available feature, for select APIs, in the upcoming 3.2 OpenSearch release.

Problem

Currently, external plugins (e.g. k-NN, neural-search, security plugin) are not able to leverage the vast benefits of GRPC/Protobuf. In order to do so, those external plguins will need a way to extend the transport-grpc plugin. This cannot be done today, as the transport-grpc plugin is currently implemented as a plugin in core, which cannot be extended by other plugins. The transport-grpc plugin itself needs to be installed before other external plguins (e.g. k-NN plugin) are built.

Describe the solution you'd like

Proposal

For the upcoming 3.2 OpenSearch release, where GRPC will be GA'ed, we propose to move the transport-grpc plugin from a plugin to a module.

Why keep this plugin core, instead of making it an external plugin

An alternative solution to the problem, was to make the transport-grpc plugin onvert an external plugin (e.g. similar to the job-scheduler plugin), which allows other plugins to likewise extend it. However, this option was ruled out, as this plugin is a core feature parallel to the HTTP/REST API, and would be better maintained by keeping it in core. Notably, keeping it in core makes it easier to:

1. Maintain parity with the HTTP APIs. Currently unit tests exist to check for parity between the HTTP REST -> POJO conversion vs. the GRPC Proto -> POJO conversion, which should ideally fail when users change the HTTP side of the REST -> POJO conversion logic. Additionally the GRPC's fromProto/toProto methods references the HTTP's fromXContent/toXContent methods in its JavaDocs, which provides an additional layer of safety that links the two API conversions together, to minimize chances of divergence.
2. Make GRPC the default protocol in the future. In the long term future, it looks promising that GRPC can become the default protocol, fully replacing HTTP, due to its vast performance benefits, especially for indexing. It would make sense to have a core feature of the OpenSearch server to reside in the core repo.

Why make it a module before GRPC GA (3.2)

Moving it to a module before GA means that users will not be impacted later down the line, when the plugin becomes deprecated and a manual need to be installed is no longer necessary.

Notes about the change

With this change, it will make transport-grpc a plugin that is installed by default with all opensearch builds. However, it will be an opt-in feature. Thus users should not be impacted unless it is enabled explicitly via the GRPC settings.

Related component

Plugins

Describe alternatives you've considered

To allow the GRPC plugin to be extensible by other plugins, we also considered:

1. Making transport-grpc an external plugin. Similar to job-scheduler, making it an external plugin can similarly solve the prolbme of allowing other plugins to depend on it. However, this was ruled out because the GRPC protocol is likely to become a long-term feature in OpenSearch servers, with its promising performance impact, and thus would make sense to reside in proximity with the HTTP/REST API and server implementations.
2. Changing opensearch-build to install all core plugins by default. This is ruled out as installing all optional core plugins by default is not necessary for all users, and it would be a change with a large blast radius as it impacts all existing core plugins.

Additional context

Core PR to make the transport-grpc query types extensible to other plugins: #18516

[peterzhuamazon]

My only concern is whether enable it as a module would have performance issues on the cluster.
Previously as a core plugin there is no default installation. But as a module it would be enabled all the time.

Thanks.

[andrross]

I support this proposal. gRPC and protobuf are proven, industry-standard tools that I suspect we'll continue to integrate more deeply in the future. The downsides of this proposal are pretty minimum and potential benefits are large. It's also a simple solution to the concrete problem that many plugins will want to add support for gRPC APIs and will need this module to be present on the classpath.

@msfroh @yupeng9 can you add your thoughts as well?

@peterzhuamazon As Karen mentioned, the gRPC module would not be enabled unless the settings to turn it on and specify the port to listen on are provided. It should have no impact on performance. The only difference is that the classes would be loaded by the JVM, which solves the problem of plugins like k-NN needing the classes to be there to add gRPC support to vector search.

[finnegancarroll]

Adding some similar discussion and meta related to this from security repo where gRPC is being taken as a test dependency.

An alternative solution to the problem, was to make the transport-grpc plugin onvert an external plugin (e.g. similar to the job-scheduler plugin), which allows other plugins to likewise extend it. However, this option was ruled out, as this plugin is a core feature parallel to the HTTP/REST API, and would be better maintained by keeping it in core.

Some additional points against moving gRPC to an external plugin:

• To ensure backwards compatibility we would still need to ensure gRPC is available for install with the min distribution. @prudhvigodithi had some idea on how to do this but moving to a module avoids the issue altogether.
• As a third party plugin users would need to install gRPC as a dependency of KNN plugin even if they never plan on utilizing this transport. Users may question why they need to install this additional resource which they do not utilize when upgrading KNN versions.