[FEATURE] Security support for extensions

[saratvemulapalli]

Coming from:
[Meta] Making OpenSearch Extensible opensearch-project/OpenSearch#2447
[Meta] Plugin Sandboxing (a.k.a extensions) opensearch-project/OpenSearch#1422

Problem

As we are building extensions, security is an integral part of everything we deliver. We see a bunch of problems for plugin security and would like to solve them from Day 1 for extensions.

• Plugins implement security constructs by themselves, evaluating permissions and gating resources manually.
  Anomaly Detection: https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/ad/util/ParseUtils.java#L465
  Alerting: https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/kotlin/org/opensearch/alerting/transport/SecureTransportAction.kt#L78
• All plugins have super admin permissions and can play with any information they'd like on the cluster.
  Anomaly Detection: https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/ad/transport/IndexAnomalyDetectorTransportAction.java#L95
  Alerting: https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportExecuteMonitorAction.kt#L68
• Intercommunication with Plugins multiplied with permissions, is manually managed and caused bunch of security loop holes.
  Alerting-Anomaly Detection integration: https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/kotlin/org/opensearch/alerting/InputService.kt#L149
• Plugins define the permissions they need with Java Security Manager but there is no gating mechanism to deny/validate permissions.
  Anomaly Detection: https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/plugin-metadata/plugin-security.policy
  Alerting: https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/plugin-metadata/plugin-security.policy
• Security for plugins is driven by thread context assuming they run in the same JVM. Extensions run within, outside of JVM and existing security interfaces do not work.
• Extensions are built on SDK which is not coupled with JSM (as its deprecated) and we don't have a way to restrict extensions with host level resources (like filesystem, network etc)
• Security plugin has support to authenticate/authorize on transport APIs but do not act on any Rest APIs (which is infact the reason why all plugins implemented security at transport layer). Rest APIs are integral part of OpenSearch eco-system and cannot open all APIs to all extensions.
• Extensions access control for accessing resources within OpenSearch is not gated, which brings risk for OpenSearch cluster running 3rd party extensions.
• Plugins are validated during install time as they are only bootstrapped once during a lifetime of a node. They are validated by bin/opensearch-plugin installation tool. Extensions can be installed/modified/removed on the fly and needs identity validation before it joins the cluster.

Requirements

• [Extensions] Extensions should be able to call a common API which evaluates permissions and gates resources.
• [Extensions] Extensions by default should not have any permissions, and are explicitly granted based on defined policies.
• [Security into Core and Extensions] All API calls between extensions and OpenSearch are access controlled based on extension permission policies.
• [Extensions and SDK] Extensions SDK should be able to restrict host level access(file system, network etc) based on security policy.
• [Security into Core] Add support to gate Rest APIs for OpenSearch
• [Extensions and SDK] Add support to validate the identity of an extension during runtime. This helps in identifying the extension and claiming if it really is. This is an important requirement for extensions since extensions can be installed/modified/removed on the fly and needs identity validation before it joins the cluster.
• [Extensions and SDK] Compliance: Add support for extensions to do audit logging.

Refactored these requirements from June 2022 to explain the problems above.

To build a MVP for extensions we are working on building Anomaly Detection plugin and help run it as an extension.
Our end goal is to have:

• Load third party extensions securely
• Protect the cluster resources when extensions are installed
• Access control for resources created by extensions (Including APIs, application resources etc)
• Access control OpenSearch resources from extensions.

We can chat over this and get more feedback, but these are the resources we could think of to enable secure extensions.

For a start as we are looking at AD: opensearch-project/OpenSearch#3635
We need help to figure out how AD extension can work with security (adding extension support for common-utils):

• AD uses User information for resource access control
• AD uses InjectUser to inject user permissions for background jobs.

[peternied]

This security plugin is not in a good position to support a feature that is built into OpenSearch's codebase, looking at how we are considering building extensions security related features should be built into the core of OpenSearch instead of through its plugin systems. I am going to close this issue.

However, having a hand in developing security for extensions is a discussion we would like to influence - I will create an issue in core and we can have a basis of discussion there.

For the specific topics mentioned:

AD uses User information for resource access control

There is an existing user object in the common-utils codebase, this object can't be injected/extracted as is currently done since it requires a shared thread context model. There is work in opensearch-project/opensearch-sdk-java#14 that I believe is the foundation for resolving this problem. Is there a place where API contracts of extensions are under discussion or should I start proposing the api request/responses?

AD uses InjectUser to inject user permissions for background jobs.

Persisting identity is a very difficult problem to solve. I think we need a new series of APIs that allow for out of band/non-interactive request to create actions on behalf of a user. This will require first coming to terms with the identity management system in OpenSearch, but also the access mechanisms (eg, do we follow a pattern from AWSs assume role APIs, or is there another convention we should be adopting

[peternied]

All in all - I think we have an opportunity to design a better security posture for OpenSearch and its extensibility models. Asks like this highlight the need for more thought and alignment generation in this space, thanks for calling out these concerns.

[peternied]

Note; Leaving this open until those called out areas are documented in the SDK

[jimishs]

Hi Sarat, Thanks for starting this thread. Could you pls elaborate on specific needs across the following themes that you highlighted (comments inline)

• Load third party extensions securely
  [Jimish] What does it mean to load extensions securely? Are you suggesting that extensions run inside a sandbox that is provided by a general security framework? Something like a separate container perhaps?

• Protect the cluster resources when extensions are installed
  [Jimish] Can you elaborate. I believe in earlier discussions you mentioned host level security. Are we thinking of resource limiting controls for extensions (eg: max CPU, memory etc)? What are other controls you are looking for?

• Access control for resources created by extensions (Including APIs, application resources etc)
  [Jimish] Can you expand on what resources will be created by extensions? Is the reference to APIs about security plugin APIs to access data or something else? What application resources would you like to have access control configuration for?

• Access control OpenSearch resources from extensions
  [Jimish] What resources are you referring to? Also what granularity of control do you need for such resources?

In addition im also curious to get your thoughts on whether extensions will be able to create their own metadata/storage files, or is that something that could be part of a security SDK?

Thanks

[saratvemulapalli]

Thanks @jimishs for these questions. This was written 6months ago, I have updated the issue with more granular details of the problems.

[Jimish] What does it mean to load extensions securely? Are you suggesting that extensions run inside a sandbox that is provided by a general security framework? Something like a separate container perhaps?

"Plugins are validated during install time as they are only bootstrapped once during a lifetime of a node. They are validated by bin/opensearch-plugin installation tool. But, Extensions can be installed/modified/removed on the fly and needs identity validation before it joins the cluster."
We need to add a mechanism to support verification of identity of extensions which they really claim to be.

[Jimish] Can you elaborate. I believe in earlier discussions you mentioned host level security. Are we thinking of resource limiting controls for extensions (eg: max CPU, memory etc)? What are other controls you are looking for?

"Extensions SDK should be able to restrict host level access(file system, network, execution etc) based on security policy."
That is right, plugins do have this via JSM using security policy, and we need similar support for extensions.

[Jimish] Can you expand on what resources will be created by extensions? Is the reference to APIs about security plugin APIs to access data or something else? What application resources would you like to have access control configuration for?

Sure, Anomaly Detection extension will create detectors, Alerting will create monitors. These resources have to be gated by user identity.

[Jimish] What resources are you referring to? Also what granularity of control do you need for such resources?

"[Extensions] Extensions by default should not have any permissions, and are explicitly granted based on defined policies.
[Security into Core and Extensions] All API calls between extensions and OpenSearch are access controlled based on extension permission policies."
All APIs(Rest and transport) extensions make (via SDK) to OpenSearch should be evaluated based on extension permissions.
Not all APIs will be accessible by all extensions.

I hope that helps. Let me know what other questions do you have.

On a side note: @peternied @dbwiddis @dblock @nknize I've updated the issue with sets of problems/requirements, would love your feedback and thoughts.