Basic View operations in OpenSearch

[peternied]

Description

Implement the basic view feature in OpenSearch that is persistent. This issue contains an initial proposal built in a proof of concept, but is not hard requirements for the feature. It is highly encouraged to make the minimal implementation and collect data from the community before investing in more advanced features such as required parameters, row level security based on identity, search pipelines, audit logging, etc...

[Proposal 1] View data

Views create a mapping to the resources that hold information to be searched over in a consistent manner. This abstraction allows for indirection with the backing indices, so they might be changed without callers being impacted. This can also be used to simplify the security model - searches over views do not require permissions to the backing indices only permissions to the view itself.

classDiagram
    class View {
        +String name
        +String description
        +long createdAt
        +long modifiedAt
        +List<Target> targets
        +toXContent(XContentBuilder, Params) XContentBuilder
        +writeTo(StreamOutput) void
    }
    class Target {
        +String indexPattern
        +toXContent(XContentBuilder, Params) XContentBuilder
        +writeTo(StreamOutput) void
    }
    class StreamOutput
    class XContentBuilder

    View  -- Target : contains
    View -- StreamOutput : writes to
    View -- XContentBuilder : outputs to
    Target -- StreamOutput : writes to
    Target -- XContentBuilder : outputs to

Loading

[Proposal 1] View persistence

Views are long lived objects in OpenSearch, all operations on them should be fully committed before responding to the caller. Views are intentionally created for user scenarios following a similar creation cadence to indices.

Committed implies that the updates are synchronized across all nodes in a cluster. The Cluster Metadata Store is already available and allows for acknowledging that changes have been applied to all nodes. While this data could be stored in a new purpose built index, index data replication has delays and ensuring synchronization is non-trivial to implement as is seen in the Security plugins [1].

• [1] [FEATURE] When internal security configuration is updated, its should be confirmed to be a new value #3275

sequenceDiagram
    participant Client
    participant HTTP_Request as ActionHandler
    participant Cluster_Metadata as Cluster Metadata Store
    participant Data_Store as Indices

    Client->>HTTP_Request: View List/Get/Update/Create/Delete<BR>/views or /views/{view_id}
    HTTP_Request->>Cluster_Metadata: Query Views
    alt Update/Create/Delete
        Cluster_Metadata->>Cluster_Metadata: Refresh Cluster
    end
    Cluster_Metadata-->>HTTP_Request: Return
    HTTP_Request-->>Client: Return

    Client->>HTTP_Request: Search View<br>/views/{view_id}/search
    HTTP_Request->>Cluster_Metadata: Query Views
    Cluster_Metadata-->>HTTP_Request: Return
    HTTP_Request->>HTTP_Request: Rewrite Search Request
    HTTP_Request->>HTTP_Request: Validate Search Request
    HTTP_Request->>Data_Store: Search indices
    Data_Store-->>HTTP_Request: Return
    HTTP_Request-->>Client: Return

Loading

Exit Criteria

• Add/update views in a persistant way
• Be able to execute a search over a view
• Get buy in from OpenSearch maintainers for experimental feature release approval in 2.x

[stephen-crawford]

[Triage] Hi @peternied thank you for filing this issue. We can mark this triaged since it is a description of your proposed feature.

[peternied]

• Draft PR is out Projected Views OpenSearch#11957