Skip to content

[FEATURE] Add test(s) to compare workflow steps configuration to API specification #651

@dbwiddis

Description

@dbwiddis

Is your feature request related to a problem?

Workflow Steps are tightly coupled to the APIs they implement.

The APIs are hosted in external repositories, and there is not signal to our plugin that an API has added features.

OpenSearch APIs have OpenAPI spec files written for them: https://github.com/opensearch-project/opensearch-api-specification

I have opened an issue on ML Commons requesting they do the same: opensearch-project/ml-commons#2297

We should establish automated testing that runs on a periodic basis (Scheduled GitHub Action runs) to test whether we have implemented all API parameters.

What solution would you like?

  1. Write code to remotely fetch the API specification from wherever it resides. For example, OpenSearch APIs are published here: https://github.com/opensearch-project/opensearch-api-specification/tree/main/spec . It may be acceptable to fetch these file via wget in the GHA script to make it available locally, but if the effort to use a testing client (like we do in integ tests) is low, that would be preferable.
  2. Parse the YAML into an object. This may be complex: for example, creating an index references a request body with a path that has to be looked up elsewhere.
  3. Use the WorkflowSteps enum to compare the schema/content/requestBody definitions with the required and optional inputs. The enum may need to be updated to include other information for comparison.

What alternatives have you considered?

Manually following change logs, documentation, and PRs

Do you have any additional context?

It may not be necessary to go deeply into object parsing; just the first level presence or absence of an object key in the request body is likely sufficient.

Metadata

Metadata

Assignees

No one assigned

    Labels

    CICI related issuesenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions