-
Notifications
You must be signed in to change notification settings - Fork 63
[FEATURE] Add test(s) to compare workflow steps configuration to API specification #651
Description
Is your feature request related to a problem?
Workflow Steps are tightly coupled to the APIs they implement.
The APIs are hosted in external repositories, and there is not signal to our plugin that an API has added features.
OpenSearch APIs have OpenAPI spec files written for them: https://github.com/opensearch-project/opensearch-api-specification
I have opened an issue on ML Commons requesting they do the same: opensearch-project/ml-commons#2297
We should establish automated testing that runs on a periodic basis (Scheduled GitHub Action runs) to test whether we have implemented all API parameters.
What solution would you like?
- Write code to remotely fetch the API specification from wherever it resides. For example, OpenSearch APIs are published here: https://github.com/opensearch-project/opensearch-api-specification/tree/main/spec . It may be acceptable to fetch these file via wget in the GHA script to make it available locally, but if the effort to use a testing client (like we do in integ tests) is low, that would be preferable.
- Parse the YAML into an object. This may be complex: for example, creating an index references a request body with a path that has to be looked up elsewhere.
- Use the WorkflowSteps enum to compare the schema/content/requestBody definitions with the required and optional inputs. The enum may need to be updated to include other information for comparison.
What alternatives have you considered?
Manually following change logs, documentation, and PRs
Do you have any additional context?
It may not be necessary to go deeply into object parsing; just the first level presence or absence of an object key in the request body is likely sufficient.