[ENHANCEMENT] Integrate with latest backend enhancements #571
Description
There are a few backend enhancements in the Flow Framework plugin, and in core (search pipelines), that will improve different aspects of this plugin:
- [backport 2.x] Add synchronous execution option to workflow provisioning(#990) flow-framework#1009 will allow us to wait for provisioning to complete before pushing state updates. Currently we wait a set amount of time and re-fetch the workflow to try our best to mimic this type of functionality, which can lead to unexpected behavior.
- Add verbose pipeline parameter to output each processor's execution details OpenSearch#16843 will allow us to expose per-processor details of configured search flows, when executed. To start, we can leverage this for fine-grained error handling. In the future, we can leverage this to show more information, like the intermediate data transformed between each processor.
Aside
- similar processor-level error handling is available on the ingest side via verbose param of simulate ingest pipeline API. We can additionally integrate with this, for a consistent UX across ingest & search.