[Feature Request] Add configurability to run ingest pipelines during document update operations

[q-andy]

Is your feature request related to a problem? Please describe

Currently, behavior for ingest pipelines with single/bulk update/upsert operations is inconsistent. Some combinations of parameters will trigger ingest pipelines, but some will not. Many users want functionality to bulk update documents and have their updates pass through ingest pipelines and are confused when this is not the case. This is currently supported by single update API but NOT in the bulk update API, and #17679 will make single update consistent with bulk update, meaning neither will trigger ingest pipelines. single/bulk consistency has been put on hold until 4.0.0.

However, even if this is by design, many users have asked for the bulk update ingest pipeline use case. For example, if an ingest pipeline has a text embedding processor set up to generate embeddings on passage text,zwe want the embeddings to be regenerated via pipeline processor when the passage text field is updated, but bulk update operation does not support that. Currently the only workaround is to use the docAsUpsert flag with a partial doc, and it is unintuitive that enabling such a flag would have an effect on how ingest pipelines are applied.

We want a straightforward and clear way for users to specify to invoke ingest pipeline for their update operations in both single update and bulk cases.

Issues opened due to this inconsistency/confusion:

• [BUG] update document with _bulk fails to generate embeddings in inference processors #17494
• [FEATURE] Add support with the Update API neural-search#213
• [BUG] Ingest pipeline bulk update issue #16663
• [BUG] Bulk upsert does not behave like a single Upsert, with an ingestion pipeline #10864
• Using script upsert with _bulk API won't triggered #2607
• [Cleanup] TransportUpdateAction should extend TransportSingleItemBulkWriteAction #16980

Describe the solution you'd like

We want a method of specifying that we want documents to pass through ingest pipelines update and bulk update scenarios. Some ideas:

• Having a pipeline-level configuration option (preferred)
• Having an additional parameter in the bulk API interface to force ingest pipeline invocation (similar to docAsUpsert workaround but more explicit)
• Having a new type of pipeline process that converts update requests to index requests (may not be possible)

The functionality to trigger pipelines from update requests already exists in the current single update operation that is being removed in #17679 and this logic can be reused.

Related component

Indexing

Describe alternatives you've considered

No response

Additional context

No response

[q-andy]

You can assign this to me, I can look into this further

[msfroh]

This is currently supported by single update API but NOT in the bulk update API, and #17679 will make single update consistent with bulk update, meaning neither will trigger ingest pipelines.

Technically, once #17679 goes in, we'll still run ingest pipelines, but only with the fields sent in the update request, as far as I know. Of course, if the ingest pipeline has processors that depend on other fields, it won't see them.

I like the idea of making it possible (in an opt-in fashion) to apply the update logic to turn the update request into an index request (with all the fields) before the ingest pipeline runs. We'll probably need to invoke that logic from TransportBulkAction before the ingest pipelines get called.

Thanks for looking into this @q-andy!

[q-andy]

Discussed with @owaiskazi19, #18277 contains a large amount of the work required to support this by introducing the childSlot to decouple the one-to-one mapping for bulk request operations to document pipeline executions and allows for ingest pipeline resolution, execution, and graceful error/success propagation.

In terms of code changes, in the future we can enable pipelines on partial docs in all update requests by flipping this line to resolve all pipelines instead of just system ingest pipelines. This would enable partial doc update on all bulk requests, similar to the existing doc_as_upsert=true workaround, but with compatibility with upsert requests.

If we go this route, we also have to call out in the documentation that this is a) a breaking change that would need to be communicated to users clearly and b) could possible cause failures depending on the user's pipeline processor setup. For example, if using script processors that rely on specific doc fields, and you pass in a partial doc without those fields, the user would have to configure the script processor would need additional conditional validations to proceed gracefully when missing fields, or else the processor would NPE and fail the entire operation

Also, partial doc updates would still be inconsistent with single update. To keep parity with single document update, there is additional work needed to do a shard fetch to merge the partial docs with the full documents before executing pipelines, which is the current functionality of single update.