[BUG] update document with _bulk fails to generate embeddings in inference processors

[will-hwang]

Describe the bug

Bug Description:

embeddings are not generated when documents are updated with /index/_bulk operation, when successfully generated with insert /index/_doc/ operation. Is there any change made in Opensearch that skips embedding generation for bulk operation only?

How to reproduce the error:

Related component

No response

To Reproduce

1. deploy model

POST /_plugins/_ml/models/{model_id}/_deploy

2. configure pipeline

"processors": [
    {
      "text_embedding": {
        "model_id": {model_id},
        "field_map": {
          "text": "passage_embedding"
        }
      }
    }

3. ingest doc

PUT /my-nlp-index/_doc/1
{
  "text": "hello world"
}

4. update doc with _bulk

PUT /my-nlp-index/_bulk
{ "update": { "_index": "my-nlp-index", "_id": "1" } }
{ "doc" : { "text": "bye world" } }

Expected behavior

Embeddings are created for the initial ingest for "text":"hello world", but not updated with bulk operation
{ "doc" : { "text": "bye world" } }

Embeddings should be re-generated for bulk operation by calling text_embedding_processor

Additional Details

Plugins
opensearch-ml, opensearch-knn, opensearch-neural-search

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS] MAC OS
• Version [e.g. 22] Sequoia 15.3

Additional context
Add any other context about the problem here.

[mgodwan]

Which version have you started seeing the issue from? Are you observing this for old opensearch versions as well?

[will-hwang]

yes, i'm seeing this issue in both versions 3.0.0-alpha and 2.19

[mgodwan]

Thanks @will-hwang
Do you see the issue with old OpenSearch versions (e.g. 2.0) as well by any chance? I'm trying to figure out if the behaviour has always been there or is there some change. There have been known consistency differences in _bulk and _update APIs in the past as well.[1][2]

[1] #12883
[2] #16663

[will-hwang]

@mgodwan looks like the issue is there in version 2.0 as well. This issue is regarding _bulk only

[ylwu-amzn]

See same issue on 2.17 , The bulk update doesn't invoke ingest pipeline .

[ylwu-amzn]

This PR #12883 should have fixed this issue, but not sure why this issue still happens.
@gaobinlong , can you help check ?

[gaobinlong]

This PR #12883 should have fixed this issue, but not sure why this issue still happens. @gaobinlong , can you help check ?

Update operation not able to work with ingest pipeline is by design, I think.

#12883 didn't resolve this issue, it targets for upsert operation in bulk update rather than the update operation in bulk update. There’s a similar issue for the case of update operation in bulk update , we had some discussions and gave the conclusion that update operation is not designed to work with ingest pipeline, ingest pipeline was designed to work with index operation in INDEX DOCUMENT API or upsert operation in BULK API. In addition, the default pipeline or final pipeline in an index can be executed when calling UPDATE DOCUMENT API, it's just an accident, the reason is that UPDATE DOCUMENT API converts the update operation to an index operation before executing ingest pipeline, so we have decided to deprecate performing update operation with default pipeline or final pipeline in this PR, check the documentation: https://opensearch.org/docs/latest/api-reference/document-apis/update-document/.

So for this issue, to make sure the default ingest pipeline can be executed, we can consider using INDEX DOCUMENT API(note all other fields except the passage_embedding field need to be included in the request body):

POST /my-nlp-index/_doc/1
{
  "text": "nex text",
 "other field": ""
}

or using UPDATE_BY_QUERY API

// update the value of `passage_embedding` in all documents in once
POST test1/_update_by_query

//  update the value of `passage_embedding` for specific documents by specifying document ids
POST test1/_update_by_query
{
  "query": {
    "ids": {
      "values": [
        "1"
      ]
    }
  },
  "script": {
    "source": "ctx._source.text=params.text",
    "lang": "painless",
    "params": {
      "text": "new text1"
    }
  }
}

[andrross]

Catch All Triage - 1 2