[BUG] Bulk API doesn't honor the default_pipeline and final_pipeline defined in index template when auto-create index happens

[gaobinlong]

Describe the bug

Update action with upsert in bulk API doesn't honor the default_pipeline and final_pipeline defined in an index template when the auto-created index matches with the template, this should be a bug because if the specified index already exists, the default_pipeline and final_pipeline will be executed.

Related component

Indexing

To Reproduce

1. Create an index template

PUT _index_template/test_index_template
{
  "index_patterns": ["test*"],
  "template": {
    "settings": {
      "index": {
        "final_pipeline": "test_pipeline"
      }
    }
  }
}

2. Create an ingest pipeline

PUT _ingest/pipeline/test_pipeline
{
  "processors": [
    {
      "set": {
        "field": "z",
        "value": 100
      }
    }]
}

3. Bulk update with upsert

POST /_bulk
{"update":{"_index":"test1","_id":"1"}}
{"upsert":{"x":3,"y":5},"script":{"source":"ctx._source.x += 1"}}

4. Check the result

GET test1/_doc/1
{
  "_index": "test1",
  "_id": "1",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "x": 3,
    "y": 5
  }
}

, we can see that the pipeline test_pipeline was not executed.

But if the index test1 already exists before bulk, the result is different:

1. Delete the document 1

DELETE test1/_doc/1

2. Bulk update with upsert

POST /_bulk
{"update":{"_index":"test1","_id":"1"}}
{"upsert":{"x":3,"y":5},"script":{"source":"ctx._source.x += 1"}}

3. Check the result

GET test1/_doc/1
{
  "_index": "test1",
  "_id": "1",
  "_version": 1,
  "_seq_no": 2,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "x": 3,
    "y": 5,
    "z": 100
  }
}

Expected behavior

When the auto-creation happens, the default_pipeline and final_pipeline should be resolved correctly and then be executed.

Additional Details

Host/Environment (please complete the following information):

• OS: [MacOS]
• Version [main branch]

[peternied]

[Triage - attendees 1 2 3 4 5 6 7]
@gaobinlong Thanks for the detailed issue and pull request.

[gaobinlong]

Closed by #12891.