Add verbose pipeline parameter to output each processor's execution details

[junweid62]

Jan 16 revision:

1. Add two new fields: status and error

• status: Represents the execution status of the processor, with possible values such as SUCCESS and FAIL.
• error: Captures the error message or details if the processor encounters an issue during execution. If no error occurs, this field remains empty.

2. When verbose pipeline is enabled, errors in a processor will not interrupt the search but will document the error and continue searching

When the verbose pipeline mode is enabled, if a processor encounters an error during execution, the search process will not be interrupted. Instead, the error will be documented in the processor's execution details (e.g., in ProcessorExecutionDetail) and the remaining search process will proceed as normal.

{
            "processor_name": "rename_field",
            "duration_millis": 0,
            "status": "fail",
            "error": "Document with id 1 is missing field messag123e",
            "input_data": [
                {
                    "_index": "my_index",
                    "_id": "1",
                    "_score": 1.0,
                    "_source": {
                        "message": "This is a public message",
                        "visibility": "public"
                    }
                },
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 1.0,
                    "_source": {
                        "message": "This is a private message",
                        "visibility": "private"
                    }
                }
            ],
            "output_data": null
        },
        {

Description

Related RFC : #16705

This PR introduces enhancements to OpenSearch's search pipeline functionality, focusing on improving the traceability and debugging of search request and response transformations. It addresses the increasing complexity of search pipeline processors by implementing verbose mode support, which provides detailed insights into processor execution.

1. Adds Verbose Mode for Search Pipelines:

   • Introduced the verbose_pipeline parameter to search requests, default to false.
   • Tracks each processor’s input, output, execution time, and status (success/failure).
   • Provides detailed logs of the data flow through request and response processors.

2. Improves Pipeline Debugging:

   • Captures step-by-step data transformations applied by each processor.
   • Includes execution metadata (e.g., timestamps and elapsed time) in search responses for better analysis.

3. Supports All Pipeline Configurations:

   • Works seamlessly with:
     • Default pipelines configured at the index level.
     • Pipelines explicitly specified in the search request.
     • Ad-hoc pipelines defined inline.

4. Test Framework Enhancements:

   • Added tests for verbose mode to ensure correct functionality and compatibility.

Example output with request processor: filter_query response processor: rename_field and sort

{
    "took": 29,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 0.9116078,
        "hits": [
            {
                "_index": "my_index",
                "_id": "2",
                "_score": 0.9116078,
                "_source": {
                    "notification": "This is a public message 2",
                    "visibility": "public"
                }
            }
        ]
    },
    "processor_result": [
        {
            "processor_name": "filter_query",
            "duration_millis": 0,
            "input_data": {
                "query": {
                    "bool": {
                        "must": [
                            {
                                "match": {
                                    "message": {
                                        "query": "This is a public message 1",
                                        "operator": "OR",
                                        "prefix_length": 0,
                                        "max_expansions": 50,
                                        "fuzzy_transpositions": true,
                                        "lenient": false,
                                        "zero_terms_query": "NONE",
                                        "auto_generate_synonyms_phrase_query": true,
                                        "boost": 1.0
                                    }
                                }
                            }
                        ],
                        "adjust_pure_negative": true,
                        "boost": 1.0
                    }
                },
                "search_pipeline": "my_pipeline"
            },
            "output_data": {
                "query": {
                    "bool": {
                        "must": [
                            {
                                "bool": {
                                    "must": [
                                        {
                                            "match": {
                                                "message": {
                                                    "query": "This is a public message 1",
                                                    "operator": "OR",
                                                    "prefix_length": 0,
                                                    "max_expansions": 50,
                                                    "fuzzy_transpositions": true,
                                                    "lenient": false,
                                                    "zero_terms_query": "NONE",
                                                    "auto_generate_synonyms_phrase_query": true,
                                                    "boost": 1.0
                                                }
                                            }
                                        }
                                    ],
                                    "adjust_pure_negative": true,
                                    "boost": 1.0
                                }
                            }
                        ],
                        "filter": [
                            {
                                "term": {
                                    "visibility": {
                                        "value": "public",
                                        "boost": 1.0
                                    }
                                }
                            }
                        ],
                        "adjust_pure_negative": true,
                        "boost": 1.0
                    }
                },
                "search_pipeline": "my_pipeline"
            }
        },
        {
            "processor_name": "rename_field",
            "duration_millis": 0,
            "input_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "message": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ],
            "output_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ]
        },
        {
            "processor_name": "sort",
            "duration_millis": 0,
            "input_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ],
            "output_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ]
        }
    ]
}

Related Issues

Resolves #14745

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for fec0178: SUCCESS

[msfroh]

Thanks, @junweid62! It's looking pretty good.

I just had a few comments, mostly around code that I think you can safely remove.

[owaiskazi19]

LGTM! Thanks for patiently addressing all the comments. Can merge after @msfroh review. Please make sure to perform the BWC process on the PR

[github-actions]

❌ Gradle check result for 386d1f9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 1ccc9c6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for e52dbd5: SUCCESS