[Pull-based Ingestion] Add ingestion error metrics and make internal queue size configurable

[varunbharadwaj]

Description

1. Add error metrics for poller and processor in pull-based ingestion flows.
2. Make internal queue size used to decouple poller/processor configurable during index creation.

Sample response:

{
  "polling_ingest_stats": {
    "message_processor_stats": {
      "total_processed_count": 1,
      "total_invalid_message_count": 0,
      "total_version_conflicts_count": 0,
      "total_failed_count": 0,
      "total_failures_dropped_count": 0,
      "total_processor_thread_interrupt_count": 0
    },
    "consumer_stats": {
      "total_polled_count": 1,
      "total_consumer_error_count": 0,
      "total_poller_message_failure_count": 0,
      "total_poller_message_dropped_count": 0,
      "lag_in_millis": 1038653008429289138
    }
  }
}

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for ec1e747: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 43f17d2: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 92.03540% with 9 lines in your changes missing coverage. Please review.

Project coverage is 72.52%. Comparing base (a06cf2c) to head (71dd6b9).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...ndices/pollingingest/MessageProcessorRunnable.java	81.48%	5 Missing ⚠️
...g/opensearch/cluster/metadata/IngestionSource.java	78.57%	1 Missing and 2 partials ⚠️
...rch/indices/pollingingest/DefaultStreamPoller.java	94.44%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18088      +/-   ##
============================================
- Coverage     72.57%   72.52%   -0.06%     
- Complexity    67239    67262      +23     
============================================
  Files          5476     5476              
  Lines        310370   310446      +76     
  Branches      45124    45125       +1     
============================================
- Hits         225250   225137     -113     
- Misses        66730    66992     +262     
+ Partials      18390    18317      -73     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ Gradle check result for 08522b5: SUCCESS

[yupeng9]

LGTM

[github-actions]

❌ Gradle check result for 71dd6b9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 71dd6b9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 71dd6b9: SUCCESS