[Pull-based Ingestion] Support transient failure retries and create mode

[varunbharadwaj]

Description

1. Support upto 2 retries to handle transient failures before dropping a message, when using drop error policy.
2. Support create mode along with currently supported index and delete. This will allow users to set document ID maintained by them while also improving performance by inserting the document in create/append mode.

Related Issues

Builds on top of #17427 and #17822

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 46a2662: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 627b12a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 627b12a: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 78.12500% with 7 lines in your changes missing coverage. Please review.

Project coverage is 72.54%. Comparing base (e40b47d) to head (a9856a8).
Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
...ndices/pollingingest/MessageProcessorRunnable.java	80.00%	2 Missing and 4 partials ⚠️
...a/org/opensearch/index/engine/IngestionEngine.java	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18250      +/-   ##
============================================
- Coverage     72.56%   72.54%   -0.02%     
+ Complexity    67380    67345      -35     
============================================
  Files          5488     5488              
  Lines        310994   311005      +11     
  Branches      45212    45213       +1     
============================================
- Hits         225665   225634      -31     
- Misses        66906    67011     +105     
+ Partials      18423    18360      -63     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ Gradle check result for 005b4a7: SUCCESS

[github-actions]

❌ Gradle check result for a9856a8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[yupeng9]

LGTM

[github-actions]

❕ Gradle check result for a9856a8: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.