[Pull-based Ingestion] Add support for dynamically updating ingestion error handling strategy with minor fixes

[varunbharadwaj]

Description

1. This PR is a follow up for [Pull-based Ingestion] Add error handling strategy to pull-based ingestion #17427 to add support for dynamically updating ingestion error strategy using update_settings API.
2. Message processor will indefinitely retry failed messages after a wait time, if a BLOCK error strategy is used. Updating to DROP strategy will skip the failed messages.
3. Additionally, the PR includes minor fixes - fixes initial global checkpoint in p2p segRep mode which is validated by flows such as CloseIndex API and handle race condition on flush before poller is initialized completely.

This PR forms the base on which subsequent PRs will build on for adding pause/resume APIs.

Related Issues

Resolves part of #17442. Subsequent PRs will add pause/resume APIs.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 92b576e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 9f94093: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 70.73171% with 12 lines in your changes missing coverage. Please review.

Project coverage is 72.43%. Comparing base (e306d51) to head (4399432).
Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
...a/org/opensearch/index/engine/IngestionEngine.java	54.54%	4 Missing and 1 partial ⚠️
...ndices/pollingingest/MessageProcessorRunnable.java	72.22%	3 Missing and 2 partials ⚠️
...in/java/org/opensearch/index/shard/IndexShard.java	50.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17565      +/-   ##
============================================
+ Coverage     72.27%   72.43%   +0.15%     
- Complexity    65611    65696      +85     
============================================
  Files          5311     5311              
  Lines        304942   304975      +33     
  Branches      44225    44229       +4     
============================================
+ Hits         220407   220912     +505     
+ Misses        66448    65918     -530     
- Partials      18087    18145      +58     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andrross]

Pause message processor when an error is encountered on BLOCK error strategy.

Would it be better to keep retrying indefinitely at some fixed rate? It can be hard to determine if errors are transient or not, and in this case if you enter the paused state because of a transient failure you'll just be stuck until something intervenes, right?

[varunbharadwaj]

Pause message processor when an error is encountered on BLOCK error strategy.

Would it be better to keep retrying indefinitely at some fixed rate? It can be hard to determine if errors are transient or not, and in this case if you enter the paused state because of a transient failure you'll just be stuck until something intervenes, right?

Good point, as discussed today, updated to indefinitely retry and skip only after user switches to a DROP policy.

[github-actions]

❌ Gradle check result for e6e34ed: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 5c0788c: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 589c235: SUCCESS

[github-actions]

❌ Gradle check result for 12bbe28: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 4399432: UNSTABLE

• TEST FAILURES:

      2 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.classMethod
      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testSnapshotWithStuckNode

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[yupeng9]

LGTM