[Pull-based Ingestion] Support segment replication for pull-based ingestion

[varunbharadwaj]

Description

This PR is a follow up for pull-based-ingestion to support segment replication with remote store. The primary shard will ingest from the streaming source and replica shards will rely on segment replication.

This PR refactors IngestionEngine to inherit from InternalEngine to support replication, recovery and avoid duplicate code. Some of the changes required to support segRep and peer recovery are enhancing IngestionEngine to include required listeners, support working with NRTReplicationEngine, tracking latest index commits, prevent snapshotted index deletion, among many others. These changes are already available in InternalEngine, and can be reused by IngestionEngine after this change.

Integration tests are added to validate end-to-end pull-based ingestion with segment replication, peer recover, replica promotion and remote store.

Related Issues

Resolves #16929

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 7a682ef: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for fae7e91: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 22464ed: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[andrross]

Looks like there are some build failures here. FYI you should be able to run ./gradlew precommit locally to find these before pushing your commit.

[github-actions]

✅ Gradle check result for f56f90f: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 83.01887% with 9 lines in your changes missing coverage. Please review.

Project coverage is 72.53%. Comparing base (0ffed5e) to head (c5d7445).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...opensearch/index/translog/NoOpTranslogManager.java	73.33%	3 Missing and 1 partial ⚠️
...a/org/opensearch/index/engine/IngestionEngine.java	86.36%	1 Missing and 2 partials ⚠️
.../indices/pollingingest/IngestionEngineFactory.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17359      +/-   ##
============================================
+ Coverage     72.42%   72.53%   +0.11%     
- Complexity    65611    65675      +64     
============================================
  Files          5304     5304              
  Lines        304743   304464     -279     
  Branches      44189    44145      -44     
============================================
+ Hits         220701   220840     +139     
+ Misses        65888    65495     -393     
+ Partials      18154    18129      -25     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❌ Gradle check result for bb213a7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for c5d7445: SUCCESS

[yupeng9]

LGTM

[msfroh]

This looks pretty reasonable to me as a first step. In the long run, I think I'd like to extract an abstract base class for both InternalEngine and IngestionEngine, but I think that belongs in another PR, since this one's already big enough.

Thanks, @varunbharadwaj!

[varunbharadwaj]

This looks pretty reasonable to me as a first step. In the long run, I think I'd like to extract an abstract base class for both InternalEngine and IngestionEngine, but I think that belongs in another PR, since this one's already big enough.

Thanks, @varunbharadwaj!

Thanks for reviewing. Yeah, will create a follow up PR for the refactoring.

[mch2]

minor comment, lgtm!

[gbbafna]

@varunbharadwaj , @yupeng9 : Are we only supporting only segrep with pull based ingestion ?

[varunbharadwaj]

@varunbharadwaj , @yupeng9 : Are we only supporting only segrep with pull based ingestion ?

Yes, the plan is to start with segrep. We will probably explore multi-node ingestion in the future, but is still in discussion.