[Meta] [Segment Replication] Run all integration tests with segment replication enabled

[dreamer-89]

Leading up to GA #5147 of segment replication, we need to do a round of sanity testing with existing integration tests. This is to ensure segment replication is compatible with existing features.

This issue is to track effort on running these tests, identifying root cause of failures. The fix for individual failures can be tracked in separate issues. These tests should be RUN against 2.x branch as we targetting this exercise for SegRep GA (going in 2.7). We can start with server module and run all integration tests internalClusterTest. Once these failures are resolved, we can run remaining integration tests.

General steps:

When running these tests, we need to turn Segment Replication as default replication strategy (this is to overcome fixing replication type for each integration test). Below are the changes needed for this:
1 - update INDEX_REPLICATION_TYPE_SETTING in IndexMetadata to return ReplicationType.SEGMENT.
2 - Update ReplicaitonType.java to return SEGMENT if a NPE is caught.
3 - Update FeatureFlags to set REPLICATION_TYPE_SETTING’s default to true.
4 - Some tests use mockEngineFactory, change OpenSearchIntegTestCase#addMockInternalEngine to return false from this method by default.
5 - Some methods use MockEngineFactory as a node level plugin - override this by also updating the class

@Override
    public Engine newReadWriteEngine(EngineConfig config) {
        if (config.isReadOnlyReplica()) {
            return new NRTReplicationEngine(config);
        }
        return new MockInternalEngine(config, wrapper);
    }

• [BUG] [Segment Replication] Primary mode assertion failure on source #6740
• [BUG] [Segment Replication] Handle closed indices on segrep source  #6724
• IndexWriter AlreadyClosedException on segment replication failure #6776
• [BUG] [Segment Replication] testRelocateWhileContinuouslyIndexingAndWaitingForRefresh flaky test failure #6778
• Handle IndexShard.getLatestReplicationCheckpoint for closed indices #6828
• [BUG] [Segment Replication] NO-OP recovery not attempted #7161
• [BUG] [Segment Replication] Resync failures results in removal of in-sync allocation id  #7163
• Segment replication cancellation of in-sync allocation ids in ReplicationTracker #7213
• Coming up with a plan to resolve integ tests failures without manually changing every single integ test in opensearch codebase.
• [Segment Replication] Create new Segment Replication client to support running Integ Tests with Segment Replication enabled #8925
• Segment Replication - Support realtime reads with GET requests #8536
• [Segment Replication] Verify All Integ Tests are Passing with Segment Replication Enabled #8927