[Rmote Store] Run integ tests with segrep and remote store enabled

[Bukhtawar]

Is your feature request related to a problem? Please describe

We run integ tests without segrep or remote store enabled defaulting to docrep, which might create regression and mostly tech debt during release activities where we need to depend on manual sign-offs.

Describe the solution you'd like

Given the rate of changes we make we need to enable running randomised testing on decrep/segrep/remote store modes to improve code fidelity.

Related component

Storage

Describe alternatives you've considered

No response

Additional context

No response

[andrross]

We run integ tests without segrep or remote store enabled defaulting to docrep

We have added parameterized tests for replication-focused tests to run both docrep and segrep. We've also added randomization to randomly select replication strategy for other tests not focused on replication. @Rishikesh1159 can you provide more detail about where we're at now and the testing you've done with enabling segrep-with-remote-store?

[Rishikesh1159]

Running Integ Tests with Segment Replication and Remote Store.

Goal :

Run integ tests with segrep and remote store enabled.

Problem :

We have two problems to acheive our goal:

1. Waiting after refresh : After a refresh, we need to wait some time for replicas to catch up with primary shards. The tests will fail with assertion on replica shard before it has caught up with primary.

2. Running all integ Tests : Every Integ Test is different, so there is no single framework for us to follow and enable segrep for all Integ Test at once. There are certain tests that test the behaviour not supported by segrep in this case those tests will fail if we run them with segment replication enabled.

Plan to Overcome Problems :

1. Waiting after refresh :

• To overcome this problem we have introduced a new method called refreshAndWaitForReplication .
• This refreshAndWaitForReplication will be used after an indexing operation, so with this method we refresh and then wait for replicas to catch up. This way only any assertions further made in test will be done only after replica has caught up.
• Basically we will need to replace the refresh() call in IntegTest with refreshAndWaitForReplication .

2. Running all integ Tests :

Every Integ Test that we want to run with segrep/remote-store can be divided into below 4 categories:

A. Critical Tests related to segrep/remote-store :

• The tests which are critical to and directly tests behaviour of segrep/remote-store can be put in this category.
• These are tests which are already exists before and we want run them with both segrep and docrep or remote-store/non remote-store.
• That means we need to run these critical test twice, once with segrep and another with docrep.
• We have used parameterized testing approach for these critical tests to run a single test multiple times with different parameters.
• Example PR: [Segment Replication] Add Segment Replication Specific Integration Tests #11773.

B. Non Crititical Tests :

• The tests which don’t directly test segrep/remote-store behaviour can be put in this category.
• Although these tests don’t test segrep behaviour directly but still can be related to a different feature/operation that is failing because of segrep/remote-store.
• To identify these edge cases and proper working of segrep/remote-store we need to run these non-critical tests with segment replication as well.
• As these are not critical tests we can use randomization here, by randomly picking a replication strategy for test, this way we would cover running tests with both segrep, docrep without running the test multiple times as we did above for critical tests. This increases the test coverage for segrep/remote-store.
• Example PR: [Segment Replication] Add random replication strategy #12297

C. Segrep/Remote-Store Specific Integ Tests :

• These are tests which previously didn’t exists with docrep. These tests are newly created specifically to test certain behaviors of segrep/remote-store.
• As they are tests Specific to segrep/remote-store they are run only once with segrep/remote-store.
• Example Integ Test : SegmentReplicationIT.java

D. Integ Tests not compatible with segrep/remote-store :

• There are certain Integ tests related to a specific feature/operation that is currently not directly supported by segrep/remote-store. These kind of tests can be put in this category.
• As these Integ tests are not compatible segrep/remote-store we completely avoid/ignore running them with segrep/remote-store enabled.
• Example : ShrinkIndexIT

How to a run an existing Integ test with segrep/remote-store :

Step 1 :

• Identify in which of the categories mentioned above does the test comes under.
• If it is a critical test, we need to use parameterization approach. This way the test always runs multiple times with different parameters of segrep/remote-store and docrep. Example: [Segment Replication] Add Segment Replication Specific Integration Tests #11773.
• If it is a non-critical test, we need to use randomization. Example : [Segment Replication] Add random replication strategy #12297.

Step 2 :

• In the Integ Test replace the usage of refresh() with refreshAndWaitForReplication() .
• If the test uses indexRandom(forceRefresh:true) then indexRandom() internally uses refreshAndWaitForReplication() so we don’t need to worry about refresh in this case.
• For all other remaining type of invocations of refresh like : client().prepareIndex("test").setId("1").setSource("field","value1").setRefreshPolicy(RefreshPolicy.IMMEDIATE).get();we manually we need to call waitForReplication() after a write operation. This way we wait for replica shards to catch up with primary after every write operation.

Step 3 :

• Sometimes few tests in a Integ Test suite make assertions on features/operations which are not compatible with segrep/remote-store. In those cases, we conditionally need to ignore the assertion.
• Example here in IndexStatsIT, the test testSimpleStats() was asserting on count of write operations on replica shards. But with segment replication these write operation never happens on replica shards, so the test will fail with assertion. To cover such cases we add conditional check before making few assertions.

Step 4 :

• After following all above steps, we have to make sure we run the Integ Test suite for few hundred iterations locally.
• This step is needed to make sure that we are not adding any new flakiness to existing tests.

Current Status :

To reach the goal of Running Integ tests with segrep and remote store enabled. We need completion of following steps:

Step 1 :

• Add new refreshAndWaitForReplication() and waitForReplication() methods to support tests waiting for replica shards catching up after a write operation.
• Implemented in PR: [Segment Replication] Add Segment Replication Specific Integration Tests #11773
• Status: COMPLETED.

Step 2 :

• Support Parameterization for running critical tests with Segment Replication enabled.
• Implemented in PR: [Segment Replication] Add Segment Replication Specific Integration Tests #11773
• Status: COMPLETED.

Step 3 :

• Support Randomization for running non-critical tests with segment replication enabled randomly.
• Implemented in PR: [Segment Replication] Add random replication strategy #12297
• Status: COMPLETED.

Step 4 :

• Support Parameterization for running critical tests with Remote Store enabled.
• Implemented in PR: [Remote Store] Add support for Running Parameterized Remote Store Enabled Integration Tests #12487
• Status: COMPLETED.

Step 5 :

• Support Randomization for running non-critical tests with remote store enabled randomly.
• Implemented in PR: [Remote Store] Add support for randomizing Remote Store enabled testing. #12488
• Status: COMPLETED.

Step 6 :

• Make of list of existing integ tests which should be critical, non-critical and not compatible with segrep/remote-store. Once the list is made we need to modify each Integ Test to support them run them with segrep/remote-store enabled.
• Implemented in PR: N/A
• Status: NOT STARTED.

[Rishikesh1159]

We have added parameterized tests for replication-focused tests to run both docrep and segrep. We've also added randomization to randomly select replication strategy for other tests not focused on replication. @Rishikesh1159 can you provide more detail about where we're at now and the testing you've done with enabling segrep-with-remote-store?

Sure @andrross . @Bukhtawar here is current status:

Current Status :
To reach the goal of Running Integ tests with segrep and remote store enabled. We need completion of following steps:

Step 1 :

• Add new refreshAndWaitForReplication() and waitForReplication() methods to support tests waiting for replica shards catching up after a write operation.
• Implemented in PR: [Segment Replication] Add Segment Replication Specific Integration Tests #11773
• Status: COMPLETED.

Step 2 :

• Support Parameterization for running critical tests with Segment Replication enabled.
• Implemented in PR: [Segment Replication] Add Segment Replication Specific Integration Tests #11773
• Status: COMPLETED.

Step 3 :

• Support Randomization for running non-critical tests with segment replication enabled randomly.
• Implemented in PR: [Segment Replication] Add random replication strategy #12297
• Status: COMPLETED.

Step 4 :

• Support Parameterization for running critical tests with Remote Store enabled.
• Implemented in PR: [Remote Store] Add support for Running Parameterized Remote Store Enabled Integration Tests #12487
• Status: COMPLETED.

Step 5 :

• Support Randomization for running non-critical tests with remote store enabled randomly.
• Implemented in PR: [Remote Store] Add support for randomizing Remote Store enabled testing. #12488
• Status: COMPLETED.

Step 6 :

• Make of list of existing integ tests which should be critical, non-critical and not compatible with segrep/remote-store. Once the list is made we need to modify each Integ Test to support them run them with segrep/remote-store enabled.
• Implemented in PR: N/A
• Status: NOT STARTED.

As mentioned above we are currently working on steps 4,5. Once those are completed the process and framework will be completed and only step 6 will be remaining.