[Test] SearchOperationTest is flaky

[peternied]

Description

• There are many tests that use audit log validation of scenarios and it seems either the number of requests is not always consistent
• There are many tests that use reindexing and it looks like the operation isn't being confirmed as completed before the tests continue
• There are many tests that use alias creation / clean up and it seems resolution on these actions is not completing before the tests continue causing issues.

Repo

1. Go to src/integrationTest/java/org/opensearch/security/SearchOperationTest.java
2. Find the @Ignore many tests in this class
3. Delete that attribute
4. Run the tests with ./gradlew integrationTest --tests org.opensearch.security.SearchOperationTest

Exit Criteria

• Find the root cause(s) of the audit log mismatches, all audit log data should expect exact counts, or have a deterministic function to calculate the count based on number of requests, etc.
• Find the root cause of the reindexing issues, add a deterministic process to be sure this has completed, note; this could involve a product fix
• Find the root cause of the aliases issues, add a deterministic process to be sure this has completed, note; this could involve a product fix
• Run the test at least 10 times and see no failures - how to do this

[MaciejMierzwa]

Looking into it

[MaciejMierzwa]

This issue is linked to this one: #2141
Looking into both, the 2141 one is still open but somebody assigned issue to themself.

[MaciejMierzwa]

Here's PR: #3602
Seems like issue is similar to this one #1917.

[stephen-crawford]

This is the PR that resolves this issue: #3602