[BUG] Potential Cluster Slowdown/Lags after merging #13748(#14348) #14338(#14391) in 2.15

[peterzhuamazon]

Describe the bug

During 2.15.0 release process, Infra has generated multiple Release Candidates (RC) for testing before launch date.

Between RC3 and RC4, the only two major changes are: opensearch-project/opensearch-build#4785

• [Backport 2.15] Fix unassigned batch allocation (#13748) (#13748) #14348
• [Backport 2.15] Use remote publication flag to decide which custom objects to upload #14391

Once generated RC4, we have observed potential slow downs in Dashboards related plugin integTests.

• RC3 OSD X64 TAR test: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/5950/pipeline
• RC4 OSD X64 TAR test: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/5964/pipeline

These failures are not all the same, but observe very similar symptoms, as elements are not loaded in on time. These behaviors seems as if OS is slowing down causing OSD GUI to load much slower as a side effect.

• securityDashboards: https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/2.15.0/7742/linux/arm64/tar/test-results/5963/integ-test/securityDashboards/with-security/stdout.txt

  1) Home(Get Started) page
       should load Home page properly:
     AssertionError: Timed out retrying after 10000ms: Expected to find content: 'Get started' within the selector: 'h1' but never did.
      at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/integration/plugins/security/get_started_spec.js:170:10)

• reportsDashboards: https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/2.15.0/7742/linux/x64/tar/test-results/5964/integ-test/reportsDashboards/without-security/stdout.txt

  1) Cypress
       "before all" hook for "Visits Reporting homepage":
     AssertionError: Timed out retrying after 10000ms: Expected to find element: `div[data-test-subj="sampleDataSetCardflights"]`, but never found it.

Because this error occurred during a `before all` hook we are skipping the remaining tests in the current suite: `Cypress`
      at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/integration/plugins/reports-dashboards/01-create.spec.js:167:8)

• queryWorkbenchDashboards: https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/2.15.0/7742/linux/x64/tar/test-results/5964/integ-test/queryWorkbenchDashboards/with-security/stdout.txt

  1) Test PPL UI
       "before each" hook for "Confirm results are empty":
     AssertionError: Timed out retrying after 10000ms: Expected to find element: `.euiButton__text[title=PPL]`, but never found it.

Because this error occurred during a `before each` hook we are skipping the remaining tests in the current suite: `Test PPL UI`
      at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/integration/plugins/query-workbench-dashboards/ui.spec.js:207:8)

We are able to reproduce these error locally as well. After switching from RC4 to RC3 artifacts, these errors are gone and test passes.

Please take a look and let us know the results.
Thanks!

Related component

Other

To Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

OSD integTests passes.

Additional Details

Plugins
All

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):
All Dist/Arch

Additional context
Add any other context about the problem here.

[getsaurabh02]

@SwethaGuptha @shwetathareja @soosinha Can we please verify and root cause if these changes merged between RC3 and RC4 can potentially slow down OS, thereby causing delay of OSD objects and leading to assertion failures (timeouts) on multiple fronts in the dashboard plugins.

cc: @dblock @Pallavi-AWS @rramachand21

[zelinh]

Reproduced the with-security test failure for securityDashboards locally.

  7 passing (36s)
  1 failing

  1) Home(Get Started) page
       should load Home page properly:
     AssertionError: Timed out retrying after 10000ms: Expected to find content: 'Get started' within the selector: 'h1' but never did.
      at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/integration/plugins/security/get_started_spec.js:170:10)

[shwetathareja]

@zelinh : The local repro is with RC4? Can you try if it reproduces with RC3 as well?
Also, are you able to profile OSD to confirm if the extra time spent was in OSD itself or due to delay in OS process bootstrapping.

@SwethaGuptha / @shiv0408 to help profile the 2 PRs merged in 2.15 RC4 which are the suspects:
#14348
#14391

[shwetathareja]

The remote cluster state is disabled by default #14391 made some fix related to it and also the bug fix made by #14348 is for fetching unassigned shards details in batch mode which is disabled by default.

The team can compare bootstrap times for OS process with RC3 and RC4

[shiv0408]

@peterzhuamazon I tried bootstrapping the OpenSearch process from RC3 and RC4 binaries. In fact, the time to bootstrap for RC3 (2.671s) was higher than RC4 (2.491s) on my system.