[BUG] Intermittent test failure `Tenancy persisted after Logout in SAML`

[peternied]

What is the bug?
Seeing intermittent test failures from Tenancy persisted after Logout in SAML [1] The test failed on a run where no UX elements were modified

● start OpenSearch Dashboards server › Tenancy persisted after Logout in SAML

expect(received).toEqual(expected) // deep equality

Expected: "Global"
Received: "Private"

How can one reproduce the bug?
Unclear how to reproduce the bug, consult the build output for detailed failure information.

What is the expected behavior?
Tests should only fail when an error was introduced in the codebase, or network infrastructure was unavailable.

Do you have any additional context?
Impacted pull request #1189

[1] https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/3387197856/jobs/5627610065

[peternied]

@cwperks Saw another failure on this test, slightly after the area you recently addressed, mind taking a look at it?

Feel free to use from this workflow as a starting point for testing reliability, https://github.com/peternied/security-dashboards-plugin/blob/main/.github/workflows/unit-test-many.yml

FAIL test/jest_integration/saml_auth.test.ts (78.664 s)
...
  ● start OpenSearch Dashboards server › Tenancy persisted after Logout in SAML

    NoSuchElementError: Unable to locate element: //button[@id="user-icon-btn"]Error: Unable to locate element: //button[@id="user-icon-btn"]

      323 |     await driver.findElement(By.xpath(skipWelcomeBtnXPath)).click();
      324 |
    > 325 |     await driver.findElement(By.xpath(userIconBtnXPath)).click();
          |     ^
      326 |
      327 |     await driver.wait(until.elementsLocated(By.xpath(tenantNameLabelXPath)), 10000);
      328 |

      at new WebDriverError (plugins/security-dashboards-plugin/node_modules/selenium-webdriver/lib/error.js:29:5)
      at new NoSuchElementError (plugins/security-dashboards-plugin/node_modules/selenium-webdriver/lib/error.js:212:5)
      at Object.throwDecodedError (plugins/security-dashboards-plugin/node_modules/selenium-webdriver/lib/error.js:524:15)
      at parseHttpResponse (plugins/security-dashboards-plugin/node_modules/selenium-webdriver/lib/http.js:587:13)
      at Executor.execute (plugins/security-dashboards-plugin/node_modules/selenium-webdriver/lib/http.js:515:28)
          at runMicrotasks (<anonymous>)
      at thenableWebDriverProxy.execute (plugins/security-dashboards-plugin/node_modules/selenium-webdriver/lib/webdriver.js:741:17)
      at toWireValue (plugins/security-dashboards-plugin/node_modules/selenium-webdriver/lib/webdriver.js:143:15)
      at plugins/security-dashboards-plugin/node_modules/selenium-webdriver/lib/webdriver.js:193:16
      at forEachKey (plugins/security-dashboards-plugin/node_modules/selenium-webdriver/lib/webdriver.js:187:9)
      at convertKeys (plugins/security-dashboards-plugin/node_modules/selenium-webdriver/lib/webdriver.js:192:3)
      at thenableWebDriverProxy.execute (plugins/security-dashboards-plugin/node_modules/selenium-webdriver/lib/webdriver.js:739:22)
      at Object.<anonymous> (plugins/security-dashboards-plugin/test/jest_integration/saml_auth.test.ts:325:5)

[1] https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/3465602957/jobs/5788534018

[peternied]

@cwperks Looks like you had a PR for this issue that you ended up closing out, anything that would be useful to put on to this issue for how we move forward?

[cwperks]

I wasn't able to get the test working reliably. @RyanL1997 is now setup to run these auth tests locally and has been looking into test reliability.

What I recommend for anyone who picks up the issue is to run the integration tests in parallel using github. You can modify the .github/workflows/integration-test.yml and add a matrix section like so:

jobs:
  tests:
    strategy:
      matrix:
        iteration: [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]

which helps show test reliability if more of the instances of the integration tests running pass. In my experience, the integration tests are failing ~50% of the time with the flaky Tenancy persisted after Logout in SAML test.

[cwperks]

This test has now been ported over to cypress in this PR: #1579

I am closing this issue because this failure hasn't been observed since switching to Cypress.