Skip to content
This repository was archived by the owner on Sep 3, 2022. It is now read-only.

Fix flaky test - Attempt 2#91

Merged
f2prateek merged 6 commits intomasterfrom
fix/flaky-tests/2
Jun 27, 2018
Merged

Fix flaky test - Attempt 2#91
f2prateek merged 6 commits intomasterfrom
fix/flaky-tests/2

Conversation

@fathyb
Copy link
Copy Markdown
Contributor

@fathyb fathyb commented Jun 26, 2018

  • Increase the Mocha and Karma timeout to fix timeout errors
  • Fix the cookie tests by using a random key instead, in case other tests are running

On a total of 28 tests :

  • 19 successful tests
  • 1 failure (#59) due to the remove cookie test failing
    • fixed it by using a random key for all tests
  • 6 failures (#66, #57, 48, #47, #46, 44) due to SauceLabs being down
  • 2 failures due to unrelated tests :
    • #55
      1. user #id when cookies and localStorage are disabled should not reset anonymousId if the user id changed to null - IE_11_0_0_(Windows_7_0_0).@segment/analytics.js-core.user
      AssertionError: false == true
    • #35
      1. user () should create anonymous id if missing
        IE_9_0_0_(Windows_7_0_0).@segment/analytics.js-core.user
       false == true

cc @f2prateek
Ref: https://segment.atlassian.net/browse/LIB-354

@fathyb fathyb requested a review from f2prateek June 26, 2018 19:03
@f2prateek
Copy link
Copy Markdown
Contributor

Rather than a random key, which makes the test results a bit harder to reproduce - could we instead clear all the cookies at the end of the test and each test can use a it's own key per test (e.g. a, b)?

@fathyb
Copy link
Copy Markdown
Contributor Author

fathyb commented Jun 26, 2018

@f2prateek agreed for the reproducibility. One note: this will make the tests flaky when run concurrently, cookies are shared across browser instances so it could technically create a race condition (which I did get pretty often when stressing karma to try to trigger the flaky tests) :

  • instance 1 it() set() a cookie
  • instance 2 afterEach() kicks in and remove() the cookie
  • instance 1 it() get() the deleted cookie and the test fails

Removing all cookies before/after each test worsen the problem. I don't think we're affected by this on our CI environments (IIRC SauceLabs isolates each instance) but wanted to point it out.

Pushed a commit with deterministic tests 👍

@codecov-io
Copy link
Copy Markdown

codecov-io commented Jun 26, 2018

Codecov Report

Merging #91 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master      #91   +/-   ##
=======================================
  Coverage   98.58%   98.58%           
=======================================
  Files          11       11           
  Lines         636      636           
=======================================
  Hits          627      627           
  Misses          9        9

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 51ecb87...72e55bc. Read the comment docs.

@f2prateek f2prateek merged commit 39f8e57 into master Jun 27, 2018
@f2prateek f2prateek deleted the fix/flaky-tests/2 branch June 27, 2018 20:49
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants