[wasm] Add support for a random test case orderer, for xunit tests by radical · Pull Request #65628 · dotnet/runtime

radical · 2022-02-20T02:15:25Z

This is enabled by default for wasm with $(XUnitUseRandomizedTestOrderer)=true.

When the library tests run, they print two messages like:

  info: Using random seed for test cases: 700149826
  info: Using random seed for collections: 700149826
  info: Starting:    System.Collections.Immutable.Tests.dll

These seeds are picked randomly every time the tests are run. To run the
tests with a specific seed, use environment variable
XUNIT_RANDOM_ORDER_SEED, which would run the tests in the exact same order.

When running tests, the environment variable is automatically passed to
the wasm app.

This is enabled by default for wasm with `$(XUnitUseRandomizedTestOrderer)=true`. When the library tests run, they print two messages like: ``` info: Using random seed for test cases: 700149826 info: Using random seed for collections: 700149826 info: Starting: System.Collections.Immutable.Tests.dll ``` These seeds are picked randomly every time the tests are run. To run the tests with a specific seed, use environment variable `XUNIT_RANDOM_ORDER_SEED`. When running with tests, this can be used as: `WasmXHarnessMonoArgs="--setenv=XUNIT_RANDOM_ORDER_SEED=<seed>`

ghost · 2022-02-20T02:15:29Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

This is enabled by default for wasm with
$(XUnitUseRandomizedTestOrderer)=true.

When the library tests run, they print two messages like:

  info: Using random seed for test cases: 700149826
  info: Using random seed for collections: 700149826
  info: Starting:    System.Collections.Immutable.Tests.dll

These seeds are picked randomly every time the tests are run. To run the
tests with a specific seed, use environment variable
XUNIT_RANDOM_ORDER_SEED.

When running with tests, this can be used as:

`WasmXHarnessMonoArgs="--setenv=XUNIT_RANDOM_ORDER_SEED=<seed>`

Author:	radical
Assignees:	-
Labels:	`arch-wasm`
Milestone:	-

ghost · 2022-02-20T02:15:30Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

radical · 2022-02-20T02:16:16Z

I'm not sure where to add the documentation. Thoughts?

radical · 2022-02-20T02:22:31Z

/azp run runtime-wasm

azure-pipelines · 2022-02-20T02:22:44Z

Azure Pipelines successfully started running 1 pipeline(s).

vargaz · 2022-02-20T03:22:02Z

Does this make the test order deterministic wrt to a specific seed ?

radical · 2022-02-20T03:22:41Z

Yep

radical · 2022-02-20T06:30:00Z

Random test failure with v8/linux, but seems to be reliably reproducible with XUNIT_RANDOM_ORDER_SEED=1883302047:

[03:29:54] info: Starting:    System.Runtime.Numerics.Tests.dll
[03:29:58] fail: [FAIL] System.Numerics.Tests.cast_toTest.RunDoubleExplicitCastToBigIntegerTests
[03:29:58] info: Assert.Equal() Failure
[03:29:58] info:           ↓ (pos 0)
[03:29:58] info: Expected: -0
[03:29:58] info: Actual:   0
[03:29:58] info:           ↑ (pos 0)
[03:29:58] info:    at System.Numerics.Tests.cast_toTest.VerifyDoubleExplicitCastToBigInteger(Double value)
[03:29:58] info:    at System.Numerics.Tests.cast_toTest.RunDoubleExplicitCastToBigIntegerTests()
[03:29:58] info:    at System.Reflection.RuntimeMethodInfo.InvokeWorker(Object obj, BindingFlags invokeAttr, Span`1 parameters)

radical · 2022-02-20T07:12:42Z

System.Runtime.Loader.Tests fail randomly, but deterministically with XUNIT_RANDOM_ORDER_SEED=2106784294.

[06:25:43] fail: [FAIL] System.Runtime.Loader.Tests.SatelliteAssembliesTests.SatelliteLoadsCorrectly_FromName(alc: "Empty", assemblyName: "System.Runtime.Loader.Tests", culture: "en")
[06:25:43] info: Assert.Same() Failure
[06:25:43] info: Expected: "Default" System.Runtime.Loader.DefaultAssemblyLoadContext #0
[06:25:43] info: Actual:   "Empty" System.Runtime.Loader.AssemblyLoadContext #4
[06:25:43] info:    at System.Runtime.Loader.Tests.SatelliteAssembliesTests.SatelliteLoadsCorrectly_FromName(String alc, String assemblyName, String culture)
[06:25:43] info:    at System.Reflection.RuntimeMethodInfo.InvokeWorker(Object obj, BindingFlags invokeAttr, Span`1 parameters)

Maybe these tests have some unintended dependence on order?

`System.Xml.Tests.TCCloseOutput.*` tests seem to depend on the order of execution. ``` [06:35:14] fail: [FAIL] System.Xml.Tests.TCCloseOutput.CloseOutput_4(utils: XmlWriterUtils { Async = False, WriterType = UTF8Writer }, outputType: "Stream") [06:35:14] info: System.IO.FileNotFoundException : File Not Found: writer.out [06:35:14] info: at XmlCoreTest.Common.FilePathUtil.getStream(String filename) [06:35:14] info: at System.Xml.Tests.TCCloseOutput.CloseOutput_4(XmlWriterUtils utils, String outputType) [06:35:14] info: at System.Reflection.RuntimeMethodInfo.InvokeWorker(Object obj, BindingFlags invokeAttr, Span`1 parameters) ```

radical · 2022-02-20T07:21:46Z

/azp run runtime-wasm

azure-pipelines · 2022-02-20T07:21:58Z

Azure Pipelines successfully started running 1 pipeline(s).

It seems to cause failures like, reproducible with `XUNIT_RANDOM_ORDER_SEED=2106784294`: ``` [06:25:43] fail: [FAIL] System.Runtime.Loader.Tests.SatelliteAssembliesTests.SatelliteLoadsCorrectly_FromName(alc: "Empty", assemblyName: "System.Runtime.Loader.Tests", culture: "en") [06:25:43] info: Assert.Same() Failure [06:25:43] info: Expected: "Default" System.Runtime.Loader.DefaultAssemblyLoadContext #0 [06:25:43] info: Actual: "Empty" System.Runtime.Loader.AssemblyLoadContext #4 [06:25:43] info: at System.Runtime.Loader.Tests.SatelliteAssembliesTests.SatelliteLoadsCorrectly_FromName(String alc, String assemblyName, String culture) [06:25:43] info: at System.Reflection.RuntimeMethodInfo.InvokeWorker(Object obj, BindingFlags invokeAttr, Span`1 parameters) ```

Randomized runs seem to fail with: ``` [03:29:54] info: Starting: System.Runtime.Numerics.Tests.dll [03:29:58] fail: [FAIL] System.Numerics.Tests.cast_toTest.RunDoubleExplicitCastToBigIntegerTests [03:29:58] info: Assert.Equal() Failure [03:29:58] info: ↓ (pos 0) [03:29:58] info: Expected: -0 [03:29:58] info: Actual: 0 [03:29:58] info: ↑ (pos 0) [03:29:58] info: at System.Numerics.Tests.cast_toTest.VerifyDoubleExplicitCastToBigInteger(Double value) [03:29:58] info: at System.Numerics.Tests.cast_toTest.RunDoubleExplicitCastToBigIntegerTests() [03:29:58] info: at System.Reflection.RuntimeMethodInfo.InvokeWorker(Object obj, BindingFlags invokeAttr, Span`1 parameters) ``` Reproducible with `XUNIT_RANDOM_SEED_ORDER=1883302047`.

radical · 2022-02-20T07:29:02Z

I have disabled random ordering for these for now. But I haven't opened any issues, because I'm not entirely sure if it's a bug.

radical · 2022-02-20T07:29:12Z

/azp run runtime-wasm

azure-pipelines · 2022-02-20T07:29:24Z

Azure Pipelines successfully started running 1 pipeline(s).

akoeplinger

LGTM, the random bugs you found look real, I'd file an issue for them and add a comment to the issue to the .csproj's

src/libraries/System.Runtime.Numerics/tests/System.Runtime.Numerics.Tests.csproj

radical · 2022-02-21T18:22:13Z

/azp run runtime-wasm

azure-pipelines · 2022-02-21T18:22:35Z

Azure Pipelines successfully started running 1 pipeline(s).

radical · 2022-02-21T21:57:42Z

The runtime tests failure is #65681, which was hit on a rolling build too. I will try to use this randomizer for runtime tests in a follow up PR.

The debugger tests failure is #64588 .

radical added the arch-wasm WebAssembly architecture label Feb 20, 2022

radical requested a review from akoeplinger February 20, 2022 02:15

ghost assigned radical Feb 20, 2022

radical requested review from lewing, steveisok and vargaz February 20, 2022 02:15

vargaz approved these changes Feb 20, 2022

View reviewed changes

radical added 2 commits February 20, 2022 05:50

Enable test orderer only for libraries tests

9b5fb03

[wasm] Automatically pass XUNIT_RANDOM_ORDER_SEED envvar to wasm apps

3bb53db

radical added 2 commits February 20, 2022 07:23

akoeplinger approved these changes Feb 21, 2022

View reviewed changes

src/libraries/System.Runtime.Numerics/tests/System.Runtime.Numerics.Tests.csproj Outdated Show resolved Hide resolved

This was referenced Feb 21, 2022

[wasm] System.Numerics.Tests.cast_toTest.RunDoubleExplicitCastToBigIntegerTests fails with randomized test order #65671

Open

[wasm] System.Runtime.Loader.Tests fail when run in randomized order #65672

Closed

radical added 2 commits February 21, 2022 13:21

Add issues for the failing tests

c924446

Merge remote-tracking branch 'origin/main' into tests-random-order

153a0d1

radical mentioned this pull request Feb 21, 2022

[wasm] Interpreter assertion in JIT/Methodical/NaN/r4NaNconv_il_r/r4NaNconv_il_r.sh- src/mono/mono/mini/interp/transform.c:5917 #65681

Closed

radical merged commit cacb3bf into dotnet:main Feb 21, 2022

radical deleted the tests-random-order branch February 21, 2022 21:58

ghost locked as resolved and limited conversation to collaborators Mar 24, 2022

Conversation

radical commented Feb 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Feb 20, 2022

Uh oh!

ghost commented Feb 20, 2022

Uh oh!

radical commented Feb 20, 2022

Uh oh!

radical commented Feb 20, 2022

Uh oh!

azure-pipelines bot commented Feb 20, 2022

Uh oh!

vargaz commented Feb 20, 2022

Uh oh!

radical commented Feb 20, 2022

Uh oh!

radical commented Feb 20, 2022

Uh oh!

radical commented Feb 20, 2022

Uh oh!

radical commented Feb 20, 2022

Uh oh!

azure-pipelines bot commented Feb 20, 2022

Uh oh!

radical commented Feb 20, 2022

Uh oh!

radical commented Feb 20, 2022

Uh oh!

azure-pipelines bot commented Feb 20, 2022

Uh oh!

akoeplinger left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

radical commented Feb 21, 2022

Uh oh!

azure-pipelines bot commented Feb 21, 2022

Uh oh!

radical commented Feb 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

radical commented Feb 20, 2022 •

edited

Loading

akoeplinger left a comment •

edited

Loading