Add `dotnet test` integration tests for all Apple platforms by jonathanpeppers · Pull Request #25320 · dotnet/macios

jonathanpeppers · 2026-05-04T21:58:14Z

This PR adds end-to-end dotnet test integration tests for all Apple platforms (iOS, tvOS, Mac Catalyst, macOS) using the MTP (Microsoft Testing Platform) test templates.

Changes

tests/dotnet/UnitTests/DotNetTestTest.cs: New parameterized test that creates a project from dotnet new <template>, replaces the generated tests with a single passing test, and runs dotnet test.
tests/common/DotNet.cs: Added test verb support to DotNet.Execute.

The test reads the MLAUNCH_PATH environment variable to override the mlaunch binary for mobile platforms (iOS/tvOS), enabling testing with the MTP-enabled mlaunch build.

Screenshot

Create DotNetTestTest that runs 'dotnet test' on projects created from the iostest, tvostest, maccatalysttest, and macostest templates. This verifies the MTP (Microsoft Testing Platform) pipeline works end-to-end, including the --server/--dotnet-test-pipe argument passthrough via RunArguments. Also add 'test' as a supported verb in DotNet.Execute. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

macOS and Mac Catalyst don't use mlaunch (they use 'open' via Desktop.targets), so MTP support requires a different approach. Comment out those test cases for now. Also skip the /v:diag and /consoleloggerparameters workaround args when the verb is 'test' to prevent them leaking through to the MTP test runner via RunArguments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Update mlaunch NuGet package to include MTP (Microsoft Testing Platform) support for iOS and tvOS simulator test execution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The dotnet test MTP flow uses MSBuild API internally for ComputeRunArguments but doesn't load the workload SDK resolver. With --no-build, this means the iOS SDK can't be resolved at all. Without --no-build, dotnet test does a full build+run cycle where the workload resolver is properly initialized. Also use Execution.RunAsync directly instead of DotNet.Execute to avoid /p: and /bl: args that dotnet test doesn't forward. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

vs-mobiletools-engineering-service2 · 2026-05-14T20:16:40Z

✅ [PR Build #0509368] Build passed (Build packages) ✅

Pipeline on Agent
Hash: 0509368e7452114986bdc9e6894f1b063b433127 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-14T20:16:45Z

✅ [CI Build #0509368] Build passed (Build packages) ✅

Pipeline on Agent
Hash: 0509368e7452114986bdc9e6894f1b063b433127 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-14T20:16:46Z

✅ [PR Build #0509368] Build passed (Build packages) ✅

Pipeline on Agent
Hash: 0509368e7452114986bdc9e6894f1b063b433127 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-14T20:20:54Z

✅ [CI Build #0509368] Build passed (Detect API changes) ✅

Pipeline on Agent
Hash: 0509368e7452114986bdc9e6894f1b063b433127 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-14T20:34:00Z

✅ API diff for current PR / commit

NET (empty diffs)

iOS: vsdrops gist ( No breaking changes )
tvOS: vsdrops gist ( No breaking changes )
MacCatalyst: vsdrops gist ( No breaking changes )
macOS: vsdrops gist ( No breaking changes )

✅ API diff vs stable

NET (empty diffs)

iOS: vsdrops gist ( No breaking changes )
tvOS: vsdrops gist ( No breaking changes )
MacCatalyst: vsdrops gist ( No breaking changes )
macOS: vsdrops gist ( No breaking changes )

ℹ️ Generator diff

Generator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes)

Pipeline on Agent
Hash: 0509368e7452114986bdc9e6894f1b063b433127 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-14T20:41:47Z

✅ [CI Build #0509368] Build passed (Build macOS tests) ✅

Pipeline on Agent
Hash: 0509368e7452114986bdc9e6894f1b063b433127 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-14T22:08:58Z

🔥 [CI Build #0509368] Test results 🔥

Test results

❌ Tests failed on VSTS: test results

0 tests crashed, 22 tests failed, 159 tests passed.

Failures

❌ dotnettests tests (iOS)

1 tests failed, 0 tests passed.

Failed tests

DotNet tests: Failed (Execution failed with exit code 1)
- Xamarin.Tests.DotNetTestTest.DotNetTest(iOS,"iostest"): 'dotnet test' failed with exit code 1.
  Output:
  Running the ComputeRunArguments target to discover run commands failed for this...

Html Report (VSDrops) Download

❌ dotnettests tests (tvOS)

1 tests failed, 0 tests passed.

Failed tests

DotNet tests: Failed (Execution failed with exit code 1)
- Xamarin.Tests.DotNetTestTest.DotNetTest(TVOS,"tvostest"): 'dotnet test' failed with exit code 1.
  Output:
  Running the ComputeRunArguments target to discover run commands failed for this...

Html Report (VSDrops) Download

❌ monotouch tests (macOS)

1 tests failed, 17 tests passed.

Failed tests

monotouch-test/macOS/Debug: Failed (Test run failed.
Tests run: 3708 Passed: 3601 Inconclusive: 4 Failed: 1 Ignored: 106)

Html Report (VSDrops) Download

❌ monotouch tests (tvOS)

19 tests failed, 0 tests passed.

Failed tests

monotouch-test/tvOS - simulator/Debug: LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (CoreCLR): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (CoreCLR, ARM64): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (CoreCLR, x64): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (CoreCLR, Universal): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (LinkSdk): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (static registrar): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (all optimizations): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (ARM64): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (NativeAOT, ARM64): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (trimmable static registrar, NativeAOT, ARM64): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (managed static registrar): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (trimmable static registrar): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (managed static registrar, all optimizations): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (trimmable static registrar, all optimizations): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (NativeAOT, x64): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (trimmable static registrar, NativeAOT, x64): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (interpreter): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (interpreter): LaunchTimedOut

Html Report (VSDrops) Download

Successes

✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 2 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 4 tests passed. Html Report (VSDrops) Download
✅ generator: All 5 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 4 tests passed. Html Report (VSDrops) Download
✅ introspection: All 9 tests passed. Html Report (VSDrops) Download
✅ linker: All 44 tests passed. Html Report (VSDrops) Download
✅ monotouch (iOS): All 19 tests passed. Html Report (VSDrops) Download
✅ monotouch (MacCatalyst): All 25 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ sharpie: All 1 tests passed. Html Report (VSDrops) Download
✅ windows: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. Html Report (VSDrops) Download
✅ xtro: All 1 tests passed. Html Report (VSDrops) Download

macOS tests

✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sequoia (15): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Tahoe (26): All 5 tests passed. Html Report (VSDrops) Download

Linux Build Verification

✅ Linux build succeeded

Pipeline on Agent
Hash: 0509368e7452114986bdc9e6894f1b063b433127 [PR build]

jonathanpeppers changed the base branch from main to net11.0 May 4, 2026 21:58

This comment has been minimized.

Sign in to view

rolfbjarne mentioned this pull request May 11, 2026

[CI Postmortem] Flaky: TestNSUrlSessionEphemeralHandlerCookies on iOS/MacCatalyst #25384

Closed

jonathanpeppers and others added 2 commits May 12, 2026 12:33

jonathanpeppers force-pushed the jonathanpeppers/add-mlaunch-mtp-tests branch 2 times, most recently from 32ce330 to 5c37575 Compare May 12, 2026 17:34

Bump Microsoft.Tools.Mlaunch to 1.1.115

4c9418b

Update mlaunch NuGet package to include MTP (Microsoft Testing Platform) support for iOS and tvOS simulator test execution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jonathanpeppers force-pushed the jonathanpeppers/add-mlaunch-mtp-tests branch from 5c37575 to 4c9418b Compare May 12, 2026 17:35

This comment has been minimized.

Sign in to view

jonathanpeppers and others added 2 commits May 14, 2026 14:53

Remove unnecessary env var overrides from dotnet test call

0509368

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Conversation

jonathanpeppers commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Screenshot

Related

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

vs-mobiletools-engineering-service2 commented May 14, 2026

✅ [PR Build #0509368] Build passed (Build packages) ✅

Uh oh!

vs-mobiletools-engineering-service2 commented May 14, 2026

✅ [CI Build #0509368] Build passed (Build packages) ✅

Uh oh!

vs-mobiletools-engineering-service2 commented May 14, 2026

✅ [PR Build #0509368] Build passed (Build packages) ✅

Uh oh!

This comment has been minimized.

vs-mobiletools-engineering-service2 commented May 14, 2026

✅ [CI Build #0509368] Build passed (Detect API changes) ✅

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

vs-mobiletools-engineering-service2 commented May 14, 2026

✅ API diff for current PR / commit

✅ API diff vs stable

ℹ️ Generator diff

Uh oh!

This comment has been minimized.

vs-mobiletools-engineering-service2 commented May 14, 2026

✅ [CI Build #0509368] Build passed (Build macOS tests) ✅

Uh oh!

This comment has been minimized.

vs-mobiletools-engineering-service2 commented May 14, 2026

🔥 [CI Build #0509368] Test results 🔥

Test results

Failures

❌ dotnettests tests (iOS)

Failed tests

❌ dotnettests tests (tvOS)

Failed tests

❌ monotouch tests (macOS)

Failed tests

jonathanpeppers commented May 4, 2026 •

edited

Loading