Extend gate to support all test types (UI, device, unit, XAML) by kubaflo · Pull Request #34657 · dotnet/maui

kubaflo · 2026-03-25T21:09:27Z

The gate phase (verify-tests-fail-without-fix) was hardcoded to only support UI tests. This extends it to auto-detect and run all test types.

Changes

Detect-TestsInDiff.ps1 — shared script that analyzes PR files and classifies tests by type
verify-tests-fail.ps1 — routes to correct runner per test type, iterates over all detected tests
pr-gate.md — uses detection script instead of grep for TestCases only
EstablishBrokenBaseline.ps1 — excludes test infrastructure files from fix file detection
RunTests.ps1 — unified test runner entry point

Test Types Supported

Type	Detected From	Runner
UITest	TestCases.Shared.Tests/	BuildAndRunHostApp.ps1
DeviceTest	DeviceTests/	Run-DeviceTests.ps1 (Category= filter)
UnitTest	*.UnitTests/	dotnet test
XamlUnitTest	Xaml.UnitTests/	dotnet test

Verified

Gate passed on local Share device test: without fix=FAIL ✅, with fix=PASS ✅

- verify-tests-fail.ps1: auto-detect test type from changed files, route to correct runner (BuildAndRunHostApp, Run-DeviceTests, dotnet test) - Detect-TestsInDiff.ps1: shared detection script for PR test analysis - pr-gate.md: use detection script instead of grep for TestCases only - EstablishBrokenBaseline.ps1: exclude TestUtils/DeviceTests.Runners from fix files - RunTests.ps1: unified test runner entry point - Platform is now mandatory for all test types Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-03-25T21:09:39Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 34657

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 34657"

kubaflo · 2026-03-25T21:12:53Z

Changes already on copilot-kubaflo branch which the pipeline uses

Copilot

Pull request overview

Extends the PR “gate” workflow (verify-tests-fail-without-fix) to auto-detect and run multiple test types (UI, device, unit, XAML) by introducing a shared diff/PR test detection script and updating gate documentation accordingly.

Changes:

Added a shared test detection script (Detect-TestsInDiff.ps1) that classifies changed tests and provides runner/filter metadata.
Updated verify-tests-fail.ps1 to iterate over all detected tests and dispatch to the correct runner per test type.
Added a unified local runner entry point (RunTests.ps1) and updated gate docs (pr-gate.md, skill docs) to use the new detection approach.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1	Routes execution by detected test type and runs all detected tests across both “without fix” and “with fix” runs.
.github/skills/verify-tests-fail-without-fix/SKILL.md	Documents multi-test-type support and updated usage guidance.
.github/scripts/shared/Detect-TestsInDiff.ps1	New shared detector that classifies tests from PR files/git diff and returns structured entries.
.github/scripts/RunTests.ps1	New unified local test runner wrapper for unit/device/UI/integration tests.
.github/scripts/EstablishBrokenBaseline.ps1	Expands “test infrastructure” patterns to avoid misclassifying infra changes as fix files.
.github/pr-review/pr-gate.md	Updates gate instructions to use the shared detection script and reflect multi-type gating.

Copilot · 2026-03-25T21:14:13Z

.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1

+# Maps file path patterns to test types
+$script:TestTypePatterns = @(
+    @{ Pattern = "TestCases\.(Shared\.Tests|HostApp|Android\.Tests|iOS\.Tests|Mac\.Tests|WinUI\.Tests)"; Type = "UITest" }
+    @{ Pattern = "Xaml\.UnitTests/"; Type = "XamlUnitTest" }
+    @{ Pattern = "DeviceTests/"; Type = "DeviceTest" }
+    @{ Pattern = "(?<!\w)UnitTests/|Graphics\.Tests/"; Type = "UnitTest" }
+)
+
+# Maps test types to their project paths (relative to repo root)
+$script:UnitTestProjectMap = @{
+    "Controls.Core.UnitTests"          = "src/Controls/tests/Core.UnitTests/Controls.Core.UnitTests.csproj"
+    "Controls.Xaml.UnitTests"          = "src/Controls/tests/Xaml.UnitTests/Controls.Xaml.UnitTests.csproj"
+    "Controls.BindingSourceGen.UnitTests" = "src/Controls/tests/BindingSourceGen.UnitTests/Controls.BindingSourceGen.UnitTests.csproj"
+    "SourceGen.UnitTests"              = "src/Controls/tests/SourceGen.UnitTests/SourceGen.UnitTests.csproj"
+    "Core.UnitTests"                   = "src/Core/tests/UnitTests/Core.UnitTests.csproj"
+    "Essentials.UnitTests"             = "src/Essentials/test/UnitTests/Essentials.UnitTests.csproj"
+    "Graphics.Tests"                   = "src/Graphics/tests/Graphics.Tests/Graphics.Tests.csproj"
+    "Resizetizer.UnitTests"            = "src/SingleProject/Resizetizer/test/UnitTests/Resizetizer.UnitTests.csproj"
+    "Compatibility.Core.UnitTests"     = "src/Compatibility/Core/tests/Compatibility.UnitTests/Compatibility.Core.UnitTests.csproj"
+    "Essentials.AI.UnitTests"          = "src/AI/tests/Essentials.AI.UnitTests/Essentials.AI.UnitTests.csproj"
+}
+


$TestTypePatterns, $UnitTestProjectMap, $DeviceTestProjectMap, and Get-TestTypeFromFiles appear to be unused now that detection is delegated to .github/scripts/shared/Detect-TestsInDiff.ps1 (nothing calls Get-TestTypeFromFiles). Keeping two independent detection implementations increases drift risk; consider removing this unused block or wiring the script to use it consistently.

Suggested change

# Maps file path patterns to test types

$script:TestTypePatterns = @(

@{ Pattern = "TestCases\.(Shared\.Tests|HostApp|Android\.Tests|iOS\.Tests|Mac\.Tests|WinUI\.Tests)"; Type = "UITest" }

@{ Pattern = "Xaml\.UnitTests/"; Type = "XamlUnitTest" }

@{ Pattern = "DeviceTests/"; Type = "DeviceTest" }

@{ Pattern = "(?<!\w)UnitTests/|Graphics\.Tests/"; Type = "UnitTest" }

)

# Maps test types to their project paths (relative to repo root)

$script:UnitTestProjectMap = @{

"Controls.Core.UnitTests" = "src/Controls/tests/Core.UnitTests/Controls.Core.UnitTests.csproj"

"Controls.Xaml.UnitTests" = "src/Controls/tests/Xaml.UnitTests/Controls.Xaml.UnitTests.csproj"

"Controls.BindingSourceGen.UnitTests" = "src/Controls/tests/BindingSourceGen.UnitTests/Controls.BindingSourceGen.UnitTests.csproj"

"SourceGen.UnitTests" = "src/Controls/tests/SourceGen.UnitTests/SourceGen.UnitTests.csproj"

"Core.UnitTests" = "src/Core/tests/UnitTests/Core.UnitTests.csproj"

"Essentials.UnitTests" = "src/Essentials/test/UnitTests/Essentials.UnitTests.csproj"

"Graphics.Tests" = "src/Graphics/tests/Graphics.Tests/Graphics.Tests.csproj"

"Resizetizer.UnitTests" = "src/SingleProject/Resizetizer/test/UnitTests/Resizetizer.UnitTests.csproj"

"Compatibility.Core.UnitTests" = "src/Compatibility/Core/tests/Compatibility.UnitTests/Compatibility.Core.UnitTests.csproj"

"Essentials.AI.UnitTests" = "src/AI/tests/Essentials.AI.UnitTests/Essentials.AI.UnitTests.csproj"

}

Copilot · 2026-03-25T21:14:14Z

.github/scripts/shared/Detect-TestsInDiff.ps1

+    - Type:        UITest | UnitTest | XamlUnitTest | DeviceTest
+    - TestName:    Human-readable test name (class name or method name)
+    - Filter:      dotnet test --filter value
+    - Project:     Project key for device tests (Controls, Core, etc.)
+    - ProjectPath: Relative .csproj path for unit tests
+    - Runner:      Which script runs it (BuildAndRunHostApp, dotnet-test, Run-DeviceTests)
+    - Platform:    Whether -Platform is required
+    - Files:       List of test files


The .OUTPUTS block says the script returns a Platform field indicating whether -Platform is required, but the actual objects use NeedsPlatform (and no Platform key). Please update the docs to match the real output shape so callers don’t rely on a non-existent property.

Suggested change

- Type: UITest | UnitTest | XamlUnitTest | DeviceTest

- TestName: Human-readable test name (class name or method name)

- Filter: dotnet test --filter value

- Project: Project key for device tests (Controls, Core, etc.)

- ProjectPath: Relative .csproj path for unit tests

- Runner: Which script runs it (BuildAndRunHostApp, dotnet-test, Run-DeviceTests)

- Platform: Whether -Platform is required

- Files: List of test files

- Type: UITest | UnitTest | XamlUnitTest | DeviceTest

- TestName: Human-readable test name (class name or method name)

- Filter: dotnet test --filter value

- Project: Project key for device tests (Controls, Core, etc.)

- ProjectPath: Relative .csproj path for unit tests

- Runner: Which script runs it (BuildAndRunHostApp, dotnet-test, Run-DeviceTests)

- NeedsPlatform: Whether -Platform is required

- Files: List of test files

Copilot · 2026-03-25T21:14:14Z

.github/scripts/RunTests.ps1

+        $testArgs = @(
+            "test", $projectPath,
+            "--configuration", $Config,
+            "--no-restore",


Invoke-UnitTests runs dotnet test with --no-restore, which makes the new unified runner fail on a clean checkout (no prior restore). Unless you’re restoring earlier in this script, consider removing --no-restore or adding a -NoRestore/-Restore switch so default usage works out of the box.

Suggested change

"--no-restore",

Copilot · 2026-03-25T21:14:14Z

.github/skills/verify-tests-fail-without-fix/SKILL.md

 ---
 name: verify-tests-fail-without-fix
-description: Verifies UI tests catch the bug. Supports two modes - verify failure only (test creation) or full verification (test + fix validation).
+description: Verifies tests catch the bug. Auto-detects test type (UI tests, device tests, unit tests) and dispatches to the appropriate runner. Supports two modes - verify failure only (test creation) or full verification (test + fix validation).


The skill frontmatter description mentions auto-detecting UI/device/unit tests, but this PR also adds XAML unit test support. Consider updating the description to explicitly include XAML unit tests for accuracy/consistency with the Supported Test Types table below.

Suggested change

description: Verifies tests catch the bug. Auto-detects test type (UI tests, device tests, unit tests) and dispatches to the appropriate runner. Supports two modes - verify failure only (test creation) or full verification (test + fix validation).

description: Verifies tests catch the bug. Auto-detects test type (UI tests, device tests, unit tests, XAML unit tests) and dispatches to the appropriate runner. Supports two modes - verify failure only (test creation) or full verification (test + fix validation).

Copilot · 2026-03-25T21:14:15Z

.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1

 .PARAMETER Platform
    Target platform: "android", "ios", "catalyst" (MacCatalyst), or "windows"
+    Required for UITest and DeviceTest types. Optional for UnitTest and XamlUnitTest.
+


The script header says -Platform is optional for UnitTest/XamlUnitTest, but the script’s param(...) still declares Platform as mandatory. As a result, the documented “no platform needed” scenarios won’t work unless Platform is made optional (with runtime validation only for UI/Device tests) or the docs are updated to say it’s always required.

Copilot · 2026-03-25T21:14:15Z

.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1

+        $TestLog = Join-Path $OutputPath "test-failure-$($testEntry.TestName).log"
+


$testEntry.TestName is used directly in the log file name. When -TestFilter is provided explicitly, TestName is set to the raw filter (e.g. Issue32030|ButtonUITests per docs), which contains characters like | that are invalid in Windows file names. Please sanitize/slugify the value used for file names (keep the original for display).

Suggested change

$TestLog = Join-Path $OutputPath "test-failure-$($testEntry.TestName).log"

# Sanitize test name for use in file system paths (keep original for display and reporting)

$invalidFileNameChars = [IO.Path]::GetInvalidFileNameChars()

$escapedInvalidFileNameChars = [regex]::Escape(-join $invalidFileNameChars)

$sanitizedTestName = $testEntry.TestName -replace "[${escapedInvalidFileNameChars}]", "_"

$TestLog = Join-Path $OutputPath "test-failure-$sanitizedTestName.log"

Copilot · 2026-03-25T21:14:15Z

.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1

+
+    $testLogFile = Join-Path $OutputPath "test-without-fix-$($testEntry.TestName).log"
+


Same issue as above: $testEntry.TestName is used in test-without-fix-<name>.log. If the test name comes from an explicit -TestFilter (or includes characters like |, :, /), this will fail on Windows. Sanitize the name used for the file path.

Copilot · 2026-03-25T21:14:15Z

.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1

+    Write-Log ""
+    Write-Log "--- Test $testIndex/$($AllDetectedTests.Count): $icon [$($testEntry.Type)] $($testEntry.TestName) ---"
+
+    $testLogFile = Join-Path $OutputPath "test-with-fix-$($testEntry.TestName).log"


Same file-name sanitization issue for test-with-fix-<name>.log: $testEntry.TestName can contain characters invalid for file names (notably | from OR filters). Use a sanitized identifier for file paths and keep the original name only for display/reporting.

Suggested change

$testLogFile = Join-Path $OutputPath "test-with-fix-$($testEntry.TestName).log"

$sanitizedTestName = $testEntry.TestName -replace '[\\/:*?"<>|]', '_'

$testLogFile = Join-Path $OutputPath "test-with-fix-$sanitizedTestName.log"

Copilot · 2026-03-25T21:14:16Z

.github/scripts/shared/Detect-TestsInDiff.ps1

+        if ($PRNumber) {
+            # Get per-file patch from GitHub API
+            $patch = gh api "repos/dotnet/maui/pulls/$PRNumber/files" --jq ".[] | select(.filename == `"$file`") | .patch" 2>$null
+        } else {


For PR-based analysis, the device-test method extraction calls gh api .../pulls/$PRNumber/files inside the per-file loop. That refetches the full PR file list for every file and can be very slow (and hit rate limits) on large PRs. Fetch the PR files/patches once, cache them in memory (e.g., a filename→patch map), and then look up patches per file.

Copilot AI review requested due to automatic review settings March 25, 2026 21:09

Copilot started reviewing on behalf of kubaflo March 25, 2026 21:10 View session

kubaflo closed this Mar 25, 2026

kubaflo deleted the infra/gate-all-test-types branch March 25, 2026 21:12

Copilot AI reviewed Mar 25, 2026

View reviewed changes

github-actions bot mentioned this pull request Mar 26, 2026

[repo-status] Daily Repo Status - March 26, 2026 #34665

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend gate to support all test types (UI, device, unit, XAML)#34657

Extend gate to support all test types (UI, device, unit, XAML)#34657
kubaflo wants to merge 1 commit intomainfrom
infra/gate-all-test-types

kubaflo commented Mar 25, 2026

Uh oh!

github-actions bot commented Mar 25, 2026

Uh oh!

kubaflo commented Mar 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	description: Verifies tests catch the bug. Auto-detects test type (UI tests, device tests, unit tests) and dispatches to the appropriate runner. Supports two modes - verify failure only (test creation) or full verification (test + fix validation).
	description: Verifies tests catch the bug. Auto-detects test type (UI tests, device tests, unit tests, XAML unit tests) and dispatches to the appropriate runner. Supports two modes - verify failure only (test creation) or full verification (test + fix validation).

		$TestLog = Join-Path $OutputPath "test-failure-$($testEntry.TestName).log"

-        $TestLog = Join-Path $OutputPath "test-failure-$($testEntry.TestName).log"
+        # Sanitize test name for use in file system paths (keep original for display and reporting)
+        $invalidFileNameChars = [IO.Path]::GetInvalidFileNameChars()
+        $escapedInvalidFileNameChars = [regex]::Escape(-join $invalidFileNameChars)
+        $sanitizedTestName = $testEntry.TestName -replace "[${escapedInvalidFileNameChars}]", "_"
+        $TestLog = Join-Path $OutputPath "test-failure-$sanitizedTestName.log"


		$testLogFile = Join-Path $OutputPath "test-without-fix-$($testEntry.TestName).log"

	$testLogFile = Join-Path $OutputPath "test-with-fix-$($testEntry.TestName).log"
	$sanitizedTestName = $testEntry.TestName -replace '[\\/:*?"<>\|]', '_'
	$testLogFile = Join-Path $OutputPath "test-with-fix-$sanitizedTestName.log"

Conversation

kubaflo commented Mar 25, 2026

Changes

Test Types Supported

Verified

Uh oh!

github-actions bot commented Mar 25, 2026

Uh oh!

kubaflo commented Mar 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants