Add find-regression-risk skill and gh-aw workflow#34650
Add find-regression-risk skill and gh-aw workflow#34650
Conversation
Adds a new skill and gh-aw workflow that detects if a PR modifies or reverts code from a recent bug-fix PR, which could re-introduce a previously-fixed bug. Motivation: Three P/0 regressions (#34634, #34635, #34636) shipped in 10.0.60 because the agent approved PRs that unknowingly reverted lines from prior fix PRs. This skill would have caught them — e.g., PR #33908 removed a line from PR #32278 that fixed #32436, and the exact same bug reappeared. How it works: - For each implementation file in the PR diff, queries git log for recent PRs (6 months) - Checks if those PRs were bug fixes (via PR labels or linked issue labels: i/regression, t/bug, p/0, p/1) - Compares lines ADDED by fix PRs against lines REMOVED by the current PR - Reports revert risks (🔴), overlaps (🟡), or clean (🟢) Triggers: - pull_request: auto on src/**/*.cs or src/**/*.xaml changes - workflow_dispatch: manual — enter PR number - issue_comment: /check-regression on a PR Files: - .github/skills/find-regression-risk/SKILL.md - .github/skills/find-regression-risk/scripts/Find-RegressionRisks.ps1 - .github/workflows/copilot-find-regression-risk.md - .github/workflows/copilot-find-regression-risk.lock.yml - .github/aw/actions-lock.json (updated) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 34650Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 34650" |
There was a problem hiding this comment.
Pull request overview
Adds a new gh-aw workflow + Copilot skill to detect potential regression risks on PRs by identifying when current changes remove lines that were previously added by recent bug-fix PRs.
Changes:
- Added
copilot-find-regression-riskgh-aw workflow (source + compiled lock file) that can run on PRs, workflow_dispatch, or/check-regressioncomments. - Added
find-regression-riskskill and PowerShell script to cross-reference PR diffs against recent bug-fix PRs. - Updated gh-aw
actions-lock.jsonwith the pinnedgithub/gh-aw-actions/setup@v0.62.5entry.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/copilot-find-regression-risk.md | New workflow source that gates, checks out PR, then invokes the skill and posts results via safe-outputs. |
| .github/workflows/copilot-find-regression-risk.lock.yml | Compiled workflow output from gh aw compile with pinned actions and generated job graph. |
| .github/skills/find-regression-risk/SKILL.md | Skill definition describing intent, usage, and expected reporting format. |
| .github/skills/find-regression-risk/scripts/Find-RegressionRisks.ps1 | Implements the regression-risk detection logic (recent fix PR discovery + revert-line matching). |
| .github/aw/actions-lock.json | Adds the SHA pin for github/gh-aw-actions/setup@v0.62.5 used by the new lock workflow. |
| if (-not $labelsRaw) { | ||
| Write-Host " (could not fetch)" -ForegroundColor DarkGray | ||
| continue | ||
| } | ||
| $labels = $labelsRaw -split "`n" | ||
|
|
There was a problem hiding this comment.
This treats an empty label list as a fetch failure: gh pr view ... --jq '.labels[].name' returns no output when the PR has 0 labels, so $labelsRaw is empty and the script continues, skipping the linked-issue label check and potentially missing bug-fix PRs. Check $LASTEXITCODE (or fetch the JSON once and parse) instead of -not $labelsRaw to decide whether the call failed.
| if (-not $labelsRaw) { | |
| Write-Host " (could not fetch)" -ForegroundColor DarkGray | |
| continue | |
| } | |
| $labels = $labelsRaw -split "`n" | |
| if ($LASTEXITCODE -ne 0) { | |
| Write-Host " (could not fetch)" -ForegroundColor DarkGray | |
| continue | |
| } | |
| if ([string]::IsNullOrWhiteSpace($labelsRaw)) { | |
| $labels = @() | |
| } | |
| else { | |
| $labels = $labelsRaw -split "`n" | |
| } |
| $commits = git log --oneline --since="$sinceDate" --all -- $filePath 2>$null | ||
| if (-not $commits) { | ||
| Write-Host " 🟢 No recent commits found for this file." -ForegroundColor Green | ||
| continue | ||
| } |
There was a problem hiding this comment.
git log --since=... relies on local history being present. In the gh-aw workflows the repo checkout is typically shallow, so this may often return no commits (or far fewer than 6 months), producing false 🟢 results. Consider ensuring sufficient history before running this (e.g., git fetch --unshallow / git fetch --depth=<n> for origin/main, or use the GitHub API to query recent PRs touching a file instead of local git history).
| # Step 4: Check each recent PR | ||
| foreach ($recentPR in $prNumbers) { | ||
| Write-Host " 📋 Checking PR #$recentPR..." -ForegroundColor Gray -NoNewline | ||
|
|
||
| $labelsRaw = gh pr view $recentPR --repo $repo --json labels --jq '.labels[].name' 2>$null | ||
| if (-not $labelsRaw) { | ||
| Write-Host " (could not fetch)" -ForegroundColor DarkGray | ||
| continue | ||
| } | ||
| $labels = $labelsRaw -split "`n" | ||
|
|
||
| $isBugFix = $false | ||
| $matchedLabels = @() | ||
| foreach ($label in $labels) { | ||
| if ($label -match '^(i/regression|t/bug|p/0|p/1)$') { | ||
| $isBugFix = $true | ||
| $matchedLabels += $label | ||
| } | ||
| } | ||
|
|
||
| # Get PR body and extract linked issues | ||
| $prBody = gh pr view $recentPR --repo $repo --json body --jq '.body' 2>$null | ||
| if ($prBody -is [array]) { | ||
| $prBody = $prBody -join "`n" | ||
| } | ||
|
|
||
| $fixedIssues = @() | ||
| if ($prBody) { | ||
| $issueMatches = [regex]::Matches($prBody, '(?:Fixes|Closes|Resolves)\s+(?:https://github\.com/dotnet/maui/issues/)?#?(\d+)') | ||
| foreach ($m in $issueMatches) { | ||
| $fixedIssues += "#$($m.Groups[1].Value)" | ||
| } | ||
| # Bare "- #XXXXX" lines (Copilot agent PRs) | ||
| $normalizedBody = $prBody -replace "`r`n", "`n" | ||
| $bareIssueMatches = [regex]::Matches($normalizedBody, '(?m)^\s*-\s+#(\d+)\s*$') | ||
| foreach ($m in $bareIssueMatches) { | ||
| $ref = "#$($m.Groups[1].Value)" | ||
| if ($fixedIssues -notcontains $ref) { | ||
| $fixedIssues += $ref | ||
| } | ||
| } | ||
| $bareUrlMatches = [regex]::Matches($normalizedBody, '(?m)^\s*-\s+https://github\.com/dotnet/maui/issues/(\d+)\s*$') | ||
| foreach ($m in $bareUrlMatches) { | ||
| $ref = "#$($m.Groups[1].Value)" | ||
| if ($fixedIssues -notcontains $ref) { | ||
| $fixedIssues += $ref | ||
| } | ||
| } | ||
| } | ||
|
|
||
| # Check linked issue labels if PR itself isn't labeled | ||
| if (-not $isBugFix -and $fixedIssues.Count -gt 0) { | ||
| foreach ($issueRef in $fixedIssues) { | ||
| $issueNum = $issueRef -replace '#', '' | ||
| $issueLabelsRaw = gh issue view $issueNum --repo $repo --json labels --jq '.labels[].name' 2>$null | ||
| if ($issueLabelsRaw) { | ||
| foreach ($il in ($issueLabelsRaw -split "`n")) { | ||
| if ($il -match '^(i/regression|t/bug|p/0|p/1)$') { | ||
| $isBugFix = $true | ||
| $matchedLabels += "$il (from $issueRef)" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| if (-not $isBugFix) { | ||
| Write-Host " not a bug fix, skipping." -ForegroundColor DarkGray | ||
| continue | ||
| } | ||
|
|
||
| Write-Host " bug fix [$($matchedLabels -join ', ')]" -ForegroundColor Yellow | ||
|
|
||
| $fixedIssueStr = if ($fixedIssues.Count -gt 0) { $fixedIssues -join ', ' } else { '(unknown)' } | ||
| $prTitle = gh pr view $recentPR --repo $repo --json title --jq '.title' 2>$null | ||
| if (-not $prTitle) { $prTitle = "(unknown)" } | ||
|
|
There was a problem hiding this comment.
For each recentPR, the script calls gh pr view multiple times (labels, body, title) plus gh pr diff, and may also call gh issue view for each linked issue. On PRs that touch many files/commits this can quickly hit rate limits and slow runs. Consider fetching PR metadata once per recentPR (single gh pr view --json title,body,labels) and caching results, and only downloading diffs when the PR is confirmed to be a fix.
| $prFiles = gh pr diff $PRNumber --name-only 2>$null | ||
| if (-not $prFiles) { | ||
| Write-Host "❌ Could not get PR diff. Make sure the PR branch is available." -ForegroundColor Red | ||
| exit 1 | ||
| } | ||
| $FilePaths = $prFiles | Where-Object { | ||
| $_ -match '\.(cs|xaml)$' -and | ||
| $_ -notmatch '(Tests|TestCases|tests|snapshots|samples)/' -and | ||
| $_ -notmatch '\.Designer\.cs$' | ||
| } | ||
| Write-Host " Found $($FilePaths.Count) implementation file(s)" -ForegroundColor Gray | ||
| } | ||
|
|
||
| if ($FilePaths.Count -eq 0) { | ||
| Write-Host "🟢 No implementation files to check. Exiting." -ForegroundColor Green | ||
| exit 0 | ||
| } | ||
|
|
||
| # Step 2: Get lines REMOVED by the current PR for each file | ||
| Write-Host "" | ||
| Write-Host "📝 Getting current PR diff..." -ForegroundColor Yellow | ||
| $prDiffRaw = gh pr diff $PRNumber 2>$null | ||
| if (-not $prDiffRaw) { | ||
| Write-Host "❌ Could not get PR diff." -ForegroundColor Red | ||
| exit 1 |
There was a problem hiding this comment.
gh pr diff is called without --repo here (and again later for $prDiffRaw). Most later gh calls explicitly pass --repo $repo. For consistency and to avoid accidentally querying the wrong repository when the script is run from a different working directory, pass --repo $repo for the current PR diff calls as well.
| $risks = @() | ||
| $checkedPRs = @{} | ||
|
|
||
| foreach ($filePath in $FilePaths) { | ||
| Write-Host "" | ||
| Write-Host "🔍 Checking: $filePath" -ForegroundColor Cyan | ||
|
|
||
| $commits = git log --oneline --since="$sinceDate" --all -- $filePath 2>$null | ||
| if (-not $commits) { | ||
| Write-Host " 🟢 No recent commits found for this file." -ForegroundColor Green | ||
| continue | ||
| } | ||
|
|
||
| $prNumbers = @() | ||
| foreach ($commitLine in ($commits -split "`n")) { | ||
| if ($commitLine -match '\(#(\d+)\)') { | ||
| $foundPR = [int]$Matches[1] | ||
| if ($foundPR -ne $PRNumber -and -not $checkedPRs.ContainsKey($foundPR)) { | ||
| $prNumbers += $foundPR | ||
| $checkedPRs[$foundPR] = $true | ||
| } | ||
| } |
There was a problem hiding this comment.
$checkedPRs is used to de-duplicate PR numbers across all files, which can cause false negatives: if a single fix PR touched multiple files in $FilePaths, it will only be analyzed for the first file encountered and skipped for subsequent files, so reverts in the other files won’t be detected. Consider de-duping per file (or caching PR metadata/diff by PR number while still running the per-file revert comparison for every file).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Closed in favour of #34656 |
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Description
Adds a gh-aw (GitHub Agentic Workflows) workflow that detects regression risks on PRs by cross-referencing changes against recently-merged bug-fix PRs.
Motivation
Three P/0 regressions (#34634, #34635, #34636) shipped in 10.0.60 because the agent approved PRs that unknowingly reverted lines from prior fix PRs. For example, PR #33908 removed
|| parent is IMauiRecyclerViewfromMauiWindowInsetListener.cs— a line PR #32278 had specifically added to fix issue #32436 (increasing gap at bottom while scrolling). The exact same bug reappeared as #34634.This workflow would have caught it:
What it does
When a PR modifies implementation files, this workflow:
find-regression-riskskill via Copilot CLI in a sandboxed containerFor each implementation file in the PR diff:
git logfor recent PRs (6 months) that touched the same filei/regression,t/bug,p/0,p/1)Triggers
pull_requestsrc/**/*.csorsrc/**/*.xamlchangespre_activationgateworkflow_dispatchissue_comment(/check-regression)Security model
Same as
copilot-evaluate-tests(PR #34548):steps:checks out PR code but never executes workspace scripts.github/skills/,.github/instructions/,.github/copilot-instructions.mdrestored from base branchpull_requestevents blocked for forks.github/aw/actions-lock.jsonFiles added/modified
.github/skills/find-regression-risk/SKILL.md— Skill definition.github/skills/find-regression-risk/scripts/Find-RegressionRisks.ps1— Detection script.github/workflows/copilot-find-regression-risk.md— gh-aw workflow source.github/workflows/copilot-find-regression-risk.lock.yml— Compiled workflow (auto-generated bygh aw compile).github/aw/actions-lock.json— Updated with new action SHA pin