Skip to content

Allow fork PRs to auto-trigger evaluate-pr-tests workflow#34655

Merged
PureWeen merged 4 commits intomainfrom
feature/evaluate-tests-fork-support
Mar 25, 2026
Merged

Allow fork PRs to auto-trigger evaluate-pr-tests workflow#34655
PureWeen merged 4 commits intomainfrom
feature/evaluate-tests-fork-support

Conversation

@PureWeen
Copy link
Copy Markdown
Member

@PureWeen PureWeen commented Mar 25, 2026

Summary

Enables the copilot-evaluate-tests gh-aw workflow to run on fork PRs by adding forks: ["*"] to the pull_request trigger and removing the fork guard from Checkout-GhAwPr.ps1.

Changes

  1. copilot-evaluate-tests.md: Added forks: ["*"] to opt out of gh-aw auto-injected fork activation guard. Scoped Checkout-GhAwPr.ps1 step to workflow_dispatch only (redundant for other triggers since platform handles checkout).

  2. copilot-evaluate-tests.lock.yml: Recompiled via gh aw compile — fork guard removed from activation if: conditions.

  3. Checkout-GhAwPr.ps1: Removed the isCrossRepository fork guard. Updated header docs and restore comments to accurately describe behavior for all trigger×fork combinations (including corrected step ordering).

  4. gh-aw-workflows.instructions.md: Updated all stale references to the removed fork guard. Documented forks: ["*"] opt-in, clarified residual risk model for fork PRs, and updated troubleshooting table.

Security Model

Fork PRs are safe because:

  • Agent runs in sandboxed container with all credentials scrubbed
  • Output limited to 1 comment via safe-outputs: add-comment: max: 1
  • Agent prompt comes from base branch (runtime-import) — forks cannot alter instructions
  • Pre-flight check catches missing SKILL.md if fork isn't rebased on main
  • No workspace code is executed with GITHUB_TOKEN (checkout without execution)

Testing

Copilot AI review requested due to automatic review settings March 25, 2026 19:51
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 25, 2026

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 34655

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 34655"

Add forks: ["*"] to pull_request trigger. Safe because:
- Agent runs in sandboxed container with scrubbed credentials
- Checkout-GhAwPr.ps1 restores trusted skills from base branch
- safe-outputs limits to 1 sanitized comment
- All filenames escaped via Escape-ForCodeSpan

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@PureWeen PureWeen force-pushed the feature/evaluate-tests-fork-support branch from 7100785 to c7d545e Compare March 25, 2026 19:54
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the Evaluate PR Tests gh-aw workflow to broaden when it can run (intended to include fork PRs) and hardens the supporting scripts around PR checkout and report generation.

Changes:

  • Adds forks: ["*"] to the workflow’s pull_request trigger and updates the compiled .lock.yml.
  • Introduces a shared .github/scripts/Checkout-GhAwPr.ps1 to checkout PRs and restore trusted agent infrastructure.
  • Enhances Gather-TestContext.ps1 with -PrNumber support and additional markdown escaping for PR-sourced filenames.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

Show a summary per file
File Description
.github/workflows/copilot-evaluate-tests.md Adjusts workflow trigger configuration to (intended) allow fork PRs.
.github/workflows/copilot-evaluate-tests.lock.yml Regenerated compiled workflow reflecting the trigger and execution model changes.
.github/skills/evaluate-pr-tests/scripts/Gather-TestContext.ps1 Adds PR-number-based diffing/download and improves output escaping.
.github/scripts/Checkout-GhAwPr.ps1 Adds a shared pre-agent checkout + base restore mechanism for gh-aw workflows.
.github/instructions/gh-aw-workflows.instructions.md Adds repo guidance for authoring gh-aw workflows and fork handling patterns.
.github/aw/actions-lock.json Adds pinned entries for github/gh-aw-actions/setup versions used by the workflow compilation.
Comments suppressed due to low confidence (3)

.github/workflows/copilot-evaluate-tests.lock.yml:112

  • On pull_request events from forks, GitHub Actions does not provide repository secrets to the workflow run. This workflow depends on secrets.COPILOT_GITHUB_TOKEN (validated here and used later to run Copilot), so fork-triggered runs would fail even if fork gating is removed. If the intent is truly “auto-trigger on fork PRs”, this likely needs a different trigger strategy (e.g., pull_request_target with a hardened checkout/restore model) or an explicit maintainer-approved dispatch/comment flow instead.
      - name: Validate COPILOT_GITHUB_TOKEN secret
        id: validate-secret
        run: ${RUNNER_TEMP}/gh-aw/actions/validate_multi_secret.sh COPILOT_GITHUB_TOKEN 'GitHub Copilot CLI' https://github.github.com/gh-aw/reference/engines/#github-copilot-default
        env:
          COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }}

.github/workflows/copilot-evaluate-tests.lock.yml:1008

  • The workflow still has a pre_activation membership gate (activated: ...is_team_member == 'true'). For fork PRs opened by external contributors, this will evaluate to false, so the workflow won’t activate even with forks: ["*"]. Either adjust the activation gating to allow pull_request events from forks (while keeping stricter gating for /evaluate-tests), or update the PR title/description because this change alone doesn’t enable auto-triggering for fork PRs.
  pre_activation:
    if: >
      (github.event_name == 'pull_request' && github.event.pull_request.draft == false) || github.event_name == 'workflow_dispatch' || (github.event_name == 'issue_comment' &&
      github.event.issue.pull_request &&
      startsWith(github.event.comment.body, '/evaluate-tests'))
    runs-on: ubuntu-slim
    outputs:
      activated: ${{ steps.check_membership.outputs.is_team_member == 'true' }}
      matched_command: ''
    steps:
      - name: Setup Scripts
        uses: github/gh-aw-actions/setup@20045bbd5ad2632b9809856c389708eab1bd16ef # v0.62.2
        with:
          destination: ${{ runner.temp }}/gh-aw/actions
      - name: Check team membership for workflow
        id: check_membership
        uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
        env:
          GH_AW_REQUIRED_ROLES: admin,maintainer,write
        with:

.github/workflows/copilot-evaluate-tests.lock.yml:121

  • Security: the activation job checks out .github via actions/checkout without pinning ref to the base branch SHA. On pull_request events this normally checks out refs/pull/<n>/merge, meaning a fork PR can influence the working tree used for runtime-import prompt rendering (and any other files read in activation). If fork PRs are allowed to trigger this workflow, consider pinning checkout to the base SHA/ref for pull_request events or keep the fork guard so untrusted forks can’t affect the prompt/activation context.
      - name: Checkout .github and .agents folders
        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          persist-credentials: false
          sparse-checkout: |
            .github
            .agents
          sparse-checkout-cone-mode: true
          fetch-depth: 1

github-actions bot and others added 2 commits March 25, 2026 15:08
The agent runs sandboxed with no credentials and can only post 1 comment.
For issue_comment on forks, the platform may overwrite restored skills,
but the agent pre-flight catches missing SKILL.md gracefully, and even
a manipulated SKILL.md can only produce a misleading comment.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
For pull_request and issue_comment triggers, the gh-aw platform's
checkout_pr_branch.cjs already handles PR checkout. Our script was
redundant for those paths (and its restore was overwritten by the
platform re-checkout).

Now only runs for workflow_dispatch where the platform checkout is
skipped entirely.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@PureWeen PureWeen force-pushed the feature/evaluate-tests-fork-support branch from d777812 to b12fdee Compare March 25, 2026 20:31
- Update gh-aw-workflows.instructions.md: remove all references to the
  deleted fork guard, document forks: ["*"] opt-in, clarify residual
  risk for fork PRs on all triggers
- Fix Checkout-GhAwPr.ps1 comment: platform checkout runs AFTER user
  steps (not before) for pull_request triggers, matching the documented
  gh-aw execution model

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@PureWeen PureWeen force-pushed the feature/evaluate-tests-fork-support branch from b12fdee to 4328beb Compare March 25, 2026 20:34
@PureWeen PureWeen merged commit 720a9d4 into main Mar 25, 2026
9 of 10 checks passed
@PureWeen PureWeen deleted the feature/evaluate-tests-fork-support branch March 25, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants