Add weekly AI docs audit via Copilot coding agent#35
Add weekly AI docs audit via Copilot coding agent#35jfversluis wants to merge 2 commits intomainfrom
Conversation
Introduces a scheduled GitHub Actions workflow that runs weekly (Sunday 03:00 UTC) to detect when merged PRs have changed things documented in AI config files (AGENTS.md, instruction files, copilot-instructions.md). The workflow: 1. Collects all PRs merged in the past 7 days with their changed files 2. Creates an issue summarizing what changed 3. Assigns the issue to the Copilot coding agent with a custom 'docs-keeper' agent profile that knows which files to audit and how to verify accuracy The docs-keeper custom agent (.github/agents/docs-keeper.md) specializes in auditing MCP tool tables, package lists, architecture diagrams, build commands, and conventions against the actual source code. Also supports workflow_dispatch for on-demand runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the manual GitHub Actions YAML + custom agent approach with a proper GitHub Agentic Workflow (.md + compiled .lock.yml). The workflow: - Runs weekly (schedule: weekly) or on-demand (workflow_dispatch) - Agent runs read-only, reviews merged PRs from the past 7 days - Audits AI config files (AGENTS.md, instruction files) against actual source - Opens a draft PR via safe-outputs if anything is stale - Creates an issue if nothing needs updating (for audit trail) Built-in security from gh-aw: - Sandboxed agent execution with network isolation - Read-only permissions; writes via safe-outputs only - Threat detection pipeline before any PR is created - Tool allowlisting (edit, bash, github only) Files: - .github/workflows/ai-docs-audit.md — workflow definition - .github/workflows/ai-docs-audit.lock.yml — compiled Actions YAML - .github/aw/actions-lock.json — SHA-pinned action references - .gitattributes — marks lock files as generated Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
We should probably merge #34 first |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| ## Rules | ||
|
|
||
| - Only modify files listed in the `allowed-files` safe output config. |
There was a problem hiding this comment.
The workflow instructions mention an allowed-files safe-outputs config, but the frontmatter safe-outputs: section does not define any file allowlist (and there’s no other allowed-files configuration in this workflow). This makes the rule unenforceable and increases the risk of the agent generating patches that touch unintended files. Either add the missing allowlist configuration (if supported) and align the prompt to it, or remove/replace this rule with the actual enforcement mechanism used by gh-aw/safe-outputs.
| - Only modify files listed in the `allowed-files` safe output config. | |
| - Only modify AI configuration and documentation files covered by this audit (AGENTS.md, copilot-instructions.md, instruction files, and related docs). |
| ## Step 2: Audit each AI config file | ||
|
|
||
| For each file below, verify its contents against the actual source code. | ||
|
|
||
| ### AGENTS.md | ||
|
|
||
| - **MCP tools table**: Scan every `[McpServerTool]` attribute in | ||
| `src/DevFlow/**/Mcp/Tools/*.cs`. The table in AGENTS.md must list every tool | ||
| with its correct name, source file, and description. Regenerate the entire | ||
| table from source rather than patching individual rows. | ||
| - **Package list**: Check all `.csproj` files for `IsPackable=true` and | ||
| cross-reference with the documented package list. | ||
| - **Build commands**: Verify solution filter paths (`ls src/DevFlow/*.slnf | ||
| src/Client/*.slnf 2>/dev/null`) and ensure build commands are correct. | ||
| - **NuGet feeds**: Compare `NuGet.config` with the documented feed list. | ||
| - **SDK version**: Check `global.json` against documented version. | ||
|
|
||
| ### .github/copilot-instructions.md | ||
|
|
||
| - **Platform patterns**: Verify that documented code patterns (Core/Platform, | ||
| `#if` guards, handler registration) still match actual source conventions. | ||
| - **MCP tool conventions**: Check that naming conventions (`maui_` prefix, | ||
| snake_case, parameter ordering) match actual tools. | ||
| - **Arcade gotchas**: Verify documented build system quirks are still relevant. | ||
|
|
There was a problem hiding this comment.
Several files called out as existing audit targets (AGENTS.md, .github/copilot-instructions.md, and .github/instructions/*.instructions.md) are not currently present in the repo. As written, the workflow doesn’t say what to do in that case (create baseline files vs. report missing vs. skip), which could lead to unexpected PRs on the first run. Please clarify the expected behavior when an audited file/directory is missing and ensure the safe-outputs/write restrictions match that behavior.
Summary
Adds a GitHub Agentic Workflow (
gh aw) that automatically keeps AI configuration files in sync with the codebase.How it works
The workflow is defined as a Markdown file (
.github/workflows/ai-docs-audit.md) that compiles to a GitHub Actions lock file viagh aw compile. It runs weekly or on-demand.AGENTS.md— MCP tools table (scans[McpServerTool]attributes), package list, build commands, feeds.github/copilot-instructions.md— code generation guidance, conventions.github/instructions/*.instructions.md— architecture, MCP tools guide, testing, packagingWhy GitHub Agentic Workflows?
Files
.github/workflows/ai-docs-audit.md.github/workflows/ai-docs-audit.lock.yml.github/aw/actions-lock.json.gitattributes.lock.ymlfiles as generatedNotes
workflow_dispatchfor on-demand runsclose-older-issues: trueon the issue output cleans up stale audit issuesgh aw compile .github/workflows/ai-docs-audit.md