Add CI test to validate Azure MCP tool names in skill markdown files by Copilot · Pull Request #1709 · microsoft/GitHub-Copilot-for-Azure

Copilot · 2026-04-03T21:33:34Z

Summary

Adds automated validation to ensure Azure MCP tool names referenced in skill markdown stay in sync with the tools actually exposed by @azure/mcp, and fixes existing stale tool references found by that validation.

Changes:

Extend tests/mcp-tool-references/unit.test.ts to validate both mcp_azure_mcp_<tool> and azure__<tool> naming conventions against the canonical snapshot at tests/fixtures/azure-mcp-tool-names.snapshot.json.
Fix stale tool references (get_bestpractices → get_azure_bestpractices) across affected skill docs.
Update azure-upgrade and azure-cloud-migrate SKILL tool lists/usages to the corrected tool name.

File	Description
`tests/mcp-tool-references/unit.test.ts`	Extended to validate both `mcp_azure_mcp_` and `azure__` tool references with line-number diagnostics.
`plugin/skills/azure-upgrade/SKILL.md`	Fixes Azure MCP best-practices tool name in rules/tool table.
`plugin/skills/azure-upgrade/references/services/functions/assessment.md`	Fixes best-practices tool reference in tip.
`plugin/skills/azure-upgrade/references/global-rules.md`	Fixes best-practices tool reference in global rules.
`plugin/skills/azure-prepare/references/recipes/terraform/README.md`	Fixes best-practices tool reference in Terraform recipe table.
`plugin/skills/azure-prepare/references/recipes/azcli/README.md`	Fixes best-practices tool reference in Azure CLI recipe table.
`plugin/skills/azure-cloud-migrate/SKILL.md`	Fixes Azure MCP best-practices tool name in rules.
`plugin/skills/azure-cloud-migrate/references/services/functions/global-rules.md`	Fixes best-practices tool reference in rules.
`plugin/skills/azure-cloud-migrate/references/services/functions/code-migration.md`	Fixes best-practices tool reference in prerequisites/steps.
`plugin/skills/azure-cloud-migrate/references/services/functions/assessment.md`	Fixes best-practices tool reference in tip.

…eferences Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/362a2400-266c-4070-9a8b-625601205ea3 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

kvenkatrajan · 2026-04-03T22:04:21Z

@copilot - bump the version on 'azure-cloud-migrate' , 'azure-prepare', 'azure-upgrade' skills

…de skills Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/5a67a30f-b6d1-43a9-8c38-a2ad05209e1b Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Copilot · 2026-04-03T22:08:36Z

@copilot - bump the version on 'azure-cloud-migrate' , 'azure-prepare', 'azure-upgrade' skills

Done in commit 243a564. Bumped versions for the three modified skills:

azure-cloud-migrate: 1.0.1 → 1.0.2
azure-prepare: 1.1.6 → 1.1.7
azure-upgrade: 1.0.0 → 1.0.1

Copilot

Pull request overview

Adds automated validation to ensure Azure MCP tool names referenced in skill markdown stay in sync with the tools actually exposed by @azure/mcp, and fixes existing stale tool references found by that validation.

Changes:

Add a new unit test that boots @azure/mcp and validates referenced Azure MCP tool names across plugin/skills/**/*.md.
Fix stale tool references (get_bestpractices → get_azure_bestpractices) across affected skill docs.
Update azure-upgrade and azure-cloud-migrate SKILL tool lists/usages to the corrected tool name.

Show a summary per file

File	Description
tests/skills-validation/mcp-tool-names.test.ts	New unit test that queries Azure MCP `tools/list` and validates tool references in skill markdown.
plugin/skills/azure-upgrade/SKILL.md	Fixes Azure MCP best-practices tool name in rules/tool table.
plugin/skills/azure-upgrade/references/services/functions/assessment.md	Fixes best-practices tool reference in tip.
plugin/skills/azure-upgrade/references/global-rules.md	Fixes best-practices tool reference in global rules.
plugin/skills/azure-prepare/references/recipes/terraform/README.md	Fixes best-practices tool reference in Terraform recipe table.
plugin/skills/azure-prepare/references/recipes/azcli/README.md	Fixes best-practices tool reference in Azure CLI recipe table.
plugin/skills/azure-cloud-migrate/SKILL.md	Fixes Azure MCP best-practices tool name in rules.
plugin/skills/azure-cloud-migrate/references/services/functions/global-rules.md	Fixes best-practices tool reference in rules.
plugin/skills/azure-cloud-migrate/references/services/functions/code-migration.md	Fixes best-practices tool reference in prerequisites/steps.
plugin/skills/azure-cloud-migrate/references/services/functions/assessment.md	Fixes best-practices tool reference in tip.

Copilot's findings

Files reviewed: 11/11 changed files
Comments generated: 3

kvenkatrajan · 2026-04-03T22:26:07Z

Fixes #778

Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/bfd49079-1961-4c4e-b89f-d5e1a195b0ac Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

jongio

The approach works well - validating tool names against the live server catches drift that's hard to spot manually, and the 9 fixes prove the test's value.

Three suggestions on the test implementation, plus a note that the PR currently has merge conflicts to resolve.

…mes test Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/3db4fef5-f378-4506-83b4-7a2f8ee1c5d4 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Copilot

Copilot's findings

Files reviewed: 11/11 changed files
Comments generated: 2

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/776538a2-cde8-4dfe-a517-2e4a379768f0 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Copilot

Copilot's findings

Files reviewed: 11/11 changed files
Comments generated: 1

Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/e6709d33-cd52-4a46-a86c-fcd62e12ed33 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot's findings

Files reviewed: 11/11 changed files
Comments generated: 2

…icrosoft#1709) * Initial plan * Add CI test for Azure MCP tool name validation and fix invalid tool references Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/362a2400-266c-4070-9a8b-625601205ea3 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com> * Bump versions for azure-cloud-migrate, azure-prepare, and azure-upgrade skills Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/5a67a30f-b6d1-43a9-8c38-a2ad05209e1b Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com> * Reject early on MCP server process exit with stderr diagnostics Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/bfd49079-1961-4c4e-b89f-d5e1a195b0ac Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com> * Add shell:true to spawn and defensive pagination check in mcp-tool-names test Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/3db4fef5-f378-4506-83b4-7a2f8ee1c5d4 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com> * refactor: replace live MCP server with static azure-mcp-tools.json snapshot in unit test - Add tests/skills-validation/azure-mcp-tools.json with current tool list from @azure/mcp - Refactor mcp-tool-names.test.ts to read from snapshot (fast, offline, deterministic) - Remove spawn/child_process dependency, 120s beforeAll timeout, and live server logic - Test now completes in ~2s instead of up to 2 minutes Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/123eec45-bcc4-4e61-815f-89ffe090b2da Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com> * chore: bump azure-prepare version to 1.1.8 Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/770aa1df-9beb-4d56-92d3-4e3118fce322 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com> * chore: set azure-prepare version to 1.1.21 Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/7aeb6af7-773c-49e8-ae3d-d3a3de0dad4e Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com> * chore: bump azure-cloud-migrate version to 1.0.3 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update tests/skills-validation/azure-mcp-tools.json Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/skills-validation/mcp-tool-names.test.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * refactor: use canonical snapshot in mcp-tool-names test, remove duplicate azure-mcp-tools.json Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/1b208d96-74e8-4944-99d7-8f8d86bd97c4 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com> * refactor: consolidate MCP tool name validation into single test Address Jon's feedback (r3103830005): - Add azure__<tool> regex pattern to tests/mcp-tool-references/unit.test.ts so both naming conventions validate against the one auto-refreshed snapshot - Delete tests/skills-validation/mcp-tool-names.test.ts (duplicate test) - Include line numbers in failure output for easier debugging - Keep get_bestpractices alias (still referenced in azure-cost skill) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding 'CodeQL / Unused variable, import, function or class' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Update tests/mcp-tool-references/unit.test.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * test: validate snapshot shape before using toolNames in beforeAll Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/776538a2-cde8-4dfe-a517-2e4a379768f0 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com> * test: tighten Azure MCP reference token extraction and validation Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/e6709d33-cd52-4a46-a86c-fcd62e12ed33 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com> * chore: bump azure-prepare version to 1.1.22 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com> Co-authored-by: Kay Venkatrajan <kvenkatrajan@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Initial plan

9106121

Copilot AI assigned Copilot and kvenkatrajan Apr 3, 2026

Copilot started work on behalf of kvenkatrajan April 3, 2026 21:33 View session

Copilot AI linked an issue Apr 3, 2026 that may be closed by this pull request

PR CI tests for ensuring that the tool names in md file reflect actual Azure MCP server tools #778

Closed

Copilot AI changed the title ~~[WIP] Add CI tests for validating Azure MCP tool names in markdown files~~ Add CI test to validate Azure MCP tool names in skill markdown files Apr 3, 2026

Copilot finished work on behalf of kvenkatrajan April 3, 2026 21:55

Copilot AI requested a review from kvenkatrajan April 3, 2026 21:55

kvenkatrajan marked this pull request as ready for review April 3, 2026 21:59

kvenkatrajan requested review from saikoumudi, tmeschter and wbreza as code owners April 3, 2026 21:59

Copilot AI review requested due to automatic review settings April 3, 2026 21:59

Copilot started reviewing on behalf of kvenkatrajan April 3, 2026 22:03 View session

Copilot started work on behalf of kvenkatrajan April 3, 2026 22:04 View session

Copilot finished work on behalf of kvenkatrajan April 3, 2026 22:09

Copilot AI reviewed Apr 3, 2026

View reviewed changes

Comment thread tests/skills-validation/mcp-tool-names.test.ts Outdated

Comment thread plugin/skills/azure-upgrade/SKILL.md

Comment thread plugin/skills/azure-cloud-migrate/SKILL.md

Copilot started work on behalf of kvenkatrajan April 3, 2026 22:19 View session

Reject early on MCP server process exit with stderr diagnostics

26d67d8

Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/bfd49079-1961-4c4e-b89f-d5e1a195b0ac Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Copilot finished work on behalf of kvenkatrajan April 3, 2026 22:31

jongio reviewed Apr 8, 2026

View reviewed changes

Comment thread tests/skills-validation/mcp-tool-names.test.ts Outdated

Comment thread tests/skills-validation/mcp-tool-names.test.ts Outdated

Comment thread tests/skills-validation/mcp-tool-names.test.ts Outdated

Copilot started work on behalf of kvenkatrajan April 17, 2026 15:57 View session

Copilot AI review requested due to automatic review settings April 17, 2026 16:00

Copilot AI requested a review from MadhuraBharadwaj-MSFT as a code owner April 17, 2026 16:00

Copilot AI review requested due to automatic review settings April 17, 2026 23:32

Copilot started reviewing on behalf of kvenkatrajan April 17, 2026 23:32 View session

Copilot started work on behalf of kvenkatrajan April 17, 2026 23:33 View session

Copilot AI reviewed Apr 17, 2026

View reviewed changes

Comment thread tests/mcp-tool-references/unit.test.ts

Comment thread tests/mcp-tool-references/unit.test.ts Outdated

Copilot started work on behalf of kvenkatrajan April 17, 2026 23:36 View session

Copilot finished work on behalf of kvenkatrajan April 17, 2026 23:37

Update tests/mcp-tool-references/unit.test.ts

0e87988

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 17, 2026 23:37

test: validate snapshot shape before using toolNames in beforeAll

f41fdbd

Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/776538a2-cde8-4dfe-a517-2e4a379768f0 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Copilot started reviewing on behalf of kvenkatrajan April 17, 2026 23:38 View session

Copilot finished work on behalf of kvenkatrajan April 17, 2026 23:39

Copilot AI reviewed Apr 17, 2026

View reviewed changes

Comment thread tests/mcp-tool-references/unit.test.ts Outdated

Copilot started work on behalf of kvenkatrajan April 17, 2026 23:42 View session

test: tighten Azure MCP reference token extraction and validation

aa7d5c4

Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/e6709d33-cd52-4a46-a86c-fcd62e12ed33 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Copilot finished work on behalf of kvenkatrajan April 17, 2026 23:45

saikoumudi previously approved these changes Apr 17, 2026

View reviewed changes

chore: bump azure-prepare version to 1.1.22

e321aa0

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 17, 2026 23:52

kvenkatrajan dismissed saikoumudi’s stale review via e321aa0 April 17, 2026 23:52

Copilot started reviewing on behalf of kvenkatrajan April 17, 2026 23:53 View session

fix: resolve merge conflict in azure-prepare version (keep 1.1.22)

fec7ccc

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI reviewed Apr 17, 2026

View reviewed changes

Comment thread plugin/skills/azure-prepare/SKILL.md

Comment thread tests/mcp-tool-references/unit.test.ts

Copilot started work on behalf of kvenkatrajan April 18, 2026 00:09 View session

Copilot finished work on behalf of kvenkatrajan April 18, 2026 00:12

kvenkatrajan approved these changes Apr 18, 2026

View reviewed changes

saikoumudi approved these changes Apr 18, 2026

View reviewed changes

saikoumudi merged commit 946fa11 into main Apr 18, 2026
13 checks passed

saikoumudi deleted the copilot/add-ci-test-for-azure-mcp-tools branch April 18, 2026 00:40

github-actions Bot mentioned this pull request Apr 18, 2026

[aw] No-Op Runs #1950

Closed

Conversation

Copilot AI commented Apr 3, 2026 • edited by kvenkatrajan Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

kvenkatrajan commented Apr 3, 2026

Uh oh!

Copilot AI commented Apr 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kvenkatrajan commented Apr 3, 2026

Uh oh!

jongio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Copilot AI commented Apr 3, 2026 •

edited by kvenkatrajan

Loading