Skip to content

Add MCP progress reporting to SDK generation, build, and pack tools#14298

Open
raych1 wants to merge 4 commits intomainfrom
users/raych1/enable-progress-report
Open

Add MCP progress reporting to SDK generation, build, and pack tools#14298
raych1 wants to merge 4 commits intomainfrom
users/raych1/enable-progress-report

Conversation

@raych1
Copy link
Member

@raych1 raych1 commented Mar 4, 2026

Summary

Adds MCP notifications/progress support to long-running tool operations(SdkGenerationTool, SdkBuildTool, PackTool) so MCP clients like VS Code can display real-time progress indicators.

Changes

  • ProgressReporter — new helper with two reporting modes:
    • Converted ProgressReporter from a static class to an instance-based class. The constructor takes IProgress<T>, ILogger, and totalSteps — declared once, not repeated at every call.
    • NextStep(message) auto-increments the step counter and reports progress. No manual step index or total: arguments needed. Throws InvalidOperationException if more steps are reported than declared.
    • StartHeartbeat(message, ct) returns an IAsyncDisposable scope that sends periodic "still alive" notifications via PeriodicTimer. Use with await using to guarantee the heartbeat loop is fully stopped (no race conditions) before proceeding. Replaces the callback-based RunWithProgressAsync.
  • added ASCII progress bar in CLI mode.
  • enabled progress report for generate, build, and pack tools.

Milestone steps

Tool Steps Total
SdkGenerationTool (tsp-location path) Validating → Discovered repo → Regeneration complete 3
SdkGenerationTool (tspconfig path) Validating → Validating repo → Discovered repo → Generation complete 4
SdkBuildTool Validating → Detecting language → Building → Build result 4
PackTool Validating → Detecting language → Packing → Pack result 4

Related issues

#14198
#13335

Client experience

It has a blue progress bar, the tool run title is updated with the elapsed time - 15s.
image

updated with 30s
image

The build tool completed after 60 seconds in GH Copilot CLI
image

CLI progress bar:
image

@raych1 raych1 requested a review from a team as a code owner March 4, 2026 02:19
Copilot AI review requested due to automatic review settings March 4, 2026 02:19
@github-actions github-actions bot added the azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli label Mar 4, 2026
@raych1 raych1 self-assigned this Mar 4, 2026
@raych1 raych1 requested review from benbp and praveenkuttappan March 4, 2026 02:19
@raych1 raych1 moved this from 🤔 Triage to 🔬 Dev in PR in Azure SDK EngSys 📆🎇 Mar 4, 2026
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds MCP notifications/progress reporting to long-running azsdk-cli package operations so MCP clients can show heartbeats and milestone-based progress (helping avoid MCP request timeouts during generation/build/pack).

Changes:

  • Introduces ProgressReporter helper to emit heartbeat progress updates during long-running work and milestone progress updates for step-based progress bars.
  • Updates SdkGenerationTool, SdkBuildTool, and PackTool MCP tool methods to accept framework-injected IProgress<ProgressNotificationValue>? and to report progress/heartbeats.
  • Updates and adds unit tests to cover signature changes and ProgressReporter behavior.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tools/azsdk-cli/Azure.Sdk.Tools.Cli/Tools/Package/SdkGenerationTool.cs Adds MCP progress injection + milestone/heartbeat reporting around SDK generation flows.
tools/azsdk-cli/Azure.Sdk.Tools.Cli/Tools/Package/SdkBuildTool.cs Adds MCP progress injection + milestone/heartbeat reporting around build execution.
tools/azsdk-cli/Azure.Sdk.Tools.Cli/Tools/Package/PackTool.cs Adds MCP progress injection + milestone/heartbeat reporting around pack execution.
tools/azsdk-cli/Azure.Sdk.Tools.Cli/Helpers/ProgressReporter.cs New helper to send MCP progress notifications with CLI logging fallback.
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Tests/Tools/Package/SdkGenerationToolTests.cs Updates tool calls to include the new progress parameter.
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Tests/Tools/Package/SdkBuildToolTests.cs Updates tool calls to include the new progress parameter.
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Tests/Tools/Package/PackToolTests.cs Updates tool calls to include the new progress parameter.
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Tests/Helpers/ProgressReporterTests.cs Adds unit tests for ProgressReporter heartbeats and reporting behavior.
Comments suppressed due to low confidence (3)

tools/azsdk-cli/Azure.Sdk.Tools.Cli.Tests/Helpers/ProgressReporterTests.cs:69

  • These unit tests rely on real-time Task.Delay timing and assert on the number/order of heartbeat logs (e.g., expecting >= 3 logs after 350ms). This can be flaky under load or on slower CI agents. Consider making the tests deterministic by driving completion with a TaskCompletionSource and verifying that heartbeats are emitted when the heartbeat delay wins, or by injecting a delay/timer abstraction into ProgressReporter for testability so you can advance time without wall-clock sleeps.
    public async Task RunWithProgressAsync_SlowTask_EmitsHeartbeats()
    {
        // Arrange — task takes ~350ms, heartbeat every 100ms → expect initial + ~3 heartbeats
        var result = await ProgressReporter.RunWithProgressAsync(
            progress: null,
            _logger,
            "Slow operation",
            async ct =>
            {
                await Task.Delay(350, ct);
                return 42;
            },
            CancellationToken.None,
            heartbeatInterval: TimeSpan.FromMilliseconds(100));

        // Assert
        Assert.That(result, Is.EqualTo(42));
        // At least the initial message + 2 heartbeats (timing may vary slightly)
        Assert.That(_logger.Logs.Count, Is.GreaterThanOrEqualTo(3));

        // First message is the initial progress message (no elapsed)
        Assert.That(_logger.Logs[0].ToString(), Does.Contain("Slow operation"));
        Assert.That(_logger.Logs[0].ToString(), Does.Not.Contain("elapsed"));

        // Subsequent messages include elapsed time
        Assert.That(_logger.Logs[1].ToString(), Does.Contain("elapsed"));
    }

tools/azsdk-cli/Azure.Sdk.Tools.Cli/Tools/Package/SdkGenerationTool.cs:83

  • The initial progress notification always uses total: 3, but the tspconfig.yaml path reports milestones with total: 4 (steps 1-3). This can cause the client progress bar to jump/change totals mid-run. Consider computing the total based on which path is being used (tsp-location vs tspconfig) before the first ReportProgress, or omit total on the initial validation message until the branch is known so totals remain consistent for the full operation.
                ProgressReporter.ReportProgress(progress, logger, 0, "Validating inputs", total: 3);

tools/azsdk-cli/Azure.Sdk.Tools.Cli/Helpers/ProgressReporter.cs:89

  • RunWithProgressAsync reports "Xs elapsed" but tracks elapsed time by incrementing a counter by the heartbeat interval, which can drift from real wall-clock time due to scheduling delays (and can be noticeably wrong for long operations). Use a Stopwatch (or capture start time and compute actual elapsed each heartbeat) so the displayed elapsed time is accurate.
        var workTask = work(ct);
        var elapsed = TimeSpan.Zero;

        while (!workTask.IsCompleted)
        {
            // Wait for either the work to complete or the heartbeat interval to elapse
            try
            {
                await Task.WhenAny(workTask, Task.Delay(interval, ct));
            }
            catch (OperationCanceledException) when (ct.IsCancellationRequested)
            {
                // Cancellation requested. Fail fast if work has not completed yet.
                if (!workTask.IsCompleted)
                {
                    throw;
                }

                // If work already completed concurrently, exit the loop and
                // propagate the actual work result/exception below.
                break;
            }

            if (!workTask.IsCompleted)
            {
                elapsed += interval;
                var elapsedSeconds = (float)elapsed.TotalSeconds;
                var message = $"{activityMessage}... ({(int)elapsedSeconds}s elapsed)";
                ReportProgress(progress, logger, elapsedSeconds, message);
            }

else
{
// Fallback: log the progress message for CLI mode.
logger.LogInformation("[Progress] {Message}", message);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could also have some ascii art here to represent progress on the CLI side, or at least a percentage value string.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added that, this is the effect:
image

elapsed += interval;
var elapsedSeconds = (float)elapsed.TotalSeconds;
var message = $"{activityMessage}... ({(int)elapsedSeconds}s elapsed)";
ReportProgress(progress, logger, elapsedSeconds, message);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the pack tool, ReportProgress is called manually with values 1-4 for progressValue and a total of 4. But this runs in between those progress messages with a TotalSeconds value and with no total passed in, how does this show up to the user in terms of progress messages and percent to completion?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are different progress report. The step number and total are used to control the blue progress bar. The total seconds is the inline text of MCP tool run title. Please refer to my screenshot in description comment.

progress,
logger,
$"Building {languageService.Language} SDK project",
async (token) => await languageService.BuildAsync(fullPath, CommandTimeoutInMinutes, token),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wrapper/callback approach where the progress reporter runs the task seems overly complex. Stack traces will be more annoying to debug, and the reporter becoming a task runner feels like too much responsibility for the component. Is it enough to run Progress.Start() and Progress.Stop() around BuildAsync(), or some similar pattern? Possibly we could have annoying timing issues though where we report a final progress message after the main task is done.

Copy link
Member Author

@raych1 raych1 Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated it to using/idisposable pattern. Could you have another look? @benbp

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli

Projects

Status: 🔬 Dev in PR

Development

Successfully merging this pull request may close these issues.

4 participants